This cross-sectional study was conducted at the Qom University of Medical Sciences from 2023 to 2024. The study consisted of two main steps: validation of the Situational Judgment Test (SJT) and evaluation of students’ situational judgment in clinical professions.
Participants
In the validation step, 27 experts participated. Their mean (SD) age was 46 (SD = 7) and their mean work experience of 12 years (SD = 4). The expert included 12 women (44.4%) and 15 men (55.5%). For the reliability assessment, the learner group consisted of 12 men (48%) and 13 women (52%), with a mean age of 24 (SD = 2).
In the second step, senior students who had completed their internship course in teaching hospitals were included, meeting the inclusion criteria. Students who unwillingness or completed the SJT partially were excluded. The sample size was calculated using a \(n = \sigma ^2Z_1 – \alpha \over 2^2 \over d^2,\,(Z^2 = 3.84,\,\sigma ^2 = 3.54,\rmd^2 = 0.09,\,\rmN = 150)\).
Participants were selected through stratified random sampling, dividing the population into subgroups based on shared attributes [19, 20]. Stratified sampling is appropriate when the variance within groups is less than the variance between groups. Stratified sampling was employed in this study under the assumption that situational judgment skills were more similar within each major group than between different groups. To ensure a diverse and representative sample, random numbers were generated based on the list of students in each profession, and participants were selected accordingly. This randomization process ensured a diverse and unbiased selection of participants for the study. The participant group (n = 207) was divided into subgroups of medicine, surgical nursing, anesthesia nursing, and medical emergency technology students.
Study tools
Data collection tools included a demographic characteristic form and the Situational Judgment Test (SJT). The original SJT, developed by Alkhuzaee et al. in 2022, consisted of 10 scenarios, with six set in a workplace setting and four in a classroom setting, addressing professionalism domains such as accountability, respect for others, and honor and integrity [13].
First stage: adapting and validation of the SJT
The contextualization and validation processes were conducted based on a modified “Toolkit on Translating and Adapting Instruments” and “Guidelines for Translating and Adapting Tests” [21, 22]. This step involved a comprehensive evaluation of the definition and content of the construct measured by the SJT. Linguistic, psychological, and cultural differences were carefully assessed through the opinions of experts with relevant expertise, ensuring the instrument’s applicability and relevance in the Iranian context.
External validity assessment of the SJT
The first step in the contextualization and validation process involved forward and backward translation procedures to ensure the suitability of the SJT adaptation for students in clinical education courses. Two professional translators with experience in professional teaching translated the English version of the SJT into Persian. The translated SJT was then reviewed by an expert panel consisting of clinical educators, experts in medical ethics, an expert in health professions education, and two translators. The panel ensured the accuracy and cultural appropriateness of the translated version. Subsequently, a back-translation procedure was conducted by an independent translator, comparing the original and back-translated versions to ensure accuracy and quality. The external validity of the Persian version of the SJT was confirmed by experts, ensuring its applicability in the Iranian context.
Face and content validity assessment
The face and content validity of the SJT, developed by Alkhuzaee (2022) [13], was assessed in terms of cultural and regulatory issues in the investigated context. Hofstede’s research revealed cultural similarities between Saudi Arabia, where the original SJT was developed, and Iran, across of four dimensions: moderate masculinity/femininity, moderate uncertainty avoidance, large power distance, and high collectivism [16]. Thus, the SJT of developed by Alkhuzaee (2022) [13] was selected as the basis for adaptation to the Iranian context.
The content validity assessment comprised of several steps: (1) Qualitative validity assessment of the SJT stem items to evaluate students’ situational judgment skills in clinical education, (2) Development of item options based on a selected-response format, (3) Qualitative evaluation of the content and face validity of the item stem and options, (4) Quantitative validity assessment of the SJT, and (5) Collection of pilot data on the adapted SJT. An expert panel, consisting of professionals in the fields of professionalism, health professions education, medical ethics, and clinical education, actively participated in this process.
Qualitative validity assessment of the item stem of SJT
The item stems from the SJT was designed for a pharmaceutical setting in Alkhuzaee’s study. In the present study, the content validity of the translated SJT was assessed in terms of psychological and cultural differences, compliance with ethical codes of conduct, and alignment with professionalism principles. Key questions addressed in this step included: Do the SJT situations effectively assess students in clinical education courses? Are the situations included in the original version of the SJT realistic for clinical education in the Iranian context? Are the situations compatible with professionalism principles in the Iranian context?. Experts completed the rating form for questions, rating each item on a 5-point scale from “completely” to “not at all.” Stems that achieved an agreement of more than 80% in the questions were confirmed. Three stem items that scored less than 80% agreement in the questions underwent revision and were discussed in the panel for potential changes. The modifications were reviewed and approved in the step.
Development of item options based on a selected-response format
The original response format of the SJT was reviewed by experts, who suggested modifications to align with a selected response format. The SJT is designed with a specific criterion domain of professionalism, and the item options are embedded in context-specific situations. The SJT was modified to the ‘best three responses’ questions. The questions employed a selected-response format, requiring participants to choose the best three responses from six options. This approach aligned with a knowledge-based approach. A team of five experts in the professionalism field developed the item options, ensuring their relevance and appropriateness. The final version of the SJT with a selected-response format was compiled, providing a comprehensive assessment tool for students’ situational judgment capabilities.
Quantitative validity assessment of the SJT
The quantitative content validity of the selected-response SJT was evaluated using the “content validity ratio” and “content validity index.” Fifteen experts examined each question on a three-point scale (“necessary,” “useful, but not necessary,” and “not necessary”). The minimum value of the content validity ratio was determined based on the Lawsche Table [23]. For the content validity index, the relevance criterion for each question was rated using a four-point Likert scale [24]. The results of the validity assessment were discussed in the expert panel, leading to the approval of the Persian version of the selected response format of the SJT. The instrument consists of 10 scenarios, each presenting six options, with scores ranging from 0 to 30. (Appendix 1).
Ccollect pilot data on the adapted SJT
A pilot study was conducted with the participation of nursing and medicine students (n = 25), ensuring the SJTs’ applicability and reliability in assessing students’ situational judgment capabilities in clinical education. They reviewed the test instructions, SJT stems, and options in terms of comprehension and acceptability. The internal consistency of the SJT was analyzed using Cronbach’s alpha, a popular method for evaluating the reliability of the SJTs [25]. The results of the pilot study were discussed in the expert panel, and any necessary revisions to the adapted test were made. The final modified version of the SJT with a selected response format was confirmed.
The second stage: SJT Examination
During the examination sessions, participants were informed about the research purpose, the format and instructions of the SJT, and their right to participate or refuse. Confidentiality of information was assured, and informed consent was obtained. Students were asked to complete the SJT by choosing the best activities (three options) for each question within a 45-minute session. The examination was conducted under invigilation to ensure integrity.
Standard-Setting Cut-off scores were calculated using two methods: a norm-referencing standard deviation approach (mean minus standard deviation) [26] and Cohen’s method [27], which combines norm-referenced and criterion-referenced approaches. The students’ scores were compared to these cut-off scores. Cohen’s method uses the advantages of the incorporation of a norm-referenced and criterion-reference method, where 60% of the 95th percentile score is considered as the cut-off score. The cut-off score was determined using a norm-referencing standard deviation approach [26], calculated as the SJT mean score minus 1 standard deviation, following the recommendation of the Sahota study [6].
Data analysis
Data analysis was summarized using descriptive statistics (frequency (percentage), mean (standard deviation). Independent Student T-test and ANOVA assessed the univariate difference between the means of SJT scores in categorical variables such as gender and professions. Also, the Pearson correlation was used to evaluate the univariate association between the SJT scores and the age of students. The Analysis of covariance (ANCOVA) assessed the association between students’ SJT scores and profession, adjusting for gender, and age. Data was analyzed using IBM SPSS Statistics 26.0. The level of significance was considered 0.05.
link