A mixed-method study design was used [23]. Quantitative data was generated from supervisors when assessing pre-recorded, simulated patient encounters performed by students. Individual interviews, conducted directly after the assessments, were completed in accordance with a mixed-method design.
Assessment instruments
The ACIEd used in this study consists of five LOs with sub-criteria concerning what the student needs to achieve to receive a Pass grade during a course in emergency care. The LOs are designed to cover all aspects of the six-week clinical education in relation to the course objectives resulting in LOs with several goals embedded in one, for example, LO 1, which translated reads:
-
1.
Approach and support patients and their relatives in respectful consultation and perform nursing care based on the patient’s experience of the situation.
To clarify this LO and reduce the risk of inconsistencies in assessment one-sentence LOs without sub-criteria was reconstructed in the AAI, and one example of this reads as follows:
-
1.1. To what extent was the patient treated with respect?
-
1.2. To what extent was the patient allowed to describe his or her situation?
-
1.3. To what extent did the student create a safe care situation?
The complete list of LO´s used can be found in appendix one. To enable formative assessment by displaying progress using the assumption that students’ performances will generate higher grades as their clinical education progresses, the AAI has a seven-point Likert scale with a pass grade marked as 5, meaning that scores 1 to 4 results in a fail grade. The Likert scale was given descriptions from 1 = “Not at all” to 7 = “To a great extent”. Furthermore, the ACIEd separates midterm assessment from final assessment, where grades in midterm are referred to as “In line with objective” or “Plan of action is required”. The final assessment using ACIEd results in a pass or fail grade. The AAI provides a formative assessment, which is meant to be repeated frequently but can also act as a basis for grading in an equivalent way as the ACIEd.
Simulated patient encounters
Four different recorded simulated scenarios were used in the study (recorded time: 6min 58 s, 3min 14s, 5min 58s, 2min 32 s). The scenarios had variations of student performances with the intention of generating variations in supervisor assessments. Differences were seen in both students’ assessments of the patient and in their treatment strategies. Two scenarios included a student interacting with a standardised patient, while two scenarios had a student interacting with a patient simulator manikin. The standardised patient was a middle-aged woman with fatigue and dizziness. The patient simulator manikin was a young man with abdominal pain. The nursing students participating in the simulation scenarios were recruited from the fifth semester of the nursing study programme at one university in Stockholm, Sweden, after they had finished their six-week clinical education in the ambulance services and all grades had been made official. Both male and female students participated in the simulations.
Participants
A convenience sample of 50 clinical supervisors, all RN in the ambulance service, participated in the study, having varying experience of supervisorship. The participants were recruited at three different emergency departments by the main author. By selecting different emergency departments, it was possible to include participants with different ambulance service employers (private and public). The clinical supervisors from the ambulance service were asked to participate in the study after they arrived at the emergency department and the hand-over of the patient was completed. The convenience sampling of clinical supervisors was used due to the difficulty in recruiting participants at the ambulance station, as they were constantly mobile during their shifts. The supervisors were provided with both written and oral information about the study as well as a letter of consent for participation in the study. No participants were excluded due to experience, gender, educational level, or other factors.
Data collection
In total, 50 clinical supervisors from the ambulance service assessed and graded four recorded simulations in accordance with the LO described in the ACIEd and the AAI instrument. No supervisor declined the offer to participate in this part of the study. Before the assessment started, a randomisation process was conducted. Firstly, the order of the simulations was randomised using a lottery (standardised patient vs. patient simulator manikin.). Secondly, the two scenarios were randomised using lottery (scenarios 1 and 2). No power calculations were performed. The recorded simulations were watched in one sequence without time to reflect or discuss the assessment and grading with others. All supervisors had prior knowledge of the assessment instrument ACIEd but had never used AAI.
The qualitative data consisted of interviews, conducted after the participants had assessed the simulated patient encounters. The interviews started with the open question “What are your thoughts about the assessment instruments?” Probing questions were then used to explore the participants’ experience of using the assessment instruments when grading the student’s performance in the recorded simulations. The interviews varied in length from a few minutes up to 30 min. In total, 28 interviews were conducted (Female: 11 Male: 17). Fieldnotes were used during the interviews and after every finished interview the fieldnotes were summarized and reviewed. Theses reviews were used to make changes to the probing questions in relations to the aim of the study. In 22 cases there were no interviews conducted due to participants’ shortage of time. The supervisors were not informed about the questions prior to the interview.
Data analysis
The data was analysed in two parts, a quantitative and qualitative. The quantitative data was analysed by compiling the assessments generated by the two assessment instruments and compared in a simple figure where the difference was described from the perspective of how many pass grades the assessment instrument generated. Secondly, a logistic binary regression analysis was used to examine whether the grades generated by two different assessment instruments was affected by the supervisor’s gender and/or experience. To explore whether work experience as an RN affected the grading of student’s performance, a dichotomisation of the work experience variable was carried out. The variables were dichotomised into \(\le\) 6 and 6 > years of working experience as RNs (Dichotomised 1 for \(\le\) 6 and 2 for 6 >). This dichotomisation was based on the theory by Benner, that experienced nurses can use their experience, knowledge, and additional perspectives instead of relying on standardised guidelines, tests, and regulations [2] to assess the students according to the LO`s. A gender dichotomisation was also performed to investigate if there was any difference in grading related to gender which was defined as male or female with no consideration to other gender definitions (Gender was coded 1 for males and 2 for females). This dichotomisation was based on the diversity of the staff in the ambulance service. The independent variables for both instruments were coded as 1 for a pass grade and 2 for fail. After considering the number of included supervisors in the study the p value was set to 0.05. The data was analysed using Statistical Package for the Social Sciences, version 24, Chicago, IL, USA in combination with Microsoft Excel 2010 (Microsoft Corp, Richmond, WA, USA).
The interviews were analysed using a thematic analysis approach [5]. The method was chosen due to its flexible nature. The themes constructed was done in a “theory driven way” meaning that the research question was clearly present in the coding in contrast to an inductive analysis approach. First, in the analysis the field notes were read several times to gain familiarity with the content. Secondly, codes were identified that described clinical supervisors’ experience of using the two different assessment instruments were identified. Thirdly, the codes were examined, and by identifying broader patterns of meaning, potential themes were constructed. Fourthly, a thematic map was constructed, and the themes were checked on two levels. First the codes were checked against the theme making sure that the codes formed a coherent pattern. In the second level the themes were checked against the entire dataset and in relations to the other themes to ensure that the themes did not intertwine with each other and finally, the themes were named.
In the fifth step the themes and sub-themes were related back to the narrative making sure that the themes captured the full story and that each theme was unique and contained valid information. The “story” that the themes and sub-themes were checked against, was the researcher’s contextual knowledge and experience. Lastly, the report was written up using the themes and the sub- themes. The first author initiated the analysis and the corresponding author participated in the analysis process.
link