Summary of main findings
This study found that ECTVs were widely perceived by registrars and ECT visitors to be of high educational utility. Registrars also generally perceived ECTVs as having a high likelihood of influencing changes in their practice and in their approach to learning. Collectively this supports the overall educational utility of ECTVs as a WBA in vocational general practice training.
In multivariable analysis, we found no statistically significant differences in perceived educational utility of remote ECTV modalities when compared with face-to-face visits. This observation was consistent for both registrar- and ECT visitor-reported outcomes. There was some evidence (p = 0.07 on multivariable analysis), however, for ECT visitors rating video/phone ECTVs less favourably than face-to-face ECTVs.
Several features of feedback provided during ECTVs were significantly associated with registrar ratings of educational utility. The most striking finding was for the effect size regarding quality of feedback provided, where the odds of registrars rating the educational utility of the visits as ‘very useful’ were 12.8 times higher if the quality of the feedback provided was also rated as’very useful’. This was similarly reflected in likelihood of changing practice and approach to training/learning outcomes, albeit with more modest effect sizes (multivariable ORs of 2.5 and 3.2 respectively). More frequent provision of feedback during the visit (i.e., for three or more consultations/cases), and the consistency of the ECT visitor’s feedback with that previously given by the registrar’s supervisor, were also significantly associated with registrar-rated educational utility outcomes.
ECT visitor perceptions of the educational utility of ECTVs were generally consistent with registrar perceptions. However, there were some differences in ECTV content-related independent variables associated with utility ratings when compared with registrars (for example, the specific topics discussed).
Comparison with previous literature
To our knowledge, this is the first quantitative study to examine the perceived educational utility and associations of utility ratings for ECTVs, inclusive of both traditional face-to-face and remote ECTV modalities. A recent Australian qualitative study found that general practice trainees and assessors value the educational opportunities offered by ECTV direct observations and highlighted the importance of feedback conversations between consultations during the visit [17]. Our findings, about educational utility and associations of feedback provided during ECTVs, triangulate well with the observations of Sturman et al. [17] Our results about frequency of feedback also align with the findings of an earlier qualitative study investigating the value of adding RCA to direct observation assessments, which suggested that observation and feedback provision for three or more consultations is required for generating learning [16]. Our results regarding the educational utility of CNA-ECTVs, which share many similarities to RCAs conducted with direct observation, also support the conclusions of Ingham et al., where RCAs were viewed as having utility for learning within the direct observation session [16].
While we are not aware of any previous studies that have similarly assessed educational utility of face-to-face and remote formative assessment modalities within the GP training setting, there is some evidence regarding formative assessment conducted using videotaped sessions of registrar consultations [24, 25]. Although our findings are not directly comparable with these earlier studies, which did not involve real-time videoconferencing or teleconferencing, collectively the findings of these studies support a role for remote observation of practice for formative purposes. A recent systematic review of the feasibility and acceptability of remote clinical assessments in a variety of clinical settings, including general practice, similarly indicated overall in-principle support for the potential for remote assessments for summative and formative purposes [26].
Within existing medical education formative feedback theory literature, face-to-face discussion of feedback is regarded as one of the fundamental principles of effective feedback [27]. The apparent lack of difference in perceived educational utility outcomes between face-to-face and remote modalities in our study fails to support this.
Our findings about features of the feedback received (including receiving feedback for three or more consultations/cases, receiving feedback that is specific and actionable, and consistency of the feedback received with that of the supervisor) being positively associated with all three registrar educational utility outcomes, is consistent with the inherent link between feedback and achieving the formative aims of WBA [28, 29]. Effective feedback has been widely considered in medical education WBA contexts, and while there are many acknowledged complexities in what constitutes effective feedback [30], our findings empirically support the value of feedback that is interactive, timely and specific [31, 32].
Our multivariable findings are also consistent with education feedback theory, which asserts that it is the nature, content and collaborative discussion of feedback that drives its formative utility [27, 33]. Van de Vleuten et al. emphasise, in their theory-based framework for programmatic assessment, that feedback effectiveness is a function of the user (i.e., assessor and learner), rather than the instrument/s utilised. [11]. Our study provides empirical evidence to support the applicability of this theory within the direct observation WBA context in general practice training, where quality feedback was perceived as useful regardless of the ECTV modality by which the feedback was delivered.
Our findings also support the robustness of Pelgrim’s assessment feedback process schema, which identifies feedback content and ‘delivery’ as the central element of useful feedback obtainment, while noting a need for more exploration of further external/contextual variables [20]. Our observation that the features of the feedback received were most strongly related to perceived educational utility outcomes than a diversity of other situational/contextual factors, supports that contextual variables are a less prominent feature than the feedback itself within an effective feedback process model.
A specific novel aspect of our findings, not well-explored in existing literature, is that the non-direct observation formats (CBD-ECTVs and CNA-ECTVs) appear to provide a similarly useful substrate for educationally useful feedback when compared with direct observation ECTV formats.
Implications for practice, policy, and future research
Our findings support the assertion that ECTVs have a high level of perceived educational utility, affirming their ongoing inclusion as a fundamental element of formative assessment frameworks within Australian vocational general practice training. Our findings on ECTVs are likely transferable to other medical training programs internationally that use similar formative direct observation within their assessment frameworks. Implementation of ECTV-style assessments may be of interest for other medical education programs as a means of increasing the frequency of direct observation, which is emphasised as critical for optimising competency-based medical education [34]. We also suggest that in the context of formative WBA, non-direct observation modalities may offer a similarly viable substrate for useful, actionable feedback for learning, which is also likely to be generalisable across different medical education WBA training contexts.
The similarities of perceived educational utility across different ECTV modalities provides initial support for inclusion of remote ECTV modalities within post-pandemic WBA frameworks. Potentially, the number of traditional face-to-face ECTVs could be reduced by substituting in some remote ECTV modalities, without detriment to the overall educational value for the registrar. A ‘hybrid’ ECTV delivery model, combining a mixture of face-to-face and remote ECTVs across the course of training, may capitalise on the benefits of both assessment modalities, including resource optimisation, convenience, and varied learning opportunities. Furthermore, through inclusion of remote ECTV modalities, it may be possible to offer additional ECTVs without imposing substantive additional resourcing burden across a registrar’s course of training.
It must be acknowledged, however, that the different types of ‘remote ECTV’ that were included in the study ranged from real-time, videoconference, ECT visitor direct observation of patient consultations, through to telephone-based discussion of previous cases selected by the registrar. Although we did not find a difference in overall perceived educational utility of different assessment modalities, the inherent differences between these remote formats and that of traditional face-to-face direct observation ECTVs, suggest potential for differences in ‘educational richness’ that were not captured in the present study. Further research is required to explore how different context-specific learning experiences may vary by modality. For example, opportunities for observation of performing physical examination has been previously identified as a limitation of remote assessment modalities [2, 26]. Further investigation of the benefits and limitations of different modalities is required to gain a better understanding the relative merits and limitations of different ECTV modalities.
Given the salient relationship between feedback quality and educational utility of ECTVs, it is essential that ECTV training and implementation (for both registrars and ECT visitors) incorporates evidence-based principles of feedback, including specificity, actionability, timeliness, two-way/collaborative, and underpinned by mutual trust and credibility [27]. This is particularly important for remote modalities, where video- or telephone-based observations may present additional challenges in fostering the personal connection necessary for effective feedback [35, 36].
The association we identified for consistency of feedback from the ECT visitor with that of the supervisor (where registrars perceived this feedback to be of more utility) is an interesting consideration for ECTVs, where the implicit intent of the visit is to obtain feedback from an independent, experienced, and objective GP assessor. While consistency of feedback across different assessors may enhance its perceived trustworthiness, our findings raise an interesting question about registrars’ potential uptake of feedback from the ECTV in cases where it is inconsistent with that of their supervisor. This suggests a need for registrars to be supported in developing ‘feedback literacy’, defined as ‘appreciating feedback, making judgments, managing affect, and taking action’ [37, 38], to help maximise learning potential of the ECTV.
Differences in associations of educational utility ratings between our registrar and ECT visitor analyses may reflect ECT visitors and registrars having different priorities for learning within the ECTV, with ECT visitors placing greater emphasis on more holistic aspects of professional practice. These different priorities and perspectives should be considered in the feedback conversations that occur during ECTVs.
Our quantitative study provides foundational insight into the educational utility of different ECTV modalities, emphasising the key role of feedback within these WBAs. Further qualitative research is required to explore the barriers and enablers of effective feedback within ECTVs, and to gain a deeper understanding of the interplay between substrates of educational richness, feedback, and practice change.
Strengths and limitations
The strengths of this study include a large sample size, and a reasonable response rate for studies of GP clinicians [39], particularly notable given pandemic conditions. The study included participants from a variety of practice settings across multiple RTOs (encompassing urban, rural, and remote training settings, and diverse socio-economic profiles) and included GP registrars across different stages of training. Perceptions of both registrars and ECT visitors provided multiple perspectives. Many clinically and educationally relevant independent variables were included as covariates in analyses, enabling comprehensive consideration of factors that may relate to perceived educational utility of ECTVs.
The pandemic-driven shift to remote ECTV delivery concurrent with the study provided the opportunity to assess perceived educational utility of both face-to-face and remote ECTV modalities. However, the potential effects of the pandemic context of data collection on the generalisability of the findings should be acknowledged. It is possible that the pandemic setting may have impacted upon registrars’ and ECT visitors’ judgements about how educationally useful the ECTV was, although the potential direction of this bias is unknown.
Other limitations are noteworthy. First, the cross-sectional nature of the analyses precludes inference of causality despite several strong relationships identified. Second, we used registrars’ self-report for the likelihood of changing their practice and training/learning behaviours. Practical and resourcing constraints precluded objective measurement of actual behaviour change in this study and we were unable to explore the intention-behaviour gap. Despite inherent limitations of using self-reported intention, there is evidence to support intention as a valid proxy measure for behavioural change [40]. Third, there was a large ceiling effect for the utility and likelihood to change outcomes, necessitating grouping ratings of one to four, for comparison with the highest rating of five. This could have impacted our ability to detect variability between the different ECTV modalities.
Fourth, a registrar’s previous exposure to a face-to-face ECTV could have had an impact on their ratings of subsequent remote ECTVs. It is possible that some registrars participating in the study had not previously experienced a face-to-face ECTV and therefore had no criteria to contrast remote ECTVs against. Finally, information technology (IT) issues were not specifically addressed in the questionnaire. Such logistical/practical issues have been acknowledged as problematic for remote assessment [26], and must be considered and addressed in the future rollout of remote ECTV modalities.
link