When can we identify the students at risk of failure in the national medical licensure examination in Japan using the predictive pass rate? | BMC Medical Education

There are two important findings of this study. Using the logistic regression analysis, we identified 2 to 6 significant predictors of passing NMLE at the five time points in school, including at admission, 1st, 2nd, 4th, and 6th grade, respectively. Second, using the PPR calculated by the logistic regression formula, we identified the “high-risk group” for failing the NMLE at each of the five time points and showed that using the PPR at an earlier time point (even at admission) can potentially identify the students at risk of failure in the NMLE.

To our knowledge, a few studies have sought to identify or predict student failure in NMLE [9, 21, 22]. However, our results cannot be directly compared to these studies because their models were based on only academic performance and did not include the background characteristics of students.

Significant predictors of passing NMLE at each time point

In the current study, we identified some significant predictors for passing NMLE at each of the five time points. Especially, age at admission and location of the HS in the neighborhood were identified as external predictors. The number of available variables increased with the advance in the grade, increasing the accuracy of the analysis.

Among the variables of academic performance in university such as total score of clinical medicine, CBT-IRT score, Performance in clinical clerkship, and score of graduation examination, only CBT was a nationwide examination. It is administered by the Japanese Ministry of Education, Culture, Sports, Science and Technology (MEXT) for all Japanese medical schools in the fourth year using a computer before clinical clerkship to assess the ability for clinical clerkship. Factors that are significant in multivariate analysis at the sixth year have been discussed in a previous paper [11]. In Brief, CBT as a predictor is very similar to that of USMLE step 1. Elder medical students are more likely to have insufficient study time because of their duty time such as family or working part-time, decline in memory, burnout, and life events such as childbirth and parental care. As for age and gender, these factors should not be used as criteria for selection because they are important issues related to discrimination and bias [23] and should be seen as non-modifiable factors. However, making these at-risk students aware of their higher risk of failure would help mitigate the risk. It is unclear why medical students who belonged to a neighborhood HS have a better chance of passing the NMLE, but similar results have been reported in a previous study in a Japanese medical school [24].

In the multivariate analysis of the early grades, academic performance at school such as basic sciences and basic biomedical sciences was a significant positive factor, while in the multivariate analysis of the fourth and sixth years, clinical medicine score was a dominant negative factor. Although this seems to be a contradictory result, in the univariate analysis, the number of failures was significantly lower for all factors of academic performance. This may also be attributable to the fact that the clinical medicine score is similar to the graduation examinations, which have a stronger positive influence as an explanatory force. As for the other predictors of medical school performance, previous studies have shown that the Medical College Admission Test (MCAT) score is a useful indicator of the performance of students in medical schools [25,26,27]. In particular, Wiley and Koenig found that MCAT scores had a slightly stronger correlation with medical school grades than did undergraduate GPA [26]. Koenig et al. indicated that MCAT scores, alone and in combination with undergraduate GPA, are good predictors of medical school performance, but not perfect [29]. Julian also investigated the validity of MCAT scores for predicting medical school performance (medical school grades, USMLE step scores, and academic distinction or difficulty) and found that MCAT scores were better predictors of USMLE step scores than were undergraduate GPAs, and the combination only slightly outperformed the MCAT scores alone [29]. Recently, Zhao et al. reported that the repeaters for MCAT are expected to achieve lower Step 1 scores than non-repeaters [30]. In Denmark, O’Neill et al. evaluated the predictive validity of non-grade-based admission testing versus grade-based admission relative to subsequent dropout and found that admission test students had a lower relative risk for dropping out of medical schools within 2 years of admission (OR 0.56, 95% CI 0.39–0.80) [31]. In our study, on the other hand, there were differences between passers and failures for NML-J in terms of the scores of NCTUA and GPA before admission in medical schools in the univariate analysis, but not in the multivariate analysis. The differences between our results and others may be attributable to the differences in examination period between NMLE and Steps 1 and 2. In addition, the lack of significant differences in our study may be because the averages and ranges of TCTUA and GPA are higher and narrower in medical students passing the admission exam. This may also be due to the distinct context of the entrance examination system for medical schools in Japan. For instance, the medical school admission quota in Japan is very small compared to the US and European countries, and medical school candidates are forced to work hard to compete with each other (i.e., maximal magnification ratio for the last decade in GUSM was about one hundred which was highest in Japan). Donnon et al. conducted a meta-analysis of published studies to determine the predictive ability of the MCAT score for medical school performance and medical board licensing examinations. They found that the predictive ability of the MCAT score ranged from small to medium [32]. Lastly, Ramsbottom-Lucier et al. reported a modest gender difference for the NBME I, with the men performing better than the women [33]. Recently, McDougle et al. indicated that the relative risk of first attempt STEP 1 failure for medical school graduates was 3.2 for women (95%CI: 1.8–5.8, p = 0.0001) [34]. Contrarily, in the study by Koenig et al., sex and race were not included in the subsequent prediction equations [28]. In the present study, gender showed a significant influence on passing NMLE in the univariate analysis, with the women outperforming men, but not in the multivariate analysis. Further research is required because of the inconsistent results in terms of the influence of gender.

Predicting NMLE with data in lower grades

Our previous study suggested the possibility of predicting NMLE in the early grades, and as shown in Table 3, we found similar results in multivariate analysis in the early grades. This reveals that some degree of risk analysis is possible using a similar method not only at graduation but also at lower grade levels and even at admission.

Baars et al. also developed a model for the early prediction of students who fail to pass the first year of the undergraduate medical curriculum within two years after the start [8]. In their study, independent variables included 5 pre-admission and 4 or 5 post-admission variables, and the predictions for failure in the first-year curriculum were made at 0, 4, 6, 8, 10, and 12 months by logistic regression analyses [8] Their results showed that students who had passed all exams at 4, 6, or 8 months (so-called “optimals”) had a 99% chance of passing the first-year curriculum. The earliest time point with the highest specificity to predict student failure in the first-year curriculum was 6 months; however, additional factors are needed to improve this prediction or to bring forward the predictive moment [8]. It is well-known that the majority of students who are not successful fail to perform well during their first year in university [8, 13, 35]. All MSs are faced with the issue of poor-performing M-1 students [15]. The challenge is to encourage these students to take remedial programs that address their academic problems and assist them in becoming high-performing physicians [15]. Kies and Freund indicated that medical students who decompress their M-1 year prior to M-1 year failure outperform those who fail their first year and then repeat it. They suggested the need for careful monitoring of the performance of M-1 students and implementing early intervention and counseling of struggling students [6].

Improvement of actual pass rate for NMLE after intervention in 2018

In GUSM, from 2012 to 2017, specific intervention was implemented for students who performed poorly in the mock exam (ME) conducted approximately four months before the actual NMLE exam. However, this intervention was not effective because not all poorly performing students took the ME as participation in the ME was voluntary. Additionally, some young students with poor performance in the ME were able to pass the actual NMLE, while some older students with good performance in the ME did not pass the actual NMLE.

Therefore, as discussed previously, using PPR and a new sample in 2018, we picked up the 15 candidates who had lower PPR for NMLE (≤ 95%), indicating a strong likelihood to fail NMLE at the first attempt, to confirm the validity of the formula (see Table 3). Additionally, the students were provided adequate guidance by the competency committee to overcome their shortcomings before taking the actual NMLE. This dramatically improved the actual pass rate for NMLE at the first attempt in 2018. Moreover, PPR predicted all 5 failures who were included in 15 candidates. This suggested that performing risk analysis based on several variables, such as PPR, can lead to more effective intervention compared to a single variable, i.e., performance in the ME. Further prospective studies are needed in other cultural settings to confirm the validity of PPR.

Strengths and limitations

Some limitations of this study should be acknowledged. First, our results cannot be directly compared with those of previous studies because of the different independent variables used. Second, our results may be influenced by the differences with respect to the selection of medical students and the medical education system in Japan compared to other countries. Third, the applicability of our results to other Japanese MSs is not clear because no similar studies have been conducted in other schools and the duration of the prospective study was only one year. Finally, there are inherent pitfalls of using prediction models such as overfitting [36].

Implications for future research

Improving the reliability of PPR developed in the current study may help reduce the number of failures in NMLE, USMLE, or the undergraduate medical curriculum. As for the next steps, we are planning a new prospective study lasting at least several years to obtain more robust evidence of the applicability of PPR. A consistent program of support needs to be developed for students at high risk of failure when they enter the program. In addition, these data would allow for more targeted studies using area under the curves and conditional inference trees.