Comparing Subjective and Objective Measures of Knowledge During Surgical Residency

Timothy M Farrell, MD, Michael O Meyers, MD, Lindsee E McPhail, MD, Geoffrey P Kohn, MD, Mark J Koruda, MD, Anthony A Meyer, MD PhD. University of North Carolina at Chapel Hill

Objective: Surgery resident knowledge is assessed subjectively and objectively throughout training. Subjective evaluation depends upon faculty assessment. Objective measurement derives from the American Board of Surgery In-Training Examination (ABSITE) or similar tests. Few efforts have been made to characterize correlation between subjective and objective measures of knowledge, or to assess longitudinal trends.

Methods: From 2003-2008, resident knowledge was rated subjectively by the faculty on a scale of 3-15 after each rotation, and annual averages were calculated. In parallel, resident knowledge was evaluated objectively by annual ABSITE testing, and percentile scores were assigned based on postgraduate year (PGY)-stratified national normative data. Overall, 167 contemporaneous subjective and objective knowledge assessments were identified: 47 PGY-1, 38 PGY-2, 27 PGY-3, 28 PGY-4, and 27 PGY-5. Paired subjective and objective knowledge scores were compared overall and for each PGY level, and Pearson Product-Moment Correlation coefficients were generated (Table).

Results: Overall, a statistically significant (weak) positive association exists between subjective assessment of resident knowledge and ABSITE performance. Year-by-year analysis shows faculty ratings statistically correlate with resident ABSITE performance only during PGY-1 and PGY-5. There is no correlation in the middle years of residency, with a negative Pearson’s r-value in PGY-3.

Conclusions: Faculty assessment of resident knowledge correlates only weakly with ABSITE performance, with closest correspondence in PGY-1 and PGY-5 years. These variations may be explained by the higher proportion of specialty and off-site rotations that mark the PGY-2 and PGY-3 years in our program, and by the tendency for increased resident scrutiny during high-stakes entry and exit years. However, it is clear additional objective assessment tools are needed.