Jackie S Cha, MSE1, Nicholas E Anton, MS2, Tomoko Mizota, MD2, Julie M Hennings, MSN, RN, CCRN2, Megan A Rendina, BS2, Katie Stanton-Maxey, MD2, Hadley E Ritter, MD2, Dimitrios Stefanidis, MD, PhD2, Denny Yu, PhD1. 1Purdue University, 2Indiana University School of Medicine
INTRODUCTION: The objective of this study was to assess the reliability of a modified NOTECHS rating scale for the evaluation of medical students’ non-technical (NT) skills. The importance of physician NT skills for the safe care of patients is receiving increasing attention in the literature. Tools to assess NT skills such as NOTECHS that addresses communication, situation awareness, cooperation, leadership, and decision-making have been shown to be valid and reliable. Despite its importance, the assessment of NT skills of medical students, our future physicians, has received little attention.
METHODS AND PROCEDURES: Twenty-seven medical students participated in 1 of 6 acute care simulated scenarios, each approximately 10 minutes long. Video recordings of student performance were reviewed and assessed using a modified NOTECHS rating tool adapted for these scenarios with input from a team of clinicians, nurses, and human factors specialists. The rating scale ranged from 0-6, 0 representing very problematic behavior (e.g., not vocalizing concerns or decision process) and 6 representing model behavior (e.g., identifies future problems and remains calm to unexpected events). Two reviewers rated all videos independently on the 5 NOTECHS domains and specific subscales. Student scores in each NT skill domain and interrater reliability were assessed.
RESULTS: A summary of the scores of each NOTECHS domain is shown in Table 1. The highest overall average score of a participant was 4.9 while the lowest was 1.5. The intra-class correlation (ICC; two-way random model) was 0.66, and the Cronbach’s α coefficient was >0.62. The lowest ICC agreement was in the situation awareness domain (0.59) while the highest agreement was in leadership (0.73).
Table 1. Summary of NOTECHS scores from two annotators.
CONCLUSION: Medical student NT skills during acute care simulated scenarios vary significantly using a modified NOTECHS assessment. This newly developed tool provides a framework for educators to evaluate medical students’ NT skills during simulation training. It further identified domains where students scored lower, such as situation awareness, and could be targeted for education. The moderate ICC, between the 0.5-0.75 range, shows that further refinement of the tool is needed to reliably assess the constructs. Future steps to obtain validity evidence include additional raters and applying the tool in non-simulated settings.
Presented at the SAGES 2017 Annual Meeting in Houston, TX.
Abstract ID: 87307
Program Number: P331
Presentation Session: iPoster Session (Non CME)
Presentation Type: Poster