M C Vassiliou, MD, MEd, P Kaneva, MSc, B J Dunkin, MD, G M Fried, MD, J D Mellinger, MD, T Trus, MD, K I Hoffman, PhD, C Lyons, MD, J R Korndorffer, MD, M Ujiki, MD, V Velanovich, MD, G Korus, MD, D J Scott, MD, J Martinez, MD, S Tsuda, MD, J Seagull, MD, G Adrales, MD, M L Kochman, MD, A Park, MD, J M Marks, MD
McGill, Methodist, Southern Illinois, Dartmouth, USC, Tulane, NorthShore, South Florida, U Penn, UTSW, U Miami, U Nevada, U Maryland, Case Western
INTRODUCTION
The Fundamentals of Endoscopic Surgery (FES) program consists of online educational materials and both didactic and skills based tests. All components must be shown to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study, performed by the FES task force, was to evaluate the reliability and validity of the hands-on component of the FES examination.
METHODS
Expert educators and endoscopists identified the critical skill set required for flexible endoscopy by deconstructing the procedural components of upper and lower endoscopy. The skills were then modeled in a virtual reality simulator (GI MentorII, Simbionix Ltd., Israel) and metrics were created. Several pilot studies and iterations were needed in order to refine the skills and metrics. Scores were designed to measure both speed and precision. Validity was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists (GI)). Internal consistency of each test task was assessed using Cronbach’s alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting group methodology and use of receiver operating characteristic curves.
RESULTS
The 5 simulated tasks include: scope navigation, loop reduction, retroflexion, mucosal evaluation and targeting. 158 participants (16% GI) performed the simulator test. Scores on the 5 tasks showed sufficient internal consistency reliability and all had significant correlations with the participants’ level of endoscopic experience as measured by self-reported number of cases performed. A composite score was obtained by averaging the five task scores. The composite scores correlated .72 with participants’ level of endoscopic experience providing evidence of their validity and their internal consistency reliability (Cronbach’s alpha) was .82. Test-retest reliability was assessed in 7 participants, and the ICC was .96 (CI .74; .99). The passing score for the minimally qualified candidate was determined and is estimated to have a sensitivity (True positive rate) of .81 and a 1-specificity (False positive rate) of .21 given the pilot sample.
CONCLUSIONS
The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.
Session: Podium Presentation
Program Number: S103