Shanley B Deal, MD1, Dimitrios Stefanidis, MD, PhD, FACS, FASMBS2, Robert D Fanelli, MD, MHA, FACS, FASGE3, Marian P McDonald, MD, FACS4, Dana Telem, MD, MPH5, Michael Ujiki, MD6, L. Michael Brunt, MD7, Adnan A Alseidi, MD, EdM, FACS1. 1Virginia Mason Medical Center, 2Indiana University, 3The Guthrie Clinic, 4St. Luke’s University Health Network, 5University of Michigan, 6Northshore University Chicago, 7Washington University St. Louis
Introduction: Crowd-sourced assessment of technical skills is revolutionizing the way we evaluate intraoperative technical performance and provide education. Research has demonstrated strong correlation between crowds and experts in technical assessment. Discovering ways to apply crowd sourcing for advanced assessment are needed. We hypothesized, that with education, crowd workers (CW) would be non-inferior to faculty experts (FE) in identifying operative quality and safety.
Methods and Procedures: We prepared 160, blinded videos of laparoscopic cholecystectomy from public websites. Videos were edited to ≤60 seconds and ended when a structure was cut/clipped. CW were laypersons included based on performance history, passing an attention test and being ≥18 years old. CW and FE watched a tutorial reviewing how to rate videos using the CVS doublet criteria in which points were awarded if the component was seen in the anterior or posterior view. CW rated videos with a minimum of 50 ratings per video. These videos were stratified into performance quartiles and 10 videos were randomly generated from each quartile (n= 40 videos) that were rated by five FE from the SAGES Safe Cholecystectomy Task Force. Both groups assigned a pass or fail when asked if the video demonstrated a complete CVS. Krippendorff’s alpha coefficient and the intra-class correlation coefficient evaluated inter-rater reliability. Linear mixed effects models derived an average CW and FE rating using the CVS criteria for each video. Spearman correlation coefficients assessed the correlation between performance measures.
Results: A total of 617 CW generated 8,462 ratings across 160 videos in 24 hours. Five FE generated 200 ratings in 5 days, 6 hours. Intra-class correlation coefficient was 0.72 and Krippendorff’s alpha was 0.72 for the FE. Average CVS ratings per video were highly correlated between the CW and FE with a Spearman correlation coefficient of 0.89 (p<0.001). Average CVS ratings by CW were highly correlated with the probability of assigning a pass or fail rating with a Spearman correlation coefficient of 0.95 (p<0.001). Only 12.5% of the videos received a CVS ≥5 as rated by the FE compared to 0% as rated by CW.
Conclusions: There is a high degree of correlation between crowd workers and faculty experts in assessment of the critical view of safety. However, crowd workers did not identify videos with CVS ≥5 as rated by the faculty experts. Educating CW to assess surgical safety and quality is feasible and may broaden the application of this assessment and feedback tool.
Presented at the SAGES 2017 Annual Meeting in Houston, TX.
Abstract ID: 79185
Program Number: P005
Presentation Session: Poster of Distinction (Non CME)
Presentation Type: PDIST