Comprehensive Proficiency-based Inanimate Training for Robotic Surgery: Reliability, Feasibility, and Educational Benefit

Nabeel A Arain, MD MBA, Genevieve Dulan, MD, Deborah C Hogg, BS, Robert V Rege, MD, Cathryn Powers, MD, Seifu T Tesfay, RN MS, Linda S Hynan, PhD, Daniel J Scott, MD. University of Texas Southwestern Medical Center

Introduction: We previously developed a comprehensive, proficiency-based robotic training curriculum that aimed to address 23 unique technical skills identified via task deconstruction of robotic operations. Prior studies supported construct, content, and face validity. The purpose of this study was to assess reliability, feasibility, and educational outcomes associated with curricular implementation.

Methods: Over an 11-month period, novice robotic trainees (n=55: 47 residents, 5 faculty, and 3 fellows) from general surgery (n=20), urology (n=18), and gynecology (n=17) were enrolled in a 2-month curriculum which included: 1) online-didactics, 2) half-day hands-on tutorial, and 3) self-practice (individually scheduled sessions) using 9-inanimate exercises until proficiency levels were achieved. Each trainee completed a questionnaire (5-point Likert scale) and performed a single proctored repetition of each task before (pre-test) and after (post-test) training. Training was supervised by a designated proctor and feedback was given accordingly. Five exercises used FLS models with modifications (peg, clutch/camera peg, pattern-cut, and interrupted and running suture) and 4 used custom-made components (rubber-band, stair-rubber-band, clutch/camera, and cutting-rubber-band). Each task was scored for time and errors using modified FLS metrics; task scores were normalized to proficiency levels and a composite score equaled the sum of the 9 normalized task scores. For inter-rater reliability (IRR; degree of agreement among raters), 3 trainees (3 repetitions per task) were scored by two trained raters on all 9 tasks and intra-class correlation coefficients (ICC; measure of reliability) were analyzed. Data from 8 experts was analyzed using ICC and Cronbach’s alpha to determine test/retest reliability and internal consistency, respectively. Educational benefit was assessed by comparing baseline (pre-test) and final (post-test) trainee performance; comparisons used Wilcoxon signed rank test.

Results: Of 55 trainees that pre-tested, 53 (96%) completed all curricular components in 9.0 to 16.8 hours. All 53 trainees reached proficiency for all 9 exercises after completing an average of 71.6 ± 28.2 repetitions over 5.0 ± 1.4 hours; practice was completed during 3-8 self-practice sessions with relatively extensive (4.8 ± 0.5) proctor feedback. Proficiency levels were rated as moderately difficult (2.9 ± 0.9), highly appropriate (4.5 ± 0.7), and provided excellent feedback (98% agreement amongst trainees). While trainees felt moderately to very comfortable with their laparoscopic skills (3.4 ± 0.9), they had minimal prior robotic experience and poor comfort with robotic skills (1.8 ± 0.9) at baseline (pre-test) compared to final (post-test) (3.1 ± 0.8, p<0.001). Analysis of IRR data for the composite score revealed an ICC of 0.96 [95% confidence interval (CI), 0.71-1.00; p<0.001]. Test/retest reliability was 0.91 [95% CI, 0.40-0.91; p<0.001] and internal consistency was 0.81. Performance improved after training for all 9 tasks and according to composite scores (548 ± 176 vs. 914 ± 81, p<0.001) demonstrating significant educational benefit. 100% of trainees indicated that their robotic skills improved.

Conclusion: This curriculum is associated with high reliability measures, was feasible for a large number of trainees, and resulted in significant performance improvement. Further studies and adoption are encouraged.