Creation of an Operative Robotic Index to Evaluate Novice Robotic Surgeons Using a Combination of Objective Measurements and CUSUM Analyses

William B Lyman, MD¹, Michael Passeri, MD¹, Imran A Siddiqui, MD, FACS², Adeel S Khan, MD, MPH, FACS³, David A Iannitti, MD, FACS¹, John B Martinie, MD, FACS¹, Erin H Baker, MD, FACS¹, Dionisios Vrochides, MD, PhD, FACS¹. ¹Carolinas Medical Center, ²St. Vincent’s Medical Center, ³Washington University in St. Louis

Background: With the growing popularity of robotic-assisted surgery, new methods for evaluation of technical skill are necessary to determine when a surgeon is qualified to perform an operation independently. Current evaluation methods are limited to 5 point Likert scales which require a degree of subjective scoring. Surgeons in training need an objective method of evaluation to view progress and target areas for improvement. One method of objectively evaluating surgical performance is a cumulative sum control chart (CUSUM). By plotting consecutive operative outcomes on a CUSUM chart, surgeons can view their learning curve for a given task. Another method of objective evaluation is the dV Logger®, or “Black Box,” which records objective measurements directly from the da Vinci® system.

Methods: We followed two HPB fellows during dry lab simulation of 40 robotic-assisted hepaticojejunostomy reconstructions using biotissues to model a portion of a Whipple procedure. We simultaneously recorded objective measurements of dexterity from the da Vinci® system and performed CUSUM analyses for each procedural step. We modeled each variable using machine learning (a self-correcting and autoregressive modeling tool) to reflect the fellows’ learning curves for each task. Statistically significant objective variables were then combined into a single formula to create an Operative Robotic Index (ORI).

Results: Variables that significantly improved over the course of the simulation included completion time (p=0.017), economy of motion in arm 1 (p=0.001), number of times head was removed from the console (p=0.001), total time left master manipulator was active (p=0.005), total time right master manipulator was active (p<0.001), and total time that any arm was active (p<0.001). The inflection points of our CUSUM charts and plots of objective variables both showed improvement in technical performance beginning between trials 14 and 16 [Figure 1 and Figure 2]. The Operative Robotic Index showed a strong fit to our observed data and improved with additional trials (R2=0.796). [Figure 3].

Conclusions: In this study we identified objective variables recorded by the da Vinci® system which correlated with the technical dexterity of fellows during a robotics dry lab. We broke a complex procedure down in stepwise fashion with CUSUM analyses to determine targets for improvement. Using variables which correlated with the improved performance of the fellows, we effectively modeled the learning curve with the creation of an Operative Robotics Index (ORI). This study successfully models the learning curve of novice robotic surgeons using a novel combination of objective measures.