A Novel Video-Based Performance Rating System for Asynchronous Pinpoint Feedback Loops, Reduced Cycle Time, and Improved Patient Outcomes

Vivek Agrawal, MS¹, Jean-Sebastien Legare, MSc¹, Daniel Naab, BS¹, Justin P Wagner, MD², Juan Espinosa, MD³, Patrick J Colp, MSc¹, David C Chen, MD². ¹Comment Bubble, Inc., ²David Geffen School of Medicine at University of California, Los Angeles, ³Children’s Hospital Los Angeles

Objective

As video recording in the operating room has become more prevalent, there is significant potential to revolutionize clinical training and improve patient outcomes.

The landmark study of patient outcome and surgical proficiency assessed using video by Brickermeyer, et al. (N Engl J Med 2013; 369:1434-1442) showed a strong correlation. The bottom quartile of surgical skill was associated with higher complication rates (14.5% vs. 5.2%, P<0.001) and higher mortality (0.26% vs. 0.05%, P=0.01) than the top quartile, suggesting peer rating as an effective strategy for assessing surgical proficiency towards improved mortality rates.

The American Board of Surgery (ABS) recommends the Operative Performance Rating System (OPRS) as a model evaluation system, but as a paper-based tool, there are limitations. We propose a novel web-based analytics tool for pinpoint quantitative scoring and pinpoint qualitative feedback integrating video into the OPRS model.

By reducing cycle time between surgery and feedback, and providing a pinpoint analytics platform that can be superimposed on any video, a new standard can be established for assessment and feedback of operative performance for surgical residents and, potentially even practicing surgeons, resulting in higher satisfaction and improved outcomes.

Soon, surgeons will be able to review their performance in the operating room and adjust just like a professional athlete can review their performance and adjust after every game.

Description

Through collaboration between Comment Bubble, Inc., a Silicon Valley-based company at the forefront of video content analytics, and UCLA, we have designed web technology based on the patent-pending CommentBubble.com platform. Multiple feedback types include pinpoint numerical ratings, textual feedback, video feedback, and audio feedback tagged to moment of assessment in the video, adding new levels of reliability and value to the OPRS model.

Integrated analytical tools include interactive user interface with graphs of feedback vs. time, display of comments, databases, and reports. Chapter markers are created such that numerical assessments of procedure-specific criteria will be valid, along with global assessment metrics as designed in the conventional OPRS.

We focus on operative evaluation and skill acquisition in surgical residency, but it is applicable for all aspects of competency-based learning from patient interactions to skills assessment.

Preliminary results

CommentBubble.com was released in 2014 as a qualitative video analytics tool and is used worldwide in secondary and post-secondary education. Additionally, it is being used to coach and provide expert feedback. Students and faculty find the interface convenient, the analytics insightful, and the ability to automatically cue the video to the moment of assessment profound. The medical video OPRS software in development is building on these best practices.

Future directions

With the beta release of the Comment Bubble video OPRS in February 2015, metrics will be established based on ABS-approved criteria and residents will be assessed and given feedback. The satisfaction will be surveyed and compared to existing evaluation systems, along with time savings. Additionally, we will explore asynchronous international surgical collaboration and training with this tool. Long-term studies will include comparison of patient outcomes before and after using Comment Bubble.