Elif Bilgic1, Mohammed Al Mahroos1, Tara Landry2, Gerald M Fried1, Melina C Vassiliou1, Liane S Feldman1. 1Mcgill University, 2Montreal General Hospital Medical Library
Introduction: Operative skills have been shown to correlate with patient outcomes, yet at the completion of training or after learning a new procedure, these skills are not formally evaluated. SAGES and others are interested in using video of procedures for summative assessment of laparoscopic foregut surgery (LFS). As this may be used to determine competency, it must meet the robust criteria established for “high-stakes” assessments. The purpose of this review is to identify tools that have been used to assess LFS and evaluate the available validity evidence for each instrument.
Methods: A systematic search was conducted up to July 2017. Eligible studies reported data on tools used to assess performance in the operating-room during LFS. Two independent reviewers considered 1084 citations for eligibility. The characteristics of each assessment tool were recorded. Validity evidence was evaluated using 5 sources of validity (content, response process, internal structure, relationship to other variables, and consequences). Evidence was summarized by providing a score of 0-3 for each source, for a maximum score of 15(1-5 limited evidence, 6-10 moderate, 11-15 strong).
Results: Overall, 10 separate tools were identified, all evaluating technical skills. Four tools were generic to laparoscopy, 1 was specific to laparoscopic suturing(LS), and 5 specific to LFS (2 specific to nissen fundoplication(NF), 1 heller myotomy (HM), 1 paraesophageal hernia repair (PEH), and 1 LS during NF. Overall, only 2 assessments were supported by moderate evidence (one specific to LS in NF and one generic), while the others had limited or unknown evidence. Validity evidence was based mainly on internal structure (8 tools reporting reliability and item analysis) and relationship to other variables (6 comparing scores in subjects with different clinical experience). There was little or no evidence supporting test content (3 studies referencing previous papers for tool development in the context of clinical assessment, regarding laparoscopy, NF, and LS in NF), test response process (2 studies reporting rater training), and consequences (no studies). Six tools were identified to have evidence for video-assessment; 2 generic, 2 specific to NF, 1 to LS, and 1 to LS in NF. Only the tool specific for LS in NF had moderate evidence, while others had limited evidence.
Conclusion: There is limited evidence supporting the validity of intraoperative assessment tools for laparoscopic forgut surgery. This precludes their use for summative video-based assessment to verify competency. Further research is needed to develop an assessment tool for this purpose.
Presented at the SAGES 2017 Annual Meeting in Houston, TX.
Abstract ID: 85167
Program Number: S082
Presentation Session: MIS Education Session
Presentation Type: Podium