• Skip to primary navigation
  • Skip to main content
  • Skip to footer

SAGES

Reimagining surgical care for a healthier world

  • Home
    • Search
    • SAGES Home
    • SAGES Foundation Home
  • About
    • Who is SAGES?
    • SAGES Mission Statement
    • Advocacy
    • Strategic Plan, 2020-2023
    • Committees
      • Request to Join a SAGES Committee
      • SAGES Board of Governors
      • Officers and Representatives of the Society
      • Committee Chairs and Co-Chairs
      • Full Committee Rosters
      • SAGES Past Presidents
    • Donate to the SAGES Foundation
    • SAGES Store
    • Awards
      • George Berci Award
      • Pioneer in Surgical Endoscopy
      • Excellence In Clinical Care
      • International Ambassador
      • IRCAD Visiting Fellowship
      • Social Justice and Health Equity
      • Excellence in Community Surgery
      • Distinguished Service
      • Early Career Researcher
      • Researcher in Training
      • Jeff Ponsky Master Educator
      • Excellence in Medical Leadership
      • Barbara Berci Memorial Award
      • Brandeis Scholarship
      • Advocacy Summit
      • RAFT Annual Meeting Abstract Contest and Awards
    • “Unofficial” Logo Products
  • Meetings
    • NBT Innovation Weekend
    • SAGES Annual Meeting
      • 2024 Scientific Session Call For Abstracts
      • 2024 Emerging Technology Call For Abstracts
    • CME Claim Form
    • Industry
      • Advertising Opportunities
    • Future Meetings
    • Related Meetings Calendar
  • Join SAGES!
    • Membership Benefits
    • Membership Applications
      • Active Membership
      • Affiliate Membership
      • Associate Active Membership
      • Candidate Membership
      • International Membership
      • Medical Student Membership
    • Member News
      • Member Spotlight
      • Give the Gift of SAGES Membership
  • Patients
    • Healthy Sooner – Patient Information for Minimally Invasive Surgery
    • Patient Information Brochures
    • Choosing Wisely – An Initiative of the ABIM Foundation
    • All in the Recovery: Colorectal Cancer Alliance
    • Find a SAGES Member
  • Publications
    • SAGES Stories Podcast
    • SAGES Clinical / Practice / Training Guidelines, Statements, and Standards of Practice
    • Patient Information Brochures
    • TAVAC – Technology and Value Assessments
    • Surgical Endoscopy and Other Journal Information
    • SAGES Manuals
    • SCOPE – The SAGES Newsletter
    • COVID-19 Annoucements
    • Troubleshooting Guides
  • Education
    • Wellness Resources – You Are Not Alone
    • OpiVoid.org
    • SAGES.TV Video Library
    • Safe Cholecystectomy Program
      • Safe Cholecystectomy Didactic Modules
    • Masters Program
      • SAGES Facebook Program Collaboratives
      • Acute Care Surgery
      • Bariatric
      • Biliary
      • Colorectal
      • Flexible Endoscopy (upper or lower)
      • Foregut
      • Hernia
      • Robotics
    • Educational Opportunities
    • HPB/Solid Organ Program
    • Courses for Residents
      • Advanced Courses
      • Basic Courses
    • Fellows Career Development Course
    • Robotics Fellows Course
    • MIS Fellows Course
    • Facebook Livestreams
    • Free Webinars For Residents
    • SMART Enhanced Recovery Program
    • SAGES OR SAFETY Video
    • SAGES Top 21 MIS Procedures
    • SAGES Pearls
    • SAGES Flexible Endoscopy 101
    • SAGES Tips & Tricks of the Top 21
  • Opportunities
    • NEW-Area of Concentrated Training Seal (ACT)-Advanced Flexible Endoscopy
    • SAGES Fellowship Certification for Advanced GI MIS and Comprehensive Flexible Endoscopy
    • Multi-Society Foregut Fellowship Certification
    • SAGES Research Opportunities
    • Fundamentals of Laparoscopic Surgery
    • Fundamentals of Endoscopic Surgery
    • Fundamental Use of Surgical Energy
    • Job Board
    • SAGES Go Global: Global Affairs and Humanitarian Efforts
  • Search
    • Search All SAGES Content
    • Search SAGES Guidelines
    • Search the Video Library
    • Search the Image Library
    • Search the Abstracts Archive
  • OWLS
  • Log In

LapTool-Net: A Context-Aware Deep Learning System for Automatic Detection of Surgical Tools in Laparoscopic Videos

Babak Namazi, MS1, Katerina 0 Wells, MD2, Steven G Leeds, MD2, Ganesh Sankaranarayanan, PhD2, Venkat Deverajan, PhD1, James W Fleshman, MD2. 1University of Texas at Arlington, 2Baylor University Medical Center

Objective of the Technology: Monitoring the usage of different surgical instruments in a laparoscopic procedure is a critical part of automated systems for tracking surgical actions. The objective of our technology is to develop the LapTool-NetTM, a deep learning based system for automated detection of surgical tools in a laparoscopic procedure.

Description of the Technology and its Application: LapTool-Net is a multi-label classifier that was designed to detect the presence of surgical tools in a laparoscopic video. It is a deep CNN-RNN (Convolutional Neural Network – Recurrent Neural Network) model, where the CNN is a deep feed-forward artificial neural network that is trained to learn high-level features in still frames of the video, while the RNN is trained using a sequence of frames from the videos to learn the temporal features. The unique aspect of the LapTool-Net is the awareness of the context of the usage of the surgical tools, for example, the low probability of the co-occurrence of a laparoscopic shear and a clip applicator, thereby reducing the number of combinations of tools for detection. A separate RNN was trained to model the specific pattern of usage of the tools after training the CNN-RNN model, based on the sequential order of the surgical tasks.

The LapTool-Net can be used offline for the assessment of the recorded videos, information retrieval for education and operative summary report generation. It can also be used  in real-time for monitoring surgical actions to prevent errors and provide instantaneous feedback for quality improvement.

Preliminary Results: We used the publicly available M2CAI tool detection challenge dataset, which contains 15 videos of laparoscopic cholecystectomy surgeries. Images from the videos (one frame per second) were extracted and manually labeled for seven tools; bipolar, clipper, grasper, hook, irrigator, shears, and specimen bag. The training set contained 10 videos that were used to train the CNN-RNN model based on an inception-V1 and Gated Recurrent Unit (GRU) architectures. The other 5 videos were used for validating the performance of the model. In order to address the class imbalance due to the high variations in the time that each tool is being used, we employed the under-sampling method. Based on our knowledge of the co-occurrence of tools, the number of combinations for detection was reduced from 128 to 15. Frame level accuracy was the metric for evaluation, which considers a frame as correct only if all of the tools in the frame are correctly predicted. Our current accuracy is 80.96% and the average per-class F1-score (the harmonic average of the precision and recall) is 89.01%, which is by far the best performance reported for any automated tool detection system on the M2CAI dataset. The processing time for each frame is about 0.01 seconds, which makes it suitable for real-time applications.

Future Directions: The LapTool-Net is based on a supervised learning method. In the future, we plan to develop a semi-supervised learning method using a larger training dataset. In addition, we will test the performance of the proposed system on different laparoscopic procedures. 


Presented at the SAGES 2017 Annual Meeting in Houston, TX.

Abstract ID: 98856

Program Number: ETP725

Presentation Session: Emerging Technology Poster Session (Non CME)

Presentation Type: Poster

View this Poster

231

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Pinterest
  • WhatsApp
  • Reddit

Related



Hours & Info

11300 West Olympic Blvd, Suite 600
Los Angeles, CA 90064
1-310-437-0544
[email protected]
Monday - Friday
8am to 5pm Pacific Time

Find Us Around the Web!

  • Facebook
  • Twitter
  • YouTube
  • Instagram
  • TikTok

Important Links

SAGES 2024 Meeting Information

Healthy Sooner: Patient Information

SAGES Guidelines, Statements, & Standards of Practice

SAGES Manuals

 

  • taTME Study Info
  • Foundation
  • SAGES.TV
  • MyCME
  • Educational Activities

Copyright © 2023 Society of American Gastrointestinal and Endoscopic Surgeons