Raj Shekhar, Xinyang Liu, Emmanuel Wilson, Sukryool Kang, Mikael Petrosyan, Timothy D Kane. Children’s National Health System
Objectives: Conventional laparoscopic visualization is often accompanied with a flat representation of the 3D anatomy and limited visualization of structures located beneath visible organ surfaces (e.g., blood vessels, bile ducts, tumors). To address these limitations, we have developed a novel augmented reality (AR) visualization technology that merges real-time laparoscopic ultrasound (LUS) images with live stereoscopic laparoscopic video. This study constitutes initial clinical application of the developed AR system for the purposes of demonstrating technical feasibility and clinical benefits, and obtaining clinically relevant feedback to guide future development.
Methods: The AR system includes (1) a stereoscopic vision system for visualizing surgical anatomy with true depth; (2) a laparoscopic ultrasound scanner, which enables visualization of critical internal structures; (3) an optical tracking system, which tracks in real time the two imaging devices by tracking optical markers affixed to custom-designed fixtures; and (4) a computer workstation. With IRB approval, the clinical testing of the AR system in patients undergoing laparoscopic cholecystectomy continues and 7 patients (ages 4-17, 1 male, 6 females) have been recruited to date. Fig. 1 shows the setup of the AR system in the OR. The AR system calibration, performed up to a day in advance, preceded each clinical use. After calibration, the laparoscope, the LUS probe, and the tracking fixtures were sent for sterilization. A specialist assembled the sterilized items in the OR at the beginning of the surgery and attached 4 pre-sterilized optical markers on each of the 2 fixtures. Fig. 2 shows the surgeons, who wear polarized 3D glasses to perceive depth, using the AR system to visualize the hepatobiliary anatomy. The AR visualization was performed for up to 5 min prior to starting the actual surgery. When instructed, the system records live LUS images, stereoscopic laparoscopic images, and stereoscopic AR images as digital videos.
.png)
Results: The ongoing clinical testing has been a learning process for the project team to successfully use the AR system in the OR. During the first 3 cases, several technical and practical issues prevented obtaining well-registered AR images. With these issues resolved, the AR system has worked successfully in the following 4 cases. On average, the system took 15 min to set up in the OR, and this setup took place in parallel with patient setup in conventional laparoscopic surgery. The use of the AR system added approximately 5 min to the clinical procedure. The AR system was considered easy to use and helpful in the visualization of normally unseen structures. Fig. 3 shows the system’s ability to visualize the hepatic artery and the cystic duct in the conduct of laparoscopic cholecystectomy. The system was also used to visualize the hepatic vasculature and the pancreatic duct.
.png)
Conclusions: We have successfully translated the AR system from the laboratory to the operating room. The transformative clinical application envisioned for the use of this technology will be in aiding visualization and accurate resection of subparenchymal lesions (liver, kidney, lung) by minimally invasive approaches.