Robust Augmented Reality registration method for Localization of Solid Organs’ Tumors Using CT-derived Virtual Biomechanical Model and Fluorescent Fiducials

Seong-Ho Kong, MD, PhD1, Nazim Haouchine, PhD2, Renato Soares, MD3, Bruno Marques2, Michele Diana, MD3, Galyna Shabat, MD, PhD4, Bohdan Andreiuk5, Andrey Klymchenko, PhD5, Thierry Piechaud, MD, PhD6, Stéphane Cotin, PhD2, Jacques Marescaux, MD, FACS, Hon, FRCS, Hon, FJSES, Hon, APSA4. 1IHU-Strasbourg, Institute for Image-Guided Surgery, Strasbourg, France and Seoul National University Hospital, Seoul, Korea, 2INRIA Mimesis and IHU-Strasbourg, Institute for Image-Guided Surgery, Strasbourg, France, 3IHU-

1. Objective of the technology or device

Accurate localization of solid organs tumors is crucial to ensure both radicality and organ function preservation. Augmented Reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g. CT-scan). The virtual model can be superimposed to the real-time images to obtain the enhanced real-time localization. However, the 3D virtual model is rigid, and does not take into account inner structures’ deformations. We present a concept of automated navigation system, enabling transparency visualization of internal anatomy and tumor’s margins, while the organs undergo deformation during breathing or surgical manipulation.

2. Description of the technology and method of its use or application

We have developed a real-time biomechanical model based on a finite element modelling (FEM) method. FEM allows simulating soft-tissue behavior, taking into account mechanical properties of parenchyma, vessels and tumors, obtained by available database (SOFA engine). Such mechanical properties are applied to the patient-specific virtual reality model, enabling FEM to predict inner deformations, by propagation of the observed surface changes.

Re-localization during deformation needs real-time tracking. To ensure robust re-localization, we have opted for a fiducials-based tracking method. Metal fiducials were coated by polymeric near-infrared fluorescent dye. Fluorescence coating enables easier visualization of miniature, less invasive, fiducials. For the proof of the concept, we have placed a pseudo-tumor (1x10mm metal wire) and several fiducials in porcine kidneys (ex-vivo and in-vivo by ultrasound-guided percutaneous approach). A CT scan was performed for both ex-vivo and in-vivo generation of the virtual model, including fiducials. The FEM integrating biomechanical properties was applied to the model, which was finally displayed in real-time on the laparoscopic surgical field, and virtual and real fiducials registered automatically.

3. Preliminary results

The ex-vivo experiment showed a successful segmentation of the kidney including fiducials. The fluorescent signal permitted a correct detection of the fiducials on the laparoscopic images.  By progressive software-based removal of fiducials-point, we could determine that a minimum of 4 fiducials (from the initial 12) are necessary to yield a good registration. The measured distance between the estimated tumor by biomechanical propagation and the scanned tumor as a ground-truth was 2 mm.

Similarly, in the in-vivo experiment, 7 fiducials were successfully placed and were all detected by a laparoscopic fluorescent camera. Segmentation was successful; however, a pre-processing segmentation step was required to eliminate scanning artefacts issued from metal fiducials. Four fiducials were necessary for the registration, and the measured error was about 5%.

4. Conclusions / future directions

Our preliminary experiments showed a potential of a biomechanical model with fluorescent fiducials to propagate solid organs’ surface deformation to its inner structures including tumors. Future works will focus on less invasive fiducials that will facilitate a safe translation to human operation.


« Return to SAGES 2016 abstract archive