3D structure reconstruction of lesions in colon with colonoscopic images

Atul Kumar¹, Shih-Wei Huang, Dr², Yen-Yu Wang¹, Kai-Che Liu¹, Ching-Chun Huang¹, Yi-Chun Lee¹, Jing-Jim Ou², Wan-Chi Hung¹. ¹Chang Bing Show Chwan Memorial Hospital, IRCAD-Taiwan, ²Chang Bing Show Chwan Memorial Hospital

Objective: Conventional colonoscopy can only provide a 2D image of a lesion; however, a 3D detailed morphology of the lesions would be helpful to understand the type of lesion and its extension. This study aims towards developing a system that would help in visualizing a dense 3D structure of colon segment and lesion associated with it.

Description of the technology and method of its use (Fig 1 to 4): A shape from shading (SfS) algorithm is applied to each frame of the colonoscopic video to generate a 3D shape from the image. The algorithm first calculates the direction of the illumination in the image and then finds the depth for each pixel coordinate in the image by solving a non-linear equation. Characteristic feature points in the first frame (F1 ) of the video sequence and the subsequent frame (Fn ) are detected using the speeded-up robust feature (SURF) detection algorithm. The detected interest points are assigned feature point descriptor known as binary string feature point descriptor (BRIEF) which is based upon the intensity differences among the pixels near the interest points. For matching the descriptors, a nearest neighbor search is used with Hamming distance measurements in the descriptor space. RANdom Sample Consensus (RANSAC) is applied to remove the outliers. The pixel positions of the points of correspondence (between two consecutive images calculated by the feature matching) provide their 2D coordinates (X, Y). Their Z co-ordinates are retrieved from their respective 3D surfaces from SfS. A rigid body transformation matrix between these matched points is calculated using a scaled ICP (Iterative Closest Point) algorithm. The matrix is used to register the 3D surfaces from consecutive video frames. Hence, all the 3D surfaces are stitched together. The pixel coordinates in 2D image and its corresponding depth calculated with SfS provided the x, y and z point coordinates of each pixel in the image. A Delaunay triangulation technique is used to make a 3D surface from the point coordinates. The points on the surface are mapped and interpolated with the color of the respective pixel in 2D image. The final 3D surface is visualized in a 3D render window.

Preliminary results: The complete method was applied using 100 frames of a colonoscopic video from a phantom. A 3D structure of the lesion and adjoining colon was reconstructed using the method and visualized in a render window. The results is shown in Fig.4.

Conclusions / future directions: The current study presents a method for reconstructing 3D structure of lesions from colonoscopic images. SfS method does not provide a metric structure; therefore, the future study would include information from structure from motion to create a metric scaled structure. The major challenge in the method is to identify characteristic feature points in the colonoscopic images. In future works, feature detection method would be improved further.

Endoscope image of phantom