We propose a colonoscope tracking method using a shape estimation network. CT image-based diagnosis of the colon is performed as one of less invasive diagnosis methods. It is called as CT colonography. If colonic polyps or early-stage cancers are found in diagnosis, a colonoscopic examination or polypectomy is performed. In a colonoscopic examination, a physician explores in the colon to find polyps. However, understanding where the colonoscope running in the colon is difficult. A colonoscope navigation system that navigates a physician to polyp positions during a colonoscope insertion is necessary. To achieve a colonoscope navigation system, a colonoscope tracking method that finds the colonoscope position in the colon needs to be developed.
Colonoscope tracking methods were proposed by some research groups. Image-based method is difficult to keep tracking when unclear colonoscopic views appear. Electromagnetic (EM) sensor will be necessary to perform tracking. A system that shows colonoscope shape in the colon using EM sensors have been developed[2, 3]. The system cannot perform navigation because it cannot map the colonoscope shape to the colon shape of a patient. A colonoscope tracking method that utilizes a CT volume and EM sensors was proposed. This method obtains two curved lines representing colonoscope and colon shapes to estimate a colonoscope position on a CT volume coordinate system. However, large tracking errors were caused due to colon deformations occurred during the colonoscope insertions. To reduce this problem, estimation methods of the colon shape were proposed[5, 7]. The shape estimation network (SEN)
accurately estimates colon shape utilizing temporal information. The SEN performs estimation using a neural network containing long short-term memory (LSTM) layer. Because the SEN employs a machine learning framework, it needs to be trained using data (colon and colonoscope shapes) obtained during colonoscope operations. However, the SEN was trained using data measured during operations of a colonoscope by an engineering researcher. The colonoscope operations were different from that of performed by physicians.
In this paper, we propose a colonoscope tracking method. We utilize EM sensors attached to the colonoscope and a CT volume of a patient. Colonoscope and colon shapes are obtained from them. The SEN estimates a colon shape that are deformed by a colonoscope inserted to it. To perform colon shape estimation based on real physician’s data, we trained the SEN using data obtained during physician’s colonoscope operations. We perform colonoscope position estimation utilizing the colonoscope shape and the estimated colon shape obtained from the trained SEN.
2 Colonoscope Tracking Method
We obtain two shapes representing a colonoscope and a colon. The colonoscope and colon shapes are registered to find a position in the CT volume coordinate system that corresponds to the colonoscope tip position. The colon shape changes during colonoscope insertions. We use the SEN to estimate deformed colon shapes. Correspondences of points on the colonoscope shape and the estimated colon shape are established based on the deformation estimation results. A colonoscope tip position in the CT volume coordinate system is calculated from the correspondence.
2.2 Colonoscope and colon shapes
We represent the colonoscope and colon shapes as sets of points. The colonoscope shape is a set of points and directions aligned along a colonoscope. is the total number of the points in the colonoscope shape. The colon shape is a set of points aligned along a colon. is the total number of the points in the colon shape. Examples of them are found in Fig. 1.
2.3 Colon shape estimation using SEN
We use the SEN to estimate colon shapes which are deformed during colonoscope insertions. The SEN is a neural network containing convolution and LSTM layers. Its input is a set of the time-series colonoscope shapes. It outputs an estimated colon shape (Fig. 1).
In the previous work, the SEN was trained using colonoscope and colon shapes measured during colonoscope operations performed by an engineering researcher. We measured colonoscope and colon shapes during colonoscope operations by physicians. The physician’s data are used to train the SEN. Measurement of the shapes are explained in the Sec. 3.2.
2.4 Colonoscope tip position calculation
A point in the colonoscope shape represents the colonoscope tip position. We find a point in the estimated colon shape which is closest to . We find a point in the colon shape that corresponds to . Each point in the colon and estimated colon shapes have one-to-one correspondence. As a result of these two-steps mappings, we obtain as a point in the CT volume coordinate system where the colonoscope tip located.
We evaluated the proposed colonoscope tracking method in phantom-based experiments. We use a colon phantom (colonoscopy training model type I-B, Koken, Tokyo, Japan), a CT volume of the phantom, a colonoscope (CF-Q260AI, Olympus, Tokyo, Japan), an EM sensor (Aurora 5/6 DOF Shape Tool Type 1, NDI, Ontario, Canada), and a distance image sensor (Kinect v2, Microsoft, WA, USA). Experimental materials are shown in Fig. 2 (a).
3.2 Colonoscope and colon shapes measurement
The colonoscope shape is measured using the EM sensor. The EM sensor is strap-shaped with six sensors at its tip and points along its body. It is inserted into the colonoscope working channel. The colonoscope shape is a set of six points.
The colon shape is measured using the distance image sensor. We obtain 12 three-dimensional points on the colon phantom using the sensor. The points are aligned along the colon centerline. The colon shape is a set of the 12 points.
Both of the colonoscope and colon shapes are registered to the CT volume coordinate system. The colonoscope shape is registered using the iterative closest point (ICP) algorithm by minimizing distance between the colonoscope shape and a point set on the colon centerline in the CT volume. The colon shape is registered manually.
To obtain training data of the SEN, we measured the colonoscope and colon shapes during colonoscope insertions to the colon phantom. Operators moved the colonoscope tip from the cecum to anus. 10 times of colonoscope movements from the cecum to anus were measured in our experiments. Two physicians (four times colonoscope movements) in the Nagoya University Hospital and an engineering researcher (six time colonoscope movements) were operated. The SEN was trained using the measured data. Figure 3 shows the configuration of experimental setup for measurement of physician’s colonoscope operation.
3.3 Definition of tracking error
We placed 12 markers on the surface of the colon phantom (Fig. 2 (b)). The markers are identifiable points both on the colon phantom and in its CT volume. We employed tracking error as a criterion of tracking performance evaluation. The tracking error is the distance between an estimated colonoscope tip position and a marker position in the CT volume, when the colonoscope tip comes closest to the marker.
3.4 Tracking and evaluation
In colonoscopic examinations, physicians observe the colon while retracting the colonoscope after its insertion up to the cecum. We assume the colonoscope tip is inserted up to the cecum when the colonoscope tracking starts. The colonoscope tracking is performed while retracting the colonoscope from the cecum to the anus.
We performed colonoscope tracking five times. An engineering researcher operated the colonoscope.
Average, maximum, and minimum tracking errors of five times colonoscope tracking are shown in Fig. 4. Tracking errors were small in the ascending, transverse, and descending colons. Large tracking errors were observed in the sigmoid colon and rectum.
A physician who specializes in gastroenterology commented that tracking errors smaller than 50 mm are acceptable for colonoscope navigation to polyp positions. This is because polyps are observable from the colonoscope camera when positions of the colonoscope tip and polyps are closer than 50 mm. The proposed method achieved average tracking errors smaller than 50 mm in the ascending, transverse, and descending colons. The previous method achieved it in the ascending and descending colons (under setting of mm). In the transverse colon, smaller tracking errors were observed in the proposed method. The previous method does not consider colon deformation. It makes tracking errors in the transverse colon large because the transverse colon is easy to deform. Consideration of colon deformation strongly contributed to reduce tracking errors.
The tracking errors in the sigmoid colon and rectum were large. This is because the sigmoid colon and rectum have heavy deformation during the colonoscope insertions. The deformation contains considerable variation. The data used to train the SEN was not enough to provide information about deformation in these colon regions. We need to correct data that containing various deformations of the sigmoid and rectum to improve both of the colon shape estimation and tracking accuracies.
We proposed a colonoscope tracking method using the SEN. Previous tracking method caused large tracking errors due to colon deformations. In our method, the SEN estimates deformed colon shape. The SEN was trained using data obtained during colonoscope operations by physicians. The estimated colon shape is used to make correspondence between the colonoscope tip position and its position in the colon. In our phantom study, tracking errors in the transverse colon were reduced by considering colon deformations. Future work includes evaluation using more markers, reduction of tracking error in the sigmoid and rectum, and application to tracking in human colon.
Acknowledgements.Parts of this research were supported by the MEXT/JSPS KAKENHI Grant Numbers 26108006, 17H00867, the JSPS Bilateral International Collaboration Grants, and the JST ACT-I (JPMJPR16U9).
-  (1992) A method for registration of 3-D shapes. Vol. 14, pp. 239–256. Cited by: §3.2.
-  (2010) Non-radiological colonoscope tracking image guided colonoscopy using commercially available electromagnetic tracking system. Cited by: §1.
-  (2015) Clinical impact of endoscopy position detecting unit (UPD-3) for a non-sedated colonoscopy. Vol. 21, pp. 4903–4910. Cited by: §1.
-  (2013) An optical flow approach to tracking colonoscopy video. Vol. 37, pp. 207–223. Cited by: §1.
-  (2018) Machine learning-based colon deformation estimation method for colonoscope tracking. Vol. 10576, pp. 1057619–1–1057619–6. Cited by: §1.
-  (2017) Robust colonoscope tracking method for colon deformations utilizing coarse-to-fine correspondence findings. Vol. 12, pp. 39–50. Cited by: §1, §5.
Colon shape estimation method for colonoscope tracking using recurrent neural networks. Note: (accepted) Cited by: §1, Figure 1, §2.1, §2.3, §2.3, §3.2.