Needle placement is a challenging problem for a variety of medical interventions, including brachytherapy or biopsy taylor2016medical . The force acting on the needle tip allows for inference about the currently penetrated tissue. This information can be used to navigate the needle and to prevent injuries of delicate structures okamura2004force . In order to distinguish tissue based on tip forces, it may be required to measure those with an accuracy of approximately Mccreery2008 . Tip forces cannot be measured with external sensors due to friction forces at the needle shaft kataoka2002measurement . Therefore, small-scale fiber-optical force estimation methods have been directly integrated into the needle tip. Several sensor concepts are based on Fabry-Pérot interferometry beekmans2017fiber and Fiber Bragg Gratings kumar2016detecting . Here, we consider a setting where optical coherence tomography is available, e.g., to study tissue deformation Otte2012 or to realize elastography Latus2017 . While OCT has been proposed for tip forces estimation before gessert2018needle ; gessert2019spatio , these approaches rely on the reconstructed gray value data. However, using the reconstructed data has two limitations. First, the signal processing is based on a number of assumptions which may cause some loss of signal information. Second, it does not incorporate the phase part of the complex OCT signal that is particularly sensitive to small axial shifts. Therefore, we explore whether the tip force estimation accuracy can be improved by directly using the raw spectral OCT data. Thus, we perform a calibration between the optical signal and forces applied to the needle tip with convolutional neural networks. We validate our approach with three different needles using a new, improved needle design.
2.1 Needle Design
We used an improved needle design for force estimation at the needle tip. A scheme and an image of the needle are shown in Figure 1. A brass tip with a piston is put on a brass sleeve such that it is able to perform a sliding motion inside of it. An epoxy layer between tip and sleeve acts as a spring. An optical fiber is embedded into a ceramic ferrule. The ferrule is positioned relative to the tip piston such that the light beam travels a distance of approximately through air until hitting the surface of the piston. For protection, the ferrule is embedded into a polymer tube which is glued to the brass sleeve. When forces act on the needle tip, the epoxy layer is compressed and the piston moves closer to the exit point of the laser beam which can be detected in the OCT signal. The diameter of the needle is .
In gessert2018needle the needle tip was constructed as a cone and attached to the needle shaft with a deformable epoxy layer. Thus, radial forces on the needle tip could easily tilt it. The improved piston construction has the advantage that it guides the tip in axial needle direction and prevents tilt. This contributes to a more reproducible signal, an important aspect in the calibration of the needle, and to the overall durability.
2.2 Calibration Data
We acquire calibration data for three custom build needles identical in construction. The data acquisition is performed similar to gessert2019spatio where a needle is driven against a flat surface with a stepper motor. The force in the axial direction of the needle is measured with a force sensor and recorded together with the associated raw OCT data. Approximately OCT-force pairs are collected for each needle for forces between and . We perform regular OCT data reconstruction, which includes the following steps:
Dechirping the data by resampling and interpolating to new sampling points, based on manufacturer specifications
Estimation of the DC spectrum using an exponential moving average with a damping coefficient of
Subtraction of the DC spectrum
Apodization by using a Hann window for filtering the spectral data
Fourier transform for mapping frequency values to spatial values
Selection of the absolute signal value as the final reconstructed intensity image
Throughout the reconstruction process, information can be lost due to the DC spectrum estimation strategy (2), the apodization (4) which eliminates high-frequency signal parts and the selection of the absolute signal value (6) as the final image. Figure 2 shows an excerpt of the collected data, both in raw (left) and reconstructed form (right).
The raw OCT signal has a size of where is the number of scans acquired over time. Reconstruction up to step (5) results in a complex signal which is also of size . Finally, by taking the absolute signal value, the A-Scan image sequence of size is obtain. This can be interpreted as a sequence of 1D depth images over time.
2.3 Deep Learning Architectures
We compare prediction performance for needle tip forces from both raw and reconstructed OCT data with different convolutional neural network (CNN) architectures. The considered architectures are variants of the ResNet resnet
, which is an extension of CNNs that enables better training through improved gradient flow. The ResNet models were originally built for 2D images with 2D convolutions. Here, the A-Scan data represents 1D images. Therefore, we adapt the architectures by replacing 2D convolutions by 1D convolutions. Furthermore, we replace the network’s original output layer for multi-class classification with a fully-connected layer with one output for needle force regression. We consider several ResNet variants, resembling different network sizes, the regular ResNet34 and ResNet18 architecture as well as a smaller architecture with 2 residual blocks and 6 convolutional layers, which we name ResNet6. All deep learning models are implemented in PyTorchpaszke2019pytorch
. Learning is performed over 150 epochs with a batch size of
and a learning rate of 0.005 using the Adam optimizer. As a loss function we use the mean squared error which is defined as
where is the ground-truth force and is the predicted force value. We use of the data as a hold-out validation set. We performed five training runs with different random seeds and averaged the individual results.
We report the mean absolute error (MAE) in mN between force predictions and force targets on the validation set. Additionally, we report inference times for the examined neural network architectures. All results are shown in Table 1. Figure 3 shows the relative differences in errors graphically. For most combinations of needle and model architecture the error for learning on raw OCT data is lower compared with learning from reconstructed OCT scans. Particularly for the ResNet34 architecture, the calibration performance improves for all needles.
|Needle 1||Needle 2||Needle 3||Inf. times|
4 Discussion and Conclusion
In this paper we address the calibration problem of OCT-based needle tip force estimation using new and improved sensor concept with a piston and a guiding sleeve. In contrast to a previous OCT-based concept gessert2018needle , the needle is not as sensitive to lateral forces by design while achieving similar calibration results with intensity data.
Furthermore, we study an approach for improving deep learning-based calibration performance even further. During OCT acquisition a spectral signal is obtained which is typically reconstructed to an intensity depth image. We illustrate that learning is possible in an end-to-end fashion, i.e., the process of image reconstruction can be avoided. Moreover, the results improve in six out of nine setups, indicating that there may be information lost in the original processing that can be used when training the network on the raw signal. The proposed approach allows for precise estimation of forces at the needle tip, which is particularly interesting for force based robotic needle placement. With typical robot control cycles of , a trade-off between accuracy and inference time must be balanced.
We find that learning forces from raw OCT data instead of reconstructed images work well, in particular, for larger deep learning models. For future work, our approach could be studied in more detail with additional deep learning methods and in different applications scenarios. Also, our approach could be extended to other applications where an imaging modality is used as a sensor signal, for example in the context of motion tracking with OCT or ultrasound.
Compliance with Ethical Standards
Funding: This work was partially supported by the TUHH initiative and DFG grants SCHL 1844/2-1 and SCHL 1844/2-2.
Conflict of Interest: The authors M. Gromniak, N. Gessert, T. Saathoff and A. Schlaefer declare that they have no conflict of interest.
Ethical Approval: This article does not contain any studies with human participants or animals.
Informed Consent: Not applicable.
- (1) Taylor, R.H., Menciassi, A., Fichtinger, G., Fiorini, P., Dario, P. (2016) Medical robotics and computer-integrated surgery. In: Springer handbook of robotics, pp. 1657–1684. Springer
- (2) Okamura, A.M., Simone, C., O’leary, M.D. (2004) Force modeling for needle insertion into soft tissue. IEEE transactions on biomedical engineering 51(10), 1707–1716
- (3) Mccreery, G., Trejos, A., Naish, M., Patel, R., Malthaner, R. (2008) Feasibility of locating tumours in lung via kinaesthetic feedback. The international journal of medical robotics and computer assisted surgery : MRCAS 4, 58–68. DOI 10.1002/rcs.169
- (4) Kataoka, H., Washio, T., Chinzei, K., Mizuhara, K., Simone, C., Okamura, A.M. (2002) Measurement of the tip and friction force acting on a needle during penetration. In: MICCAI, pp. 216–223. Springer
- (5) Beekmans, S., Lembrechts, T., van den Dobbelsteen, J., van Gerwen, D. (2017) Fiber-optic fabry-pérot interferometers for axial force sensing on the tip of a needle. Sensors 17(1), 38
- (6) Kumar, S., Shrikanth, V., Amrutur, B., Asokan, S., Bobji, M.S. (2016) Detecting stages of needle penetration into tissues through force estimation at needle tip using fiber bragg grating sensors. Journal of biomedical optics 21(12), 127,009
- (7) Otte, C., Hüttmann, G., Schlaefer, A. (2012) Feasibiliy of optical detection of soft tissue deformation during needle insertion. In: SPIE 8316, Medical Imaging 2012: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 8316, pp. 282 – 292
- (8) Latus, S., Otte, C., Schlüter, M., Rehra, J., Bizon, K., Schulz-Hildebrandt, H., Saathoff, T., Hüttmann, G., Schlaefer, A. (2017) An approach for needle based optical coherence elastography measurements. MICCAI 201: Medical Image Computing and Computer-Assisted Intervention pp. 655–663
- (9) Gessert, N., Priegnitz, T., Saathoff, T., Antoni, S.T., Meyer, D., Hamann, M.F., Jünemann, K.P., Otte, C., Schlaefer, A. (2018) Needle tip force estimation using an oct fiber and a fused convgru-cnn architecture. In: MICCAI, pp. 222–229. Springer
- (10) Gessert, N., Priegnitz, T., Saathoff, T., Antoni, S.T., Meyer, D., Hamann, M.F., Jünemann, K.P., Otte, C., Schlaefer, A. (2019) Spatio-temporal deep learning models for tip force estimation during needle insertion. Int J CARS 14, 1485–1493
- (11) He, K., Zhang, X., Ren, S., Sun, J. (2016) Deep residual learning for image recognition. pp. 770–778. DOI 10.1109/CVPR.2016.90
- (12) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S. (2019) Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc.