Needle tip force estimation by deep learning from raw spectral OCT data

06/30/2020 ∙ by M. Gromniak, et al. ∙ 0

Purpose. Needle placement is a challenging problem for applications such as biopsy or brachytherapy. Tip force sensing can provide valuable feedback for needle navigation inside the tissue. For this purpose, fiber-optical sensors can be directly integrated into the needle tip. Optical coherence tomography (OCT) can be used to image tissue. Here, we study how to calibrate OCT to sense forces, e.g. during robotic needle placement. Methods. We investigate whether using raw spectral OCT data without a typical image reconstruction can improve a deep learning-based calibration between optical signal and forces. For this purpose, we consider three different needles with a new, more robust design which are calibrated using convolutional neural networks (CNNs). We compare training the CNNs with the raw OCT signal and the reconstructed depth profiles. Results. We find that using raw data as an input for the largest CNN model outperforms the use of reconstructed data with a mean absolute error of 5.81 mN compared to 8.04 mN. Conclusions. We find that deep learning with raw spectral OCT data can improve learning for the task of force estimation. Our needle design and calibration approach constitute a very accurate fiber-optical sensor for measuring forces at the needle tip.



There are no comments yet.


page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Needle placement is a challenging problem for a variety of medical interventions, including brachytherapy or biopsy taylor2016medical . The force acting on the needle tip allows for inference about the currently penetrated tissue. This information can be used to navigate the needle and to prevent injuries of delicate structures okamura2004force . In order to distinguish tissue based on tip forces, it may be required to measure those with an accuracy of approximately Mccreery2008 . Tip forces cannot be measured with external sensors due to friction forces at the needle shaft kataoka2002measurement . Therefore, small-scale fiber-optical force estimation methods have been directly integrated into the needle tip. Several sensor concepts are based on Fabry-Pérot interferometry beekmans2017fiber and Fiber Bragg Gratings kumar2016detecting . Here, we consider a setting where optical coherence tomography is available, e.g., to study tissue deformation Otte2012 or to realize elastography Latus2017 . While OCT has been proposed for tip forces estimation before gessert2018needle ; gessert2019spatio , these approaches rely on the reconstructed gray value data. However, using the reconstructed data has two limitations. First, the signal processing is based on a number of assumptions which may cause some loss of signal information. Second, it does not incorporate the phase part of the complex OCT signal that is particularly sensitive to small axial shifts. Therefore, we explore whether the tip force estimation accuracy can be improved by directly using the raw spectral OCT data. Thus, we perform a calibration between the optical signal and forces applied to the needle tip with convolutional neural networks. We validate our approach with three different needles using a new, improved needle design.

2 Methods

2.1 Needle Design

We used an improved needle design for force estimation at the needle tip. A scheme and an image of the needle are shown in Figure 1. A brass tip with a piston is put on a brass sleeve such that it is able to perform a sliding motion inside of it. An epoxy layer between tip and sleeve acts as a spring. An optical fiber is embedded into a ceramic ferrule. The ferrule is positioned relative to the tip piston such that the light beam travels a distance of approximately through air until hitting the surface of the piston. For protection, the ferrule is embedded into a polymer tube which is glued to the brass sleeve. When forces act on the needle tip, the epoxy layer is compressed and the piston moves closer to the exit point of the laser beam which can be detected in the OCT signal. The diameter of the needle is .

In gessert2018needle the needle tip was constructed as a cone and attached to the needle shaft with a deformable epoxy layer. Thus, radial forces on the needle tip could easily tilt it. The improved piston construction has the advantage that it guides the tip in axial needle direction and prevents tilt. This contributes to a more reproducible signal, an important aspect in the calibration of the needle, and to the overall durability.

Epoxy layer

OCT Fiber

Figure 1: The design of the needles used in this work. In the upper image one of our needles is depicted below, in comparison to a standard G18 biopsy needle above. In the schema, the brass tip with a piston and the brass sleeve are depicted in yellow. The ferrule guiding the fiber is depicted in gray. The protection tube is depicted in orange.

2.2 Calibration Data

We acquire calibration data for three custom build needles identical in construction. The data acquisition is performed similar to gessert2019spatio where a needle is driven against a flat surface with a stepper motor. The force in the axial direction of the needle is measured with a force sensor and recorded together with the associated raw OCT data. Approximately OCT-force pairs are collected for each needle for forces between and . We perform regular OCT data reconstruction, which includes the following steps:

  1. Dechirping the data by resampling and interpolating to new sampling points, based on manufacturer specifications

  2. Estimation of the DC spectrum using an exponential moving average with a damping coefficient of

  3. Subtraction of the DC spectrum

  4. Apodization by using a Hann window for filtering the spectral data

  5. Fourier transform for mapping frequency values to spatial values

  6. Selection of the absolute signal value as the final reconstructed intensity image

Throughout the reconstruction process, information can be lost due to the DC spectrum estimation strategy (2), the apodization (4) which eliminates high-frequency signal parts and the selection of the absolute signal value (6) as the final image. Figure 2 shows an excerpt of the collected data, both in raw (left) and reconstructed form (right).

The raw OCT signal has a size of where is the number of scans acquired over time. Reconstruction up to step (5) results in a complex signal which is also of size . Finally, by taking the absolute signal value, the A-Scan image sequence of size is obtain. This can be interpreted as a sequence of 1D depth images over time.





piston surface
Figure 2: Example OCT image data over a period of time. Left, the raw OCT signal is shown. Right, the reconstructed A-Scans (depth profiles) are shown. One individual scan is depicted in one column. The scans were collected as part of a time series. Subsequent scans are adjacently depicted (M-Scan). The reconstructed scans allows human interpretation to some extend, e.g., the piston surface is visible. Patterns in the raw OCT data are much more subtle and not interpretable by the human eye.

2.3 Deep Learning Architectures

We compare prediction performance for needle tip forces from both raw and reconstructed OCT data with different convolutional neural network (CNN) architectures. The considered architectures are variants of the ResNet resnet

, which is an extension of CNNs that enables better training through improved gradient flow. The ResNet models were originally built for 2D images with 2D convolutions. Here, the A-Scan data represents 1D images. Therefore, we adapt the architectures by replacing 2D convolutions by 1D convolutions. Furthermore, we replace the network’s original output layer for multi-class classification with a fully-connected layer with one output for needle force regression. We consider several ResNet variants, resembling different network sizes, the regular ResNet34 and ResNet18 architecture as well as a smaller architecture with 2 residual blocks and 6 convolutional layers, which we name ResNet6. All deep learning models are implemented in PyTorch


. Learning is performed over 150 epochs with a batch size of

and a learning rate of 0.005 using the Adam optimizer. As a loss function we use the mean squared error which is defined as


where is the ground-truth force and is the predicted force value. We use of the data as a hold-out validation set. We performed five training runs with different random seeds and averaged the individual results.

3 Results

We report the mean absolute error (MAE) in mN between force predictions and force targets on the validation set. Additionally, we report inference times for the examined neural network architectures. All results are shown in Table 1. Figure 3 shows the relative differences in errors graphically. For most combinations of needle and model architecture the error for learning on raw OCT data is lower compared with learning from reconstructed OCT scans. Particularly for the ResNet34 architecture, the calibration performance improves for all needles.

Needle 1 Needle 2 Needle 3 Inf. times
raw recon raw recon raw recon
ResNet 6
ResNet 18
ResNet 34
Table 1: Mean absolute error results in mN and inference times in ms.
 Needle 1
(a) Needle 1
(b) Needle 2
(c) Needle 3
Figure 3: Bar plot showing the relative difference of the mean absolute error when learning from raw data.

4 Discussion and Conclusion

In this paper we address the calibration problem of OCT-based needle tip force estimation using new and improved sensor concept with a piston and a guiding sleeve. In contrast to a previous OCT-based concept gessert2018needle , the needle is not as sensitive to lateral forces by design while achieving similar calibration results with intensity data.

Furthermore, we study an approach for improving deep learning-based calibration performance even further. During OCT acquisition a spectral signal is obtained which is typically reconstructed to an intensity depth image. We illustrate that learning is possible in an end-to-end fashion, i.e., the process of image reconstruction can be avoided. Moreover, the results improve in six out of nine setups, indicating that there may be information lost in the original processing that can be used when training the network on the raw signal. The proposed approach allows for precise estimation of forces at the needle tip, which is particularly interesting for force based robotic needle placement. With typical robot control cycles of , a trade-off between accuracy and inference time must be balanced.

We find that learning forces from raw OCT data instead of reconstructed images work well, in particular, for larger deep learning models. For future work, our approach could be studied in more detail with additional deep learning methods and in different applications scenarios. Also, our approach could be extended to other applications where an imaging modality is used as a sensor signal, for example in the context of motion tracking with OCT or ultrasound.

Compliance with Ethical Standards

Funding: This work was partially supported by the TUHH initiative and DFG grants SCHL 1844/2-1 and SCHL 1844/2-2.

Conflict of Interest: The authors M. Gromniak, N. Gessert, T. Saathoff and A. Schlaefer declare that they have no conflict of interest.

Ethical Approval: This article does not contain any studies with human participants or animals.

Informed Consent: Not applicable.


  • (1) Taylor, R.H., Menciassi, A., Fichtinger, G., Fiorini, P., Dario, P. (2016) Medical robotics and computer-integrated surgery. In: Springer handbook of robotics, pp. 1657–1684. Springer
  • (2) Okamura, A.M., Simone, C., O’leary, M.D. (2004) Force modeling for needle insertion into soft tissue. IEEE transactions on biomedical engineering 51(10), 1707–1716
  • (3) Mccreery, G., Trejos, A., Naish, M., Patel, R., Malthaner, R. (2008) Feasibility of locating tumours in lung via kinaesthetic feedback. The international journal of medical robotics and computer assisted surgery : MRCAS 4, 58–68. DOI 10.1002/rcs.169
  • (4) Kataoka, H., Washio, T., Chinzei, K., Mizuhara, K., Simone, C., Okamura, A.M. (2002) Measurement of the tip and friction force acting on a needle during penetration. In: MICCAI, pp. 216–223. Springer
  • (5) Beekmans, S., Lembrechts, T., van den Dobbelsteen, J., van Gerwen, D. (2017) Fiber-optic fabry-pérot interferometers for axial force sensing on the tip of a needle. Sensors 17(1), 38
  • (6) Kumar, S., Shrikanth, V., Amrutur, B., Asokan, S., Bobji, M.S. (2016) Detecting stages of needle penetration into tissues through force estimation at needle tip using fiber bragg grating sensors. Journal of biomedical optics 21(12), 127,009
  • (7) Otte, C., Hüttmann, G., Schlaefer, A. (2012) Feasibiliy of optical detection of soft tissue deformation during needle insertion. In: SPIE 8316, Medical Imaging 2012: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 8316, pp. 282 – 292
  • (8) Latus, S., Otte, C., Schlüter, M., Rehra, J., Bizon, K., Schulz-Hildebrandt, H., Saathoff, T., Hüttmann, G., Schlaefer, A. (2017) An approach for needle based optical coherence elastography measurements. MICCAI 201: Medical Image Computing and Computer-Assisted Intervention pp. 655–663
  • (9) Gessert, N., Priegnitz, T., Saathoff, T., Antoni, S.T., Meyer, D., Hamann, M.F., Jünemann, K.P., Otte, C., Schlaefer, A. (2018) Needle tip force estimation using an oct fiber and a fused convgru-cnn architecture. In: MICCAI, pp. 222–229. Springer
  • (10) Gessert, N., Priegnitz, T., Saathoff, T., Antoni, S.T., Meyer, D., Hamann, M.F., Jünemann, K.P., Otte, C., Schlaefer, A. (2019) Spatio-temporal deep learning models for tip force estimation during needle insertion. Int J CARS 14, 1485–1493
  • (11) He, K., Zhang, X., Ren, S., Sun, J. (2016) Deep residual learning for image recognition. pp. 770–778. DOI 10.1109/CVPR.2016.90
  • (12) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S. (2019) Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc.