Tattoo tomography: Freehand 3D photoacoustic image reconstruction with an optical pattern

11/10/2020 ∙ by Niklas Holzwarth, et al. ∙ 11

Purpose: Photoacoustic tomography (PAT) is a novel imaging technique that can spatially resolve both morphological and functional tissue properties, such as the vessel topology and tissue oxygenation. While this capacity makes PAT a promising modality for the diagnosis, treatment and follow-up of various diseases, a current drawback is the limited field-of-view (FoV) provided by the conventionally applied 2D probes. Methods: In this paper, we present a novel approach to 3D reconstruction of PAT data (Tattoo tomography) that does not require an external tracking system and can smoothly be integrated into clinical workflows. It is based on an optical pattern placed on the region of interest prior to image acquisition. This pattern is designed in a way that a tomographic image of it enables the recovery of the probe pose relative to the coordinate system of the pattern. This allows the transformation of a sequence of acquired PA images into one common global coordinate system and thus the consistent 3D reconstruction of PAT imaging data. Results: An initial feasibility study conducted with experimental phantom data and in vivo forearm data indicates that the Tattoo approach is well-suited for 3D reconstruction of PAT data with high accuracy and precision. Conclusion: In contrast to previous approaches to 3D ultrasound (US) or PAT reconstruction, the Tattoo approach neither requires complex external hardware nor training data acquired for a specific application. It could thus become a valuable tool for clinical freehand PAT.



There are no comments yet.


page 4

page 6

page 8

page 9

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Photoacoustic imaging (PAI) is an emerging imaging modality that has undergone rapid development since the late 90s PA-review-wang . PAI is based on the photoacoustic (PA) effect PAeffect and - using a light in - sound out

principle - provides high contrast in optically absorbing tissue regions. Tissue is illuminated with nanosecond-long pulsed near-infrared laser light. When the light is absorbed by specific molecules, thermoelastic expansion causes a local pressure rise that generates an acoustic shock wave which can be measured by a standard ultrasound (US) transducer. Furthermore, acquiring multispectral PA data allows not only to spatially resolve highly concentrated absorbing tissue regions, but also to define the absorbing molecules and estimate their concentration in a process referred to as spectral unmixing

SU-collagen . It thus enables the measurement of both morphological and functional tissue information in depths up to several centimeters. A key application of PAI is the estimation of the blood oxygenation, which is relevant for a variety of diseases clinicalPAT .

Clinical PAI devices typically use handheld 2D ultrasound transducers for data acquisition. The bottleneck of such systems is that they produce only cross-sectional 2D slices of the imaged structures, although the full 3D context is crucial for various applications needle epileptic . Current hardware solutions towards 3D imaging include whole body scanners - which only allow small animal imaging - or 3D probes, which lack in penetration depth, field-of-view and spatial resolution and are associated with additional costs 2d-3d-probe . Another approach towards 3D US/PAI is based on constrained movement of a 2D probe, either introduced by additional mechanical hardware laurence or light-weight robots robots

, which also offers additional intrinsic tracking data. An alternative are software-based solutions for compounding 3D volumes based on pose estimation of the individual 2D image slices. However, these may also rely on complex hardware, such as an external tracking system

Kirchner . Mobile cameras mounted to the probe for motion tracking have also been investigated 6dof but come at the cost of a complex hardware setup as well. To overcome these issues, sensorless approaches based on speckle decorrelation chen_1997 gao_2016 have been proposed for US, but they are not well applicable to PA images, which do not show speckle effects due to the physical difference in image formation between PAI and US wang_speckle

. Recent work enhanced the speckle-based approach with deep learning based components


. However, machine learning-based approaches rely on training algorithms for specific applications and may thus lack in robustness when transferred to new settings.

To overcome the limitations of prior work, we present an entirely novel approach to 3D reconstruction that leverages the unique capability of PAI to tomographically reconstruct optical properties. Specifically, the concept is based on an optical pattern placed on the region of interest which allows the recovery of the 3D probe pose relative to the pattern coordinate system based on a single tomographic image. By transforming a sequence of freehand PAI images into the common pattern coordinate system, a consistent 3D reconstruction of PAI imaging data can be obtained without the need for an external tracking system.

The following sections present the first prototype implementation of the novel approach (patent pending) along with a feasibility study assessing the accuracy and precision of the reconstruction.

2 Material and Methods

In this section, (1) the general concept of Tattoo tomography is outlined, followed by a description of the first prototype implementation, comprising information on (2) the used PAI imaging device, (3) the first optical pattern designed for the approach, (4) the corresponding approach to pose estimation and (5) the method for compounding. Finally, (6) the validation concept is described.

2.1 Tattoo tomography concept

Figure 1: Tattoo tomography: The approach to 3D image reconstruction comprises four steps. (1) Prior to image acquisition, the optical pattern is placed on the region of interest. (2) Next, a sequence of images is acquired, each showing a part of the pattern as well as the target region. (3) The image features corresponding to the pattern (here: three points) are extracted from each slice and used to recover the pose of the probe relative to the pattern coordinate system. (4) Once all acquired slices have been transformed (T) into a common coordinate system (C), the 3D volume is compounded.

The process of reconstructing a 3D image with the Tattoo tomography concept is illustrated in Fig. 1.

  1. Pattern attachment: Prior to image acquisition, an optical pattern is placed above the anatomical region of interest. This pattern should fulfill the following key requirements: (1) It should be designed in a way that a tomographic image of it enables the estimation of the probe pose relative to the coordinate system of the pattern and that (2) it is easy to mount and remove in a clinical setting. (3) The dye that makes up the pattern (here seen in green) should absorb in the frequencies that match those of the imaging system (here: near-infrared) and that are complementary to those used for the actual imaging.

  2. Image acquisition: A sequence of images is acquired, each showing a part of the pattern as well as the target region.

  3. Pose estimation: As the pose information is encoded in the optical pattern by design, the pose of each image slice within the pattern coordinate system can be determined via the PA image features corresponding to the pattern.

  4. Image compounding: The 3D volume is reconstructed by interpolating between the transformed slices in the pattern coordinate system.

The following sections present the first prototypical implementation of this concept.

2.2 PA imaging device

All PA images used as part of this work were acquired with a multispectral optoacoustic tomography (MSOT) Acuity Echo research system (iThera Medial GmbH, Munich, Germany). It is a hybrid device, which can acquire co-registered and synchronised PA and US images. The built-in laser is a Nd:YAG laser with a tuning range from 660 – 1300 nm, a peak pulse energy of 30 mJ, a repetition rate of 25 Hz and a pulse duration of 4 – 10 ns. The concave 2D US transducer has a 4 MHz center frequency, 256 elements and a radius of 40 mm.

2.3 Optical pattern

For this first prototypical implementation, a trident design was chosen, as depicted in Fig. 2. This trident consists of an isosceles triangle (for simplicity referred to as tilted lines) and the corresponding angle bisector between the tilted lines of the triangle (referred to as central line).

Mathematically speaking, this initial prototype pattern design currently only allows accounting for three degrees of freedom (DoF), representing the intersection line of the image coordinate system with the

Tattoo coordinate system. The remaining three DoFs are recovered by imposing constraints on the image acquisition process: Firstly, the optical pattern has to be approximately placed such that it resembles a flat surface, and secondly, the PA probe is required to be positioned orthogonally to the pattern plane (orthogonality constraint).

The pattern was printed on a transparent foil commonly used in overhead projection (PrintLine Overheadfolie, Vleveka, Deurne, Netherlands) using cyan ink of a print station (TASKalfa 5052ci, Kyocera, Esslingen, Germany). These foils are well-suited for skin contact, easily sterilizable and printable in a straightforward manner. During acquisition, the pattern provides a strong contrast at a wavelength of while the functional imaging was performed at a wavelength of , where oxygenated hemoglobin is a dominant absorber in tissue. The foil is placed on tissue with a thin layer of coupling gel on top and underneath the foil and fixated using tape to avoid movement of the foil relative to the skin. The setup allows to easily place and remove the pattern on / from the tissue of interest.

2.4 Pose estimation

The pose estimation step refers to estimating the pose of an individual image slice relative to the pattern coordinate system (). To this end, the information corresponding to the optical pattern is extracted from the PAI image that reflects the wavelength in which the pattern is absorbing (here: 750 nm). When using the optical pattern proposed in the previous section, this task is equivalent to extracting the three intersection points of the image with the line-based pattern, as depicted in Fig. 1. This is achieved with the SciPy Scipy function find_peaks. Based on the three points, the line corresponding to the probe pose on the optical pattern (blue line in Fig. 2) can be unambiguously determined. It is characterized by the distance of the absorption peak of the central pattern line c to the pattern origin and the slice tilt angle as defined in Fig. 2. These are computed as follows:


where and are the distances between the right and left point to the central point respectively and is the pattern opening angle. When assuming an orthogonal pose of the probe relative to the pattern plane, the line given by and uniquely determines the pose of the probe in 3D. This results in the following transformation matrix which transforms the PA image slice into the tattoo coordinate system:


where and are the x and y coordinates of the central Tattoo point in the PA image slice and and the previously derived Tattoo coordinates.

Figure 2: 2D representation of the tattoo containing all relevant variables (left) and corresponding photoacoustic image (right). It shows the pattern (green), and the intersection of the image slice with the pattern (blue line). The image clearly shows three points representing the part of the image that intersects with the optical pattern (l, c, r). Based on the distances and between neighbouring points, the pose of the intersection line, represented by a blue line can be unambiguously determined.

2.5 3D volume compounding

The 3D compounding algorithm is inspired by the ”Volume reconstruction algorithm” of the public software library for ultrasound imaging research (PLUS) plus . A 3D zero-valued voxel grid is defined inside the bounding box of all transformed slices. The voxel spacing is set corresponding to the pixel spacing of the image in x-y direction and predefined according to the probe movement in z direction. Each image pixel in each slice is assigned to the closest voxel location. The individual voxel values are then calculated as the average pixel value of all assigned pixels.

2.6 Experimental setup

The purpose of our experiments was to assess the general feasibility of the Tattoo tomography concept. We provide (1) a qualitative visual demonstration, (2) a quantitative validation of the accuracy in a phantom experiment and (3) a quantitative validation of the precision in an in vivo experiment.

The pattern was constructed with an opening distance of 20 mm and a vertical extension () of 50 mm. The scans of all experiments were performed using the two wavelengths 750 nm and 850 nm, where 750 nm corresponds to the wavelength in which the optical pattern is absorbing strongly (and thus well-visible) and 850 nm corresponds to wavelength used for the construction of the ”functional” tissue information. The PA data was reconstructed using a delay-and-sum DAS algorithm.

Feasibility demonstration:

For demonstration of the principal feasibility of the approach, a logo of the international conference on Information Processing in Computer Assisted Interventions (IPCAI) was printed and placed inside a gel pad (AQUAFLEX Ultrasound Gel Pad, Parker Laboratories, inc., Fairfield, USA), that was cut in half in height (see Fig. 

4 a & b). The optical pattern was fixed on top of the gel pad with pins. For scanning, the probe was moved along the pattern in two principal ways: (1) Slow and without direction changes, representing a careful acquisition and (2) fast with many direction changes, representing an extremely careless acquisition (see Fig. 5, left column).

Phantom validation:

The phantom used for the phantom validation was an N-wire phantom (see Fig. 3) typically used for hand-eye calibration of tracked devices Kirchner plus . The phantom was built using three layers of N-shaped black nylon wires (0.4 mm in diameter) showing a strong broadband absorption in the near-infrared. It was placed within a milk bath containing 1.5 l of tap water and 20 g of coffee cream (10 % fat) to increase the scattering coefficient at room temperature. The phantom wiring was transferred to a 3D model based on the known location of the wires. A point representation of this model was then used as reference for the validation, as detailed below (see Fig. 6).

For image acquisition, the optical pattern was fixed on top of the phantom using Velcro strips. This setup was placed upon a movable optical breadboard, and the probe was clamped above the phantom with a fixed probe angle (rotation around y-axis). To enable the comparison with a baseline 3D reconstruction method, the probe was tracked with an optical tracking system (NDI Polaris Spectra camera, NDI Medical, Waterloo, Canada). During the scanning, the breadboard was moved under the probe. The PAI scans were then compounded using both the Tattoo method (cf. section 2.5) as well as the optical tracking baseline method described in Kirchner . Subsequently, a 3D segmentation of the PA wire volume was performed by applying a background suppressing threshold using the Medical Image Interaction Toolkit mitk .

Based on the known N-wire model and the 3D reconstructions, the accuracy of the compounded images was assessed. As the relation of the phantom coordinate system and the Tattoo coordinate system can only be approximated, we concentrated the validation on the consistency of the reconstructed model. Specifically, we applied the iterative closest point (ICP) algorithm as implemented in the MITK A-ICP framework (see Fig. 6) to register the reconstructed point cloud with a (simulated) point cloud representing the N-wire model. The fiducial registration error (FRE) was then used for quantifying the accuracy of the 3D reconstruction (see Table 1). As stopping criterion for the ICP algorithm, the change of the FRE between two ICP iterations had to be smaller than 0.001 mm.

Figure 3: Phantom used for the quantitative validation of reconstruction accuracy. The optical markers (silver spheres) enable the comparison to a baseline 3D reconstruction method. The red box (right: zoomed in) highlights the optical pattern placed on top of the N-wires. The wires have a diameter of and the holes of the frame are apart from each other.

In vivo validation:

In the in vivo validation step, we concentrated on validating the precision (reproducibility) of our method. To this end, we obtained a total of 30 Tattoo scans from three healthy volunteers. For each volunteer, the optical pattern was placed on the forearm, and 10 measurements were taken while tracking the PAI probe with an optical tracking system. All measurements were transformed to the Tattoo coordinate system (for our method) and the coordinate system of the optical tracking system (for the baseline method), and two corresponding 3D reconstructions were performed. The largest image region, corresponding to a vessel and comprising measurements of at least 7 volumes, served as target region for the assessment. To approximate precision, we performed a slice-based analysis. The reference position of the vessel center was computed as the mean of the (up to) 10 vessel locations extracted from the individual scans. The latter were again determined with the find_peaks function (see Sec. 2.4). The mean Euclidean distance between this reference and the individual measurement served as metric for the precision of the compounding and referred to as mean vessel distance (MVD).

3 Results

Feasibility demonstration

Fig. 4 and Fig. 5 show the setup and the results for the feasibility experiment. It can be seen that the Tattoo reconstruction is feasible even when images are acquired in an extremely careless and unsystematic manner.

Figure 4: The experimental setup and results of the Feasability demonstration experiment. The IPCAI logo printed on paper is placed underneath a gel pad on top of which the optical pattern is fixed and some ultrasound gel is added (a). A freehand PA scan is acquired (b), which results in a fully image-based and clear reconstruction of the IPCAI logo.
Figure 5: Good and poor reconstruction of a target region (here the text IPCAI, right) corresponding to a relatively slow and careful image acquisition (top) and to a fast and careless acquisition (bottom). The 2D images represent an intersection of the image stack at a fixed y-position before (left) and of the 3D volume after (middle) Tattoo reconstruction.

Phantom validation:

The results of the Tattoo-compounded volumes registered with a 3D model of the N-wire phantom can be seen in Fig. 6. The FREs for the registration of the compounded volumes (Tattoo and optical tracking) with the N-wire model are shown in Table 1 for three different, but fixed, rotation angles (0°, 4°, 8.5°). It is 0.63 mm on average for the Tattoo compounding and 0.87 mm on average for the optical tracking-based compounding.

Figure 6: The 3D model of the N-wire phantom (white) is displayed together with the optical pattern compounded volumes of a single scan (blue, left) and all three scans (blue, red, green, right) after applying an ICP algorithm to register the Tattoo point cloud on the N-wire model (cf. Table 1).
Scan Tattoo FRE [mm] Baseline FRE [mm]
1 0.67 0.88
2 0.57 0.84
3 0.66 0.89
Table 1: Consistency of 3D reconstruction of the proposed approach (Tattoo) and the baseline method (optical tracking), assessed by registering a model of the phantom wires with the corresponding reconstructions (see. Sec. 2.6). The fiducial registration error (FRE) of the iterative closest point (ICP) algorithm served as quality metric.

In vivo validation:

As it was not possible to identify a suitable vessel structure for one of the volunteers, our results are restricted to the 20 scans of the remaining two volunteers, referred to as v1 and v2. As summarized in Table 2, the Tattoo-based registration is slightly more reproducible compared to the optical tracking-based method.

Volunteer Tattoo MVD [mm] Baseline MVD [mm]
v1 0.63 1.40
v2 0.80 1.06
Table 2: In vivo validation results using a slice-based MVD for both the proposed approach (Tattoo) and the optical tracking-based method (baseline).

4 Discussion

In this paper, we presented Tattoo tomography, a novel approach to 3D PA image reconstruction from an acquired sequence of 2D PA image slices. The key advantage of the concept is its simplicity; applying a pattern without the need of involved calibration, training of an algorithm, or any additional complex hardware allows smoothly integrating the proposed approach into any clinical workflow. As the clinically relevant information on the tissue and the information needed to decode the probe pose can be encoded in different wavelengths, the approach does not come at the cost of reduced image quality. Our prototype implementation of the proposed concept further illustrates that such a pattern-based concept may also feature higher accuracy and precision compared to widely used techniques that rely on external tracking devices.

It is worth mentioning that we had initially planned to use the optical tracking-based approach as reference method. First experiments, however, suggested that the proposed approach might be even more accurate and precise than this method, which is why we decided on a comparative assessment. The FRE obtained for both reconstruction methods may appear relatively high ( 0.6-0.8 mm). We attribute this to the fact that the model representation of the phantom was only an approximation of reality. This hypothesis is supported by the fact that both reconstruction methods feature a systematic mismatch of the outer wires, as shown in Fig. 6.

In our in vivo experiment we examined the precision of Tattoo tomography in comparison to the optical tracking baseline. The fact that the Tattoo approach was substantially more precise then the baseline method in terms of the MVD may most likely be attributed to the different underlying world coordinate systems. The tattoo coordinate system is a relative coordinate system that moves with the target structure and is, therefore, influenced by patient motion to a lesser degree. Due to the limited number of volunteers and the lack of a reliable reference for the reconstruction, further studies are required to support these initial results.

While the general Tattoo concept proposed is potentially very powerful, our first prototype implementation comes with several limitations. The main current drawback is the fact that we use a 2D pattern. A 3D pattern, allowing for the recovery of 6 DoF from a single measurement, would eliminate the need for holding the probe orthogonally to the pattern. The second current limitation, namely the required flatness of the target surface, could then be addressed by image processing-based methods to approximate the shape of the tissue of interest.

Finally, we are planning on enhancing the optical pattern, such that it can not only be used for 3D reconstruction but also for multi-modal registration with conventional imaging data, such as computed tomography (CT) or magnetic resonance imaging (MRI) data.

In conclusion, we have presented the first 3D PAI reconstruction concept that elegantly leverages the unique capabilities of PAI. The proposed concept is simple to apply, does not require additional devices such as a tracking system, and can easily be integrated into clinical workflows. According to initial results, it may even be more accurate and precise than competing methods that rely on complex hardware setups. We therefore assume that the concept has high potential for future clinical translation.



This project has been funded by the Data Science Driven Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg and received funding from the European Unions Horizon 2020 research and innovation program through the ERC starting grant COMBIOSCOPY under Grant Agreement No. ERC-2015-StG-37960.

Conflicts of interest:

The authors declare that they have no conflict of interest.

Ethics approval:

The healthy human volunteer experiments were approved by the ethics committee of the medical faculty of Heidelberg University under reference number S-451/2020 and the study is registered with the German Clinical Trials Register under reference number DRKS00023205.

Availability of data/code and material:

Available upon request.


  • (1) Li, C., Wang, LV.: Photoacoustic tomography and sensing in biomedicine. Physics in Medicine & Biology 54(19), R59 (2009)
  • (2) Rosencwaig, A.: Theory of the photoacoustic effect with solids. Journal of Applied Physics 41(1), pp. 64–69 (1976).
  • (3) Regensburger, AP.: Detection of collagens by multispectral op- toacoustic tomography as an imaging biomarker for Duchenne muscular dystrophy. Nature Medicine 25(12), pp. 1905-1915 (2019).
  • (4) Attia, ABE.: A review of clinical photoacoustic imaging: Current and future trends. Photoacoustics.16, pp. 100144 (2019).
  • (5) Feimo, S.: Three‐dimensional sonography with needle tracking: Role in diagnosis and treatment of prostate cancer. Journal of Ultrasound in Medicine 27(6), pp. 895–905 (2008).
  • (6) Qizhi, Z.: Non-invasive imaging of epileptic seizures in vivo using photoacoustic tomography. Physics in Medicine & Biology 53(2), pp. 1921 (2008).
  • (7) Neuschmelting, V.: Performance of a multispectral optoacoustic tomography (MSOT) system equipped with 2D vs. 3D handheld probes for potential clinical translation. Photoacoustics 4(1), pp. 1–10 (2016).
  • (8) Laurence, M.: A review of calibration techniques for freehand 3-D ultrasound systems. Ultrasound in medicine & biology 31(4), pp. 449–471 (2005).
  • (9) Esteban, J.: Robotic ultrasound-guided facet joint insertion. International journal of computer assisted radiology and surgery 13(6), pp. 895–-904 (2018).
  • (10) Kirchner, T.: Freehand photoacoustic tomography for 3D angiography using local gradient information. In: Photons Plus Ultrasound: Imaging and Sensing, SPIE BiOS, San Francisco, California, United States (2016).
  • (11) Sun, Shih-Yu: Freehand 3D ultrasound volume imaging using a miniature-mobile 6-DOF camera tracking system. In: 9th IEEE International Symposium on Biomedical Imaging (ISBI), IEEE (2012).
  • (12) Chen, JF.: Determination of scan-plane motion using speckle decorrelation: Theoretical considerations and initial test. International Journal of Imaging Systems and Technology 8(1), pp. 38–-44 (1997).
  • (13) Gao, H.: Wireless and sensorless 3D ultrasound imaging. Neurocomputing 195, pp. 159-–171 (2016).
  • (14) Xu, M., Wang, LV.: Biomedical photoacoustics. Review of Scientific Instruments 77(4) (2006).
  • (15) Prevost, R.: 3D freehand ultrasound without external tracking using deep learning. Medical image analysis 48, pp. 187–-202 (2018).
  • (16) Virtanen, P.: SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, pp. 261–272 (2020).
  • (17)

    Lasso, A: PLUS: Open-source toolkit for ultrasound-guided intervention systems. IEEE Transactions on Biomedical Engineering

    61(10), pp. 2527–-2537 (2014).
  • (18) Mozaffarzadeh, M.: Double-stage delay multiply and sum beamforming algorithm: Application to linear-array photoacoustic imaging. IEEE Transactions on Biomedical Engineering 65(1), pp. 31–-42 (2017).
  • (19) Wolf, I.: The medical imaging interaction toolkit. Medical image analysis 9(6), pp. 594–604 (2005).
  • (20) Maier-Hein, L.: Convergent iterative closest-point algorithm to accomodate anisotropic and inhomogenous localization error. IEEE transactions on pattern analysis and machine intelligence 34(8), pp. 1520–1532 (2011).