In recent works, for the risk-free training and planning of surgical needle interventions with visuo-haptic virtual reality simulators the inclusion of breathing motion models  is a core component. The simulation of tissue and needle deformation for puncture interventions and the modelling of virtual patient bodies [2, 3, 4, 5, 6, 7, 8] has been an active research branch for years [9, 10, 11, 12, 13, 14] and breathing motion can be rendered in a recent 4D visuo-haptic simulator (Fig. 1) with GPU support using 4D direct volume rendering [15, 16, 17, 18, 19, 20].
In our setup , the visuo-haptic simulation of the patient’s breathing motions is a new key feature. [10, 12, 19]. In the abdominal liver region, respiratory movements dominate the visuo-haptic experience of the physician with possible breathing displacements of up to 5 cm. . Conceptually, with a 4D breathing model, a new 3D CT data set could be animated. The model can be patient-specific or an averaged population-based model, which is the main topic of this work. Such a model then can be warped by non-linear registration to the 3D CT data set of a new patient. .
Surrogate free mean motion models were proposed by our group  for the lungs. The motivation for such models is to supply a more general model by averaging out patient specific breathing patterns. Thus, transferring such models instead of a patient-specific model to a new static 3D patient can be favorable.
Typically for VR training purposes, the acquisition of new 4D CT or MR data is impossible for ethics and cost reasons. Thus the transfer of existing motion models is appealing.
Here, we present a first study of an effective and efficient building process of a mean 4D breathing atlas parametrized by a surrogate signal and apply it to 3D patient data for rendering it in a 4D VR environment .
2 Materials and Methods
2.1 Study Data
The 4D mean model is built from 4D-CT data sets (Pat. 1-5, Fig. 2a-c) of the thorax and upper abdomen. Corresponding respiratory phase images (up to 512x512x462, 1 mm) with low dimensional spirometry parameter signals were used (Fig. 2d). The new patient is represented only by a static 3D CT data set in a frame of reference (e.g. reference phase from Pat. 3). Selected reference phases correspond to phase images in a corresponding inhalation state (new 3D patients hold breath, Fig. 2a-c).
2.2 Intra-patient Motion Modeling in a Frame of Reference
Breathing motion modeling relies on 4D CT data sets with phases, indexed by and . Spirometry signals [ml] serve as surrogate signal components to represent the patient’s breathing state: . The derivative allows to model respiratory hysteresis (inhalation vs. exhalation). Signal and motion dependence are assumed (locally) linear. intra-patient inter-phase image registrations to a selected patient specific reference phases are conducted first:
Using the distance metric and regularizer
, this step yields motion fields for each 4D patient data set. These can be used to calculate vector field coefficientsfor the application of a linear multivariate regression model.
Let us introduce a bijective serialization operator that produces a long column vector in a standardized way. The serialization of is . For one patient’s reference space, the serialized vector fields as regressand
are approximated by linear regression subject to an expanded serialized surrogate signal with, i.e. as regressor and discrete time points :
where denotes the Froebenius norm .
We yield a motion estimate using the solutionfor any point in time:
Now, we can simulate each 4D patient’s breathing state over time :
Our latest VR training simulator can render this compact, personalized representation of a breathing virtual patient in realtime by efficient raycasting using a bent rays approach .
2.3 Population-based Breathing Motion Models
The next step is the averaging of the personalized breathing models to yield a mean 4D motion model.
For a common frame of reference image phase, a reference patient is selected from the 4D patient population. This patient is targeted by inter-patient registrations in the reference phase .
Nonlinear registrations , minimizing the distance metric , warp the patient’s image data to the selected reference patient :
using a distance measure , regularizer , and a regularizer weight.
Next by , we warp the intra-patient-inter-phase deformations of the 4D patients to the common reference frame of :
With all motion fields in the same common frame of reference, averaging yields the prerequisite for the mean motion model:
Averaging is also done for the surrogate signals: . Finally, the mean patient’s motion behavior can now be estimated efficiently similar to one of the 4D patients (see section 2.2):
Image data for the artificial mean patient underwent an analogous transformation and averaging pipeline as the fields.
2.4 Application of Mean Motion Models
Three applications are at hand now at this juncture:
An individual patient from the training population can be animated by the mean motion model and be presented to the user for training purposes (Fig. 3a-d). This is alternative to using the patient-individual motion model [22, 17] providing a less individual or situational breathing pattern. In radiation therapy, this could make sense in follow-up fractional treatment sessions.
The population based patient intensity and motion atlas can be used to present an interactively manipulable breathing virtual training atlas to the apprentice user (Fig. 3e-h).
The most relevant application of the 4D mean patient breathing model is warping it to new unseen 3D static patient data and render it in our current 4D VR simulator (Fig. 3i-l).
and Eq. 6 accordingly to yield the new patient’s motion behaviour:
The respiratory motion applied to the new 3D patient image data can now be calculated efficiently (see section 2.2). Thus based only on a relatively low dose 3D-CT data acquisition in reference phase and the transferred mean motion model, the breathing movements can be plausibly approximated:
Optionally, simulated surrogate signals with stochastic irregularity  in the range of the observed breathing states can be used for the 4D animation of the mean atlas and new 3D CT data by such a mean motion model. In the data sets used here, we found relative breathing volumes of ca. 0 - 1200 ml.
2.5 Study Set-up
Volume image data was resampled to a size of 256 voxels to fit into moderate graphics cards with 2 GB GPU-RAM (Nvidia GTX 680) also comprising the three vector field coefficients . From our 4D patient population with fields of view shown in Fig. 2, for one experiment a representative 4D-CT data set in maximum inhalation phase of the thorax and upper abdomen serves as common reference frame and is left out from the motion model building process. The new patient for application option three is represented only by a static 3D CT data set. The reference patient with the image data and the new 3D patient image data are assumed to be in the same breathing state. For each of the 4D patients and according to Eq. 1, we first perform the intra-patient inter-phase registrations to the chosen reference phase of the currently considered 4D image data. In Eq. 1, we use the distance measure (normalized sum of voxel-wise squared differences ) and (sliding motion’-preserving) to cope with discontinuities in the pleural cavity . In Eqs. 5 and 9, (sum of squared differences) and (diffusive non-linear regularization). Regularization weights are set as or
, as interphase registration requires lesser regularization influence (same anatomy morphology). DICE coefficients of warped expert segmentation masks (liver, left/right lungs) from a leave-one-out crossvalidation are given to classify the quality of the difficult inter-patient registration. We present sample images for the three use cases, four time instants and online movie footage for reference patient 3 and the spirometry signal from Fig.2d in application 2 (mean intensity and motion atlas).
Inter-patient registration DICE results from a leave-one-out study for expert segmentation masks: Inter-patient (vs. intra-patient) registration still poses problems as can be seen by sporadic outliers.
3 Results and Conclusion
In the puncture-relevant liver region, the patients’ breathing states are simulated plausibly for an individual patient (Fig. 3a-d), a 4D intensity and motion atlas (Fig. 3e-h)111Demo movie of 4D intensity and motion atlas: https://goo.gl/Qog138 and an animated static new 3D patient (Figs. 3i-l). Cross-validation DICE values of atlas registrations of liver and lung masks to a left-out static patient yield overall median values of 0.87 (liver), 0.93 (right lung) and 0.93 (left lung). Sporadic bad inter-patient registration outliers are possible in the current process as can be seen in Fig. 4a-c by the dots. Here by the averaged motion, they cause artifacts at bony structures of the hip. Their influence to the process should be mitigated by a selection strategy in future work. Statistical 4D breathing motion models for the lungs have been introduced in . Here, we use a population of 4D patient image data sets and learn a 4D intensity and motion atlas parametrized by a surrogate spirometry signal. The surrogate signal for the calculation of a mean motion model in this work was simply found by averaging. Among other obvious applications, such motion model can be warped to new 3D patient data. The animated atlas or the 3D patient can be used in a 4D VR training system with promising outcomes . Despite the efficient regression calculus, future work will cover the direct joint transfer of the motion model coefficients.
-  J. McClelland, D. Hawkes, T. Schaeffter, and A. King, “Respiratory motion models: A review,” Medical Image Analysis 17(1), pp. 19 – 42, 2013.
-  A. Mastmeyer, G. Pernelle, R. Ma, L. Barber, and T. Kapur, “Accurate model-based segmentation of gynecologic brachytherapy catheter collections in mri-images,” Medical image analysis 42, pp. 173–188, 2017.
A. Mastmeyer, D. Fortmeier, and H. Handels, “Random forest classification of large volume structures for visuo-haptic rendering in CT images,” inProc. SPIE Medical Imaging: Image Processing, 9784, pp. 97842H–1–8, International Society for Optics and Photonics, 2016.
-  M. Meike, D. Fortmeier, A. Mastmeyer, and H. Handels, “Real-time resampling of medical images based on deformed tetrahedral structures for needle insertion vr-simulation,” in German Conference on Medical Image Processing - BVM 2015, pp. 443–448, Springer, 2015.
-  A. Mastmeyer, D. Fortmeier, and H. Handels, “Efficient patient modeling for visuo-haptic VR simulation using a generic patient atlas,” Comput Methods Programs Biomed 132, pp. 161–175, 2016.
-  A. Mastmeyer, D. Fortmeier, E. Maghsoudi, M. Simon, and H. Handels, “Patch-based label fusion using local confidence-measures and weak segmentations,” Proc. SPIE Medical Imaging: Image Processing , pp. 86691N–1–11, 2013.
-  A. Mastmeyer, D. Fortmeier, and H. Handels, “Anisotropic diffusion for direct haptic volume rendering in lumbar puncture simulation,” German Conference on Medical Image Processing - BVM 2012 , pp. 286–291, Springer, 2012.
-  K. Engelke, V. Bousson, L. Duchemin, C. Fuchs, D. Mitton, A. Mastmeyer, J. Adams, W. Kalender, W. Skalli, and J. Laredo, “Effect-the european femur fracture study using finite element analysis and 3d computed tomography.,” in Journal of Bone and Mineral Research, 21, p. S86, 2006.
-  D. Fortmeier, A. Mastmeyer, J. Schröder, and H. Handels, “A virtual reality system for PTCD simulation using direct visuo-haptic rendering of partially segmented image data,” IEEE J Biomed Health Inform 20(1), pp. 355–366, 2016.
-  A. Mastmeyer, T. Hecht, D. Fortmeier, and H. Handels, “Ray-casting based evaluation framework for haptic force-feedback during percutaneous transhepatic catheter drainage punctures,” Int J Comput Assist Radiol Surg 9, pp. 421–431, 2014.
-  A. Mastmeyer, T. Hecht, D. Fortmeier, and H. Handels, “Ray-casting-based evaluation framework for needle insertion force feedback algorithms.,” in German Conference on Medical Image Processing (Bildverarbeitung für die Medizin), pp. 3–8, 2013.
-  D. Fortmeier, A. Mastmeyer, and H. Handels, “Image-based palpation simulation with soft tissue deformations using chainmail on the GPU,” German Conference on Medical Image Processing - BVM 2013 , pp. 140–145, 2013.
-  D. Fortmeier, A. Mastmeyer, and H. Handels, “An image-based multiproxy palpation algorithm for patient-specific VR-simulation,” Medicine Meets Virtual Reality 21, MMVR 2014 , pp. 107–113, 2014.
-  D. Fortmeier, A. Mastmeyer, and H. Handels, “Image-based soft tissue deformation algorithms for real-time simulation of liver puncture,” Current Medical Imaging Reviews 9(2), pp. 154–165, 2013.
-  A. Mastmeyer, D. Fortmeier, and H. Handels, “Evaluation of direct haptic 4d volume rendering of partially segmented data for liver puncture simulation,” Scientific Reports 7(1), p. 671, 2017.
-  A. Mastmeyer, M. Wilms, D. Fortmeier, J. Schröder, and H. Handels, “Real-time ultrasound simulation for training of US-guided needle insertion in breathing virtual patients,” in Medicine Meets Virtual Reality 22, MMVR 2016, Studies in Health Technology and Informatics 220, pp. 219–226, IOS Press, 2016.
-  D. Fortmeier, M. Wilms, A. Mastmeyer, and H. Handels, “Direct visuo-haptic 4D volume rendering using respiratory motion models,” IEEE Trans Haptics 8(4), pp. 371–383, 2015.
-  D. Fortmeier, A. Mastmeyer, and H. Handels, “Optimized image-based soft tissue deformation algorithms for visualization of haptic needle insertion,” in Medicine Meets Virtual Reality 20 - NextMed, MMVR 2013, San Diego, California, USA, February 20-23, 2013, pp. 136–140, 2013.
-  D. Fortmeier, A. Mastmeyer, and H. Handels, “Gpu-based visualization of deformable volumetric soft-tissue for real-time simulation of haptic needle insertion,” German Conference on Medical Image Processing (Bildverarbeitung für die Medizin) , pp. 117–122, 2012.
-  A. Mastmeyer, D. Fortmeier, and H. Handels, “Direct haptic volume rendering in lumbar puncture simulation,” in Medicine Meets Virtual Reality 19, MMVR 2012, Studies in Health Technology and Informatics 173, pp. 280–286, IOS Press, 2012.
-  Y. Seppenwoolde, H. Shirato, K. Kitamura, S. Shimizu, M. van Herk, J. V. Lebesque, and K. Miyasaka, “Precise and real-time measurement of 3D tumor motion in lung due to breathing and heartbeat, measured during radiotherapy,” Int J Radiation Oncololgy, Biology, Physics 53, pp. 822–834, Jul 2002.
-  A. Mastmeyer, M. Wilms, and H. Handels, “Interpatient respiratory motion model transfer for virtual reality simulations of liver punctures,” Journal of World Society of Computer Graphics - J WSCG 25(1), pp. 1–10, 2017.
-  J. Ehrhardt, R. Werner, A. Schmidt-Richberg, and H. Handels, “Statistical modeling of 4D respiratory lung motion using diffeomorphic image registration.,” IEEE Transactions on Medical Imaging 30, pp. 251–265, Sept. 2011.
-  M. Wilms, R. Werner, T. Yamamoto, H. Handels, and J. Ehrhardt, “Subpopulation-based correspondence modelling for improved respiratory motion estimation in the presence of inter-fraction motion variations,” Physics in Medicine and Biology , 2017.
-  T. Hastie, R. Tibshirani, and J. Friedman, The elements of statistical learning 2nd edition, New York: Springer, 2009.
-  M. Wilms, J. Ehrhardt, R. Werner, M. Marx, and H. Handels, “Statistical analysis of surrogate signals to incorporate respiratory motion variability into radiotherapy treatment planning,” in SPIE Medical Imaging 2014, Image-Guided Procedures, Robotic Interventions, and Modeling, Z. Yaniv and D. Holmes III, eds., 9036, p. 90360J, (San Diego, USA), 2014.
-  A. Schmidt-Richberg, R. Werner, H. Handels, and J. Ehrhardt, “Estimation of slipping organ motion by registration with direction-dependent regularization,” Medical Image Analysis 16(1), pp. 150 – 159, 2012.