1 Introduction
Current clinical systems for minimally invasive procedures, such as cardiac radiofrequency ablation, imageguided needle biopsies, and endovascular interventions, typically incorporate static 3D surfaces for guidance. Realtime dynamic tracking of 3D surfaces can help to optimize the interventional procedure, especially for complex anatomical structures undergoing gross tissue deformation, bulk organ motion, and potential topological changes during interventions.
A combination of multiple imaging modalities has been used for dynamic 3D navigation. For example, a realtime registration scheme based on both spatial registration and electrocardiography was proposed to overlay preoperative 3D magnetic resonance (MR) or computed tomography (CT) volumes onto intraoperative 2D ultrasound images for dynamic 3D navigation (Huang et al., 2009). 3D transesophageal echocardiography (TEE) was fused with 2D Xray fluoroscopic images using image localization and calibration for dynamic cardiac navigation (Gao et al., 2012)
. However, based on a combination of multiple imaging modalities, the dynamic 3D shapes were either interpolated from preoperative 3D volumes or intraoperatively collected 3D volumes with lowresolution. A 3D shape recovery scheme based on intraoperative 2D images including Xray, ultrasound, and MR could take intraoperative information into account whilst achieving highresolution at the same time. This kind of 3D shape recovery is termed dynamic shape instantiation. The scheme may or may not involve the use of template models
(Filippi et al., 2008). Without template models used, more intraoperative information and longer image acquisition time are needed; for example, at least seven intraoperative 2D images were needed for reasonable 3D prostate reconstruction (Cool et al., 2006). In this paper, a single intraoperative 2D image is targeted and hence we focus on templatebased 3D shape instantiation.For templatebased 3D shape instantiation methods, statistical shape models (SSM) (Frangi et al., 2002), free form deformation (FFD) (Koh et al., 2011), and Laplacian surface deformation (Karade & Ravi, 2015) can be used for the representation of templates. SSM (Cootes et al., 1995) is a popular technique which represents a set of 3D meshes or 2D contours with the same number of vertices and connectivities. SSMbased 3D shape instantiation learns from shape variations rather than only applying smoothness and 2D/3D similarity as the constraints. It deforms an initial 3D SSM to match intraoperative sparse inputs such as ultrasoundderived surface points (Barratt et al., 2008), digitized landmarks (Rajamani et al., 2007), or two or more calibrated Xray images (Baka et al., 2011). These methods usually learn a model from a training set of anatomies of multiple patients and deform the learned model for a new patient, which requires a high anatomical similarity between patients. This learning is not suitable for patients with anatomical anomalies. For example, patients who have undergone liver resection have a significantly different liver shape to other subjects. A possible solution for these specific cases has been proposed in (Lee et al., 2010). Here, limited optimal scan planes were determined by analyzing the preoperative and patientspecific 3D SSM of the liver with principal component analysis (PCA). The relationship between preoperative 3D SSM and synchronized 2D SSM constructed from 2D images at the optimal scan planes was learned by partial least squares regression (PLSR). Finally, with new intraoperative 2D images obtained at the same scan planes, the 3D shape was instantiated intraoperatively by applying the PLSRderived relationship. However, in (Lee et al., 2010), the optimal scan plane determination depended on the selected vertices that were deemed informative but were highly correlated and clustered. PLSR can only derive linear relationships while the deformations of most anatomies are nonlinear. Based on (Lee et al., 2010), a framework which achieves more accurate, robust, generalizable and convenient shape instantiations from a single intraoperative 2D image is proposed in this paper.
Subspace reprojection was proposed to determine an optimal scan plane for SSMbased 3D shape instantiation by fitting a plane to the most informative vertices (Lee et al., 2005). This optimal scan plane was shown to have enhanced accuracy compared to other scan planes(Lee et al., 2005). By applying PCA (Jolliffe, 2002) on the preoperative 3D SSM, the informative vertices which contribute most to the shape variations are determined by the loadings of principal components (Lee et al., 2010). The downside of using PCA is that the derived principal components are linear combinations of multiple variables and therefore the selected informative variables are highly related and difficult to interpret. This phenomenon when reflected in our application is that the selected informative vertices are clustered and are not the real and independent informative vertices. Many methods have been proposed to solve this issue, including rotation methods (Jolliffe, 1995), limited set of integers (Vines, 2000), and simplified component technique least absolute shrinkage and selection operator (SCoTLASS) (Jolliffe et al., 2003). Simple thresholding is a common and informal method usually used in practice (Lee et al., 2010); however, this method lacks theoretical support and usually causes problems (Cadima & Jolliffe, 1995). Recently, Zou et al. proposed sparse PCA (SPCA) which reformulated PCA into a regressiontype optimization problem and then added a L1 constraint to achieve sparse loadings; they demonstrated improved performance of SPCA in selecting the real informative variables over previous methods (Zou et al., 2006). A SPCA toolbox was later developed (Sjöstrand et al., 2012).
PLSR is a linear regression method which has a similar prediction accuracy to ridge regression (RR) and principal component regression (PCR)
(Frank & Friedman, 1993). It is more widely used than RR and PCR in medical problems, such as cardiac motion prediction (Ablitt et al., 2004) and craniofacial reconstruction (Duan et al., 2015), as it is more suitable for problems with a larger number of variables and fewer number of observations (Rosipal & Trejo, 2001). However, its accuracy for nonlinear motions is limited.Many nonlinear PLSR variations have been proposed and they can be divided into two groups (Rosipal & Krämer, 2006)
: the first group reformulates the linear relationship into a nonlinear one by polynomial functions, smoothing splines, artificial neural networks, and radial basis function networks while the second group maps the original variables into a higher dimensional space and regresses the mapped variables in the higher dimension, for example, kernel space. Kernel PLSR (KPLSR)
(Rosipal & Trejo, 2001) from the second group is adopted in this paper for improved computation speed as its formulation is as timeefficient as PLSR and avoids the nonlinear optimization in the first group.In this paper, the highresolution 3D shape of a dynamic anatomy was instantiated from a single intraoperative 2D image in realtime. Firstly, the anatomy was scanned by MR or CT preoperatively for multiple 3D volumes along the dynamic cycle and a 3D SSM was constructed. SPCA was applied on the preoperative 3D SSM to select the informative vertices which were used to fit an optical scan plane. Local adjustments of the scan plane parameters for better accessibility, visibility or satisfying other local constraints is possible without incurring major errors, as the later KPLSRbased 3D shape instantiation scheme is robust to optimal scan plane derivations. Secondly, 2D images synchronized with the preoperative scanning were obtained at the approximate optimal scan plane and were sampled to generate a synchronized 2D SSM. KPLSR was applied to learn the relationship between the preoperative 3D SSM and the synchronized 2D SSM. Finally, the highresolution 3D shape was instantiated intraoperatively by applying the KPLSRderived relationship onto a new intraoperative 2D image at the same scan plane. The overall framework of the proposed dynamic shape instantiation is illustrated in Fig. 1. Due to the learning of patientspecific models, the framework is applicable to any anatomy. No extra registration is needed for the preoperative 3D SSM and the synchronized 2D SSM. Validation was performed on the liver (two digital liver phantoms, one dynamic liver phantom, one in vivo porcine liver, eight metastatic livers) and the cardiac right ventricle (RV) (18 asymptomatic RVs and 9 hypertrophic cardiomyopathy (HCM) RVs); we anticipate that potential applications of our work will include percutaneous liver biopsy, cardiac catheterization (Razavi et al., 2003), and intramyocardial therapy (Saeed et al., 2005). For example, in cardiac ablation, the instantiated 3D RV shape can be used to help navigate the catheter tip to the target ablation area.
2 Methodology
The methods for determining the optimal scan plane are described in Sec. 2.1. The learning and instantiation based on KPLSR are described in Sec. 2.2. Finally, the data collection and detailed validation experiments are in Sec. 2.3.
2.1 Optimal Scan Plane Determination
By preoperatively scanning the target anatomy with CT or MR, a 4D volume consisting of multiple 3D volumes at different time frames along the dynamic cycle of the anatomy was obtained. These 3D volumes were represented with 3D meshes using the same number of vertices and connectivities, which created a preoperative 3D SSM (a point distribution model) with vertices , where is the number of time frames and is the number of vertices. By rearranging the coordinates of the vertices as independent variables, was obtained, where is the number of variables. Without loss of generality, was centered and normalized as with the mean and norm of each column as 0 and 1.
For data
, its singular value decomposition is
, where are the principal components and are the loadings of the principal components. The principal component represents the mode of variation in the anatomical deformation while the corresponding loadings represent the contribution of each variable to this mode of variation (Lee et al., 2005). The calculated by PCA are usually all nonzero values and hence the selected informative vertices are highly related and clustered. The aim of SPCA is to achieve a sparse . can be recovered by:(1) 
Here, , is a manually set positive parameter. With the addition of an constraint, Eq. 1 becomes:
(2) 
where and is a manually set parameter which controls the sparsity or the number of nonzero values of . Eq. 2 can be solved with a fixed and any by least angle regression elastic net (LARSEN) efficiently (Zou & Hastie, 2005).
However, Eq. 2 is still based on PCA due to the inclusion of . To solve this, a twostage exploratory analysis was formulated with PCA initialization and then optimization with sparse approximations.
With  the row of :
(3) 
When , then ; the detailed proof can be found in (Zou et al., 2006). If the first principal components and the lasso penalty are included, Eq. 3 becomes
(4) 
Here, are the loadings of the first principal components of PCA, when . Then are the approximated sparse loadings of .
The complete SPCA algorithm is listed in Algorithm 1. The approximated sparse loadings is a matrix with the loading or contribution of each variable to the principal component or mode of variation. The parameter controls the sparsity or the number of nonzero values in . As suggested in (Lee et al., 2005), the contribution of three coordinates (x, y, z) at was added together to represent the vertex contribution. The vertices at all time frames with nonzero contribution were selected as the informative vertices. A plane with the minimum sum of distances to all informative vertices was determined as the optimal scan plane. When calculating the sum, each distance was weighted by the vertex contribution. For multiple scan planes, can be used to determine the optimal scan plane; however, this is out of the scope of this paper as we are targeting a single scan plane.
In (Lee et al., 2010), the real scan planes were registered to the optimal scan planes. In this paper, as the proposed KPLSRbased 3D shape instantiation is robust to local scan plane deviations, which will be shown in Sec. 3.2, the final scan plane is an approximate one that is both accessible and convenient for imaging with parameters near the optimal scan plane. When the deformations or shapes of the anatomy are significantly different between patients and hence there are significantly different optimal scan planes between patients, such as the metastatic liver after oncological surgery, the optimal scan plane needs to be determined on a patientspecific basis. When the deformations or shapes of the anatomy are similar between patients and hence there are similar optimal scan planes between patients, such as the RV, the trend of the optimal scan planes for multiple patients is determined as a general optimal scan plane for the anatomy and will be used directly in subsequent interventional procedures, thus reducing the workload for clinicians. Detailed optimal scan plane determination and approximation in our experiments are given in Sec. 2.3.
SPCA 
Initialize : 
the loadings of the first principal components from PCA 
Initialize 
For j=1:k 
If , which has not converged 
Solve the following minimum by LARSEN: 
Update with the normalized new 
Update with the normalized new : 
End 
End 
Approximated sparse loadings 
2.2 3D Shape Instantiation
With the preoperative 3D SSM and the approximate optimal scan plane obtained, 2D images synchronized with the time frames for preoperative scanning were obtained at the approximate optimal scan plane. The 2D anatomical contours were segmented and sampled to the same number of 2D vertices and connectivities, resulting in a 2D SSM with vertices , where is the number of time frames and is the number of vertices. By rearranging the (x, y) coordinates of vertices as independent variables, was obtained, where is the number of variables and typically . The 3D volumes and 2D images do not need to be registered. KPLSR is then applied to learn the relationship between the 3D SSM which is the response in regression and the 2D SSM which is the predictor in regression. For 3D shape instantiation, the new intraoperative 2D image is obtained at the same scan plane and is sampled into the same number of vertices and connectivities as that for the original 2D SSM with vertices . is obtained by rearranging the coordinates as independent variables for applying the KPLSRderived relationship to predict the intraoperative response whose coordinates are then rearranged back to obtain the intraoperative 3D shape .
In the following sections, we introduce PLSR and show its extension to KPLSR.
2.2.1 Plsr
PLSR extracts the relationship between two matrices which could have different dimensions. With predictors and responses , PLSR finds the relationship:
(5) 
Here, is the prediction of . The latent variables in are determined by to explain the latent variables in maximally. is later used to predict the intraoperative response from . Nonlinear iterative partial least squares (NIPALS) is a widely applied PLSR algorithm (Rosipal & Trejo, 2001). In this paper, an alternative algorithm  SIMPLS (De Jong, 1993)  was used for increased timeefficiency.
Without loss of generality, both and are centered with the mean of each column as , which are expressed by and respectively below. The main problem for SIMPLS is to compute the weight factors and , where and is a manually set parameter denoting the number of components used. and maximize the covariance of and with the following four conditions:
1. maximized covariance: ,
2. normalized : ,
3. normalized : ,
4. orthogonalized :
To satisfy the fourth condition, , where . When , any new must be orthogonal to . This orthogonal projector is , where
is an identity matrix. The SIMPLS algorithm is listed in Algorithm
2:SIMPLS 
Initialize , are the centered matrix of respectively 
for (M is a manually set parameter  the number of components used) 
if 
first left singular vector of SVD of , () 
else 
first left singular vector of SVD of 
end 
() 
() 
end 
Coefficient: , where 
2.2.2 Kplsr
PLSR is less suitable for regressing nonlinear motions. KPLSR was used to compensate for this shortage. A kernel function maps the predictor into a new feature space nonlinearly with . satisfies the kernel trick: . PLSR is then constructed in the feature space to achieve a nonlinear regression for (Rosipal & Trejo, 2001).
The kernel used in this paper was a Gaussian kernel for its increased accuracy over a polynomial kernel:
(6) 
Here . , the Gaussian width, was adjusted to to facilitate parameter adjustment between different targets and subjects, is a manually set ratio which we term the Gaussian ratio. Substituting the in Algorithm 2 with gives us the algorithm for KPLSR.
2.3 Data Collection and Validation
The proposed framework was validated on both liver and cardiac RV studies. The experiments included two digital liver phantoms, one dynamic liver phantom, one in vivo porcine liver, eight livers from metastatic patients, 18 cardiac RVs from asymptomatic subjects, and 9 cardiac RVs from HCM patients.
The acquisition of 3D meshes and synchronized 2D contours at different time frames along the dynamic cycle for each data are given in Sec. 2.3.2  2.3.5. All data used the same methods to construct the 3D and 2D SSM. With known 3D shapes consisting of 3D vertices and connectivities at different time frames, the midstate 3D mesh was first projected to meshes at other time frames by nonrigid registration (Manu, 2016). Then the registered midstate 3D mesh was mapped onto meshes at other time frames by projecting its vertices along the normal directions. Therefore a 3D SSM with point correspondences was constructed. With known 2D contours consisting of 2D vertices and connectivities at different time frames, the construction of a 2D SSM was in the same way as that for a 3D SSM but with a different registration method (Kroon, 2016).
2.3.1 Digital Livers
XCAT is a digital whole body phantom with detailed, highresolution and dynamic tissues (Segars et al., 2010) as shown in Fig. 2a and Fig. 2b. In this paper, the isotropic resolution of the volume was set at . 21 time frames were collected between exhalation and inhalation. 3D meshes of two XCAT livers (one male and one female) were segmented and processed with Analyze (AnalyzeDirect, Inc, Overland Park, KS, USA) and MeshLab (Cignoni et al., 2008). A 3D SSM was constructed for each digital liver.
The optimal scan plane for each liver was determined with approximately informative vertices and was used to slice the meshes in the 3D SSM. The intersection contours were projected onto the slicing plane to simulate 2D contours. A 2D SSM was constructed for each liver.
2.3.2 Dynamic Liver Phantom Experiment
A detailed female phantom modeled with silicone organs (the Regina model (Lerotic & Lee, 2010)) was used. The lungs were modified to simulate respiratory motion. In each lung, foam board inserts (each thick) were used, creating seven different liver deformation positions. Each respiratory position was scanned in a Siemens 64 slice SOMATOM™ Sensation CT Scanner with images of inplane resolution and slice separation. Segmentation and processing were performed with Analyze and MeshLab.
For realtime scanning, ultrasound imaging was used. A 2D imaging transducer used with the ALOKA prosound system (Aloka Co. Ltd, Tokyo, Japan) was affixed with an NDI Polaris passive infrared tracker (Northern Digital, Inc, Waterloo, ON, Canada), enabling the recording of the spatial position and orientation of the scan plane. Calibration between systems was established by registering three known landmarks on the liver phantom in both frames of reference. The ultrasound images were captured from the Svideo output feed of the scanner. The experiment setup is shown in Fig. 2c and Fig. 2d.
Freehand 3D ultrasound systems require calibration and a number of techniques and corresponding phantoms have been developed for this (Mercier et al., 2005). To calibrate the ultrasound images to the coordinate space of a tracking device, a threepoint crossed wires phantom was built. The transforms from the coordinate space of the optical tracker to that of the CT imaging space were calculated by PRAXIS (Gegenfurtner, 1992). This defined a translation and a quaternion for the rotation between the ultrasound image points and the CT imaging space (Prager et al., 1998). The mean distance between the registered ultrasound image points and the 3D meshes scanned by CT is less than .
Due to the constraints caused by the rib cage, the optimal scan plane fitted with 30 informative vertices was selected as the actual scan plane. The silicone phantom was filled with water. For each respiratory position, the optimal scan plane was acquired with the ultrasound probe. An experienced operator scanned the phantom using an inhouse guidance system where the silicone liver was registered to a 3D guidance mesh by three manually chosen points. This guidance system provided the actual scan plane in realtime as well as the desired scan plane orientation.
A semiautomatic segmentation based on active contours (Geiger et al., 1995) was used to delineate the liver contour from the 2D ultrasound images. It could determine a contour in an ultrasound image automatically when the two end points were selected manually. The contours were transformed to the CT coordinate frame to achieve registered contour coordinates from which a registered 2D SSM was constructed. The registration between 3D volumes and 2D images was only performed for the Regina phantom for later specific comparison and was not performed for all other data.
2.3.3 Porcine Liver
One contrast enhanced 3D CT scan was captured at full exhalation using a GE Innova 4100 interventional Xray machine capable of fluoroCT imaging. Due to the respirator design, the porcine liver could not be stopped at different respiratory positions for a 3D CT scan. Instead, fluoroscopic images were obtained in an anteroposterior (AP) direction over time to cover the animal’s respiratory motion. As only one 3D volume at full exhalation was scanned with CT, 3D volumes at other respiration positions were simulated by image constrained finite element modeling (FEM) (Lerotic et al., 2009) while the collected fluoroscopic images were used as the image constraints. This created multiple liver 3D meshes at different time frames. In this paper, the surface mesh at full exhalation was first turned into a tetrahedral mesh using Gmsh (Geuzaine & Remacle, 2009). Then, the Open Source SOFA framework (Allard et al., 2007), chosen for its emphasis on realtime medical simulations, was used for the FEM. The material for the liver was set to be elastic and isotropic, with a Young’s modulus of 640 Pa and Poisson’s ratio of 0.3 (Yeh et al., 2002). A 3D SSM was constructed for the porcine liver.
The meshes in the 3D porcine liver SSM were sliced by the optimal scan plane determined with approximately informative vertices. The sliced contours were projected onto the slicing plane with 2D coordinates to simulate 2D contours. A 2D SSM was constructed for the porcine liver as well.
2.3.4 Metastatic Livers
Clinical data from eight patients (6 male, 2 female, mean age 63) with metastatic liver tumors was collected. 4D volumes were scanned using a 1.5T MR scanner (Intera, Philips, Amsterdam, Netherland) using a T1 weighted freebreathing sequence (TR = , TE = , inplane resolution, slice thickness). Each volume consisted of slices and was acquired in approximately . time frames were collected to cover the liver motion during respiration. Due to motion artifacts caused by respiration, we could only confidently segment the livers at full inhalation and full exhalation. As before, the SOFA framework was used to generate the meshes at different respiratory positions but with the 3D volumes at full inhalation and full exhalation as the constraints. These meshes were used to construct a 3D SSM for each patient.
The eight metastatic patients have significantly different liver shapes and deformations, as shown in Fig. 3 and the optimal scan planes for each patient were very different. For this reason, for the metastatic liver, the optimal scan plane was determined patientspecifically with approximately informative vertices for each patient and this was used to slice the meshes in the 3D SSM. The sliced contours were projected onto the slicing plane with 2D coordinates to simulate 2D contours. A 2D SSM was constructed for each patient.
2.3.5 Cardiac Data
27 subjects (18 asymptomatic subjects (Subjects 118) and 9 patients with HCM (Subjects 1927)) were scanned with a 1.5T MR scanner (Sonata, Siemens, Erlangen, Germany). HCM was selected as it is one of the diseases that influence both the shape and deformation of the heart significantly. Shortaxis cine sequences from the atrioventricular ring to the apex were scanned with a slice gap and a pixel spacing. time frames were collected. To recognize the slice location of the atrioventricular ring and the apex, the slice gap was interpolated to in Analyze. 3D RV meshes were segmented and built with Analyze and MeshLab. A 3D SSM was constructed for each patient.
Even though HCM influences both the shape and deformation of the RV, the optimal scan planes for the 27 subjects, which were determined with approximate informative vertices each, were mostly found to be along the long axis of the heart. Four examples are shown in Fig. 4. Even though the optimal scan planes in Fig. 4 are not exactly the same, they still share the same trend lying along the long axis of the heart. This similarity of the optimal scan planes between patients is mainly due to the similarity in deformation and shape of the RVs between patients. As later KPLSRbased 3D shape instantiation is robust to optimal scan plane deviations, we made an adjustment to the optimal scan plane to ensure the accessibility of the scan plane and the visibility of the RV considering the following three issues: 1) The longaxis is accessible for 2D MR, 2) The horizontal (fourchamber) longaxis has a clear view of the RV without overlap with other chambers, and 3) Clinicians are familiar with this plane as it features the apex and the atrioventricular ring. For these reasons, the horizontal (fourchamber) longaxis plane was selected as the actual scan plane for all RVs.
2D MR images at the horizontal (fourchamber) longaxis plane with the synchronized time frames as that for the preoperative 4D volume scanning were obtained for all 27 subjects. Analyze was used to segment the RV contours and a 2D SSM was constructed for each subject.
2.3.6 Validation
Leaveoneout cross validation was applied for all time frames for all data. The time frame in the 2D SSM was left out as a new predictor while the time frame in the 3D SSM was left out as the ground truth. All other time frames were used in the learning. The error was calculated as the Euclidean vertextovertex distance between the 3D prediction and the ground truth. The shape variation was calculated as the mean vertextovertex distance between the and the time frame in the 3D SSM.
It was shown that SPCA was able to better select the real and unrelated informative variables than PCA on a synthetic example (Zou et al., 2006). For the synthetic example, the contribution of a variable and the relations between variables were known. However, for practical data, both this contribution and the relations were unknown; a comparison of the distribution of the informative vertices selected by PCA and SPCA is given in this paper. In practice, adjusting the optimal scan plane is usually necessary for better scan plane accessibility and target visibility. To illustrate that this adjustment will not incur major errors, multiple deviated optimal scan planes were used to instantiate the 3D shape.
Both PLSR and KPLSR regress the relationships between two matrices rather than two coordinate frames. Lee et al. applied PLSR with registration of preoperative 3D SSM and synchronized 2D SSM (Lee et al., 2010). In this paper, this explicit registration is not required. To demonstrate this, both the registered and nonregistered 2D SSM of the dynamic phantom liver were used as the predictor for dynamic shape instantiation. The stability of an algorithm with respect to its parameters is important for judging its performance. PLSR has one parameter, the number of components used, while KPLSR has two parameters, the number of components used and the Gaussian ratio. To evaluate the stability of PLSR and KPLSR to the number of components used, the validation was applied on two HCM RVs with the number of components used set from . In practice, the time frames at or near the boundaries, i.e. at maximal inhalation and exhalation or at diastole and systole, are the most difficult time frames to recover, as the learning is more weak for these time frames. We term these time frames boundary time frames. In this paper, the performance of PLSR and KPLSR at boundary time frames were validated on two cardiac RVs (one asymptomatic RV and one HCM RV). The liver data was collected along half of the dynamic cycle  the first and last few time frames are the inhalation and exhalation respectively, i.e. the boundary time frames. The cardiac data was collected along the entire dynamic cycle, the first and last few time frames are at diastole while the middle few time frames are at systole, i.e. the boundary time frames.
Finally, the accuracy of the proposed dynamic shape instantiation was tested on two digital livers, one in vivo porcine liver, eight metastatic liver patients, and 27 RVs of asymptomatic subjects and HCM patients.
3 Results
The results from our experiments are shown in the following sections. The comparison between PCA and SPCA on selecting informative vertices is demonstrated in Sec. 3.1. The robustness of the KPLSRbased 3D shape instantiation to scan plane deviations is shown in Sec. 3.2. The validation on releasing the registration between preoperative 3D SSM and synchronized 2D SSM is illustrated in Sec. 3.3. The stability of PLSR and KPLSR to the number of components used is compared in Sec. 3.4. Boundary time frames are tested in Sec. 3.5. Finally, the accuracy of the proposed dynamic shape instantiation is validated on the liver and the heart, which is shown in Sec. 3.6.
3.1 Comparison between PCA and SPCA
For most subjects, including the metastatic livers and cardiac RVs, it was found that the informative vertices selected by PCA were more clustered than the informative vertices selected by SPCA. Three examples are shown in Fig. 5. Clustered informative vertices were selected by PCA due to their related motion with the informative vertices considered to be in the same area. SPCA can remove this interrelation and only select the true and sparse informative vertices.
3.2 Robustness to Scan Plane Deviations
To demonstrate the robustness of the proposed KPLSRbased 3D shape instantiation to scan plane deviations, example RV results from Subject 3 are illustrated below. 13 scan planes with some deviations from the optimal scan plane were used to slice the preoperative 3D SSM for 3D shape instantiation. The distance error and the deviation for each scan plane is shown in Fig. 6a and Fig. 6b respectively. We can see that the achieved accuracy was scarcely influenced by local scan plane deviations, demonstrating the robustness of the proposed KPLSRbased 3D shape instantiation to scan plane deviations. This is important for practically implementing the proposed framework, as due to practical constraints (access window, or other local, physical constraints), it may be necessary to deviate slightly from the theoretical optimal scan plane. Such deviation should not induce large changes in instantiation errors.
3.3 Validation of RegistrationFree Instantiation
The instantiation accuracy across all time frames with registered and nonregistered predictors which were collected in the liver phantom experiment is shown in Fig. 7. It can be seen that PLSR is influenced by the registration while KPLSR shows little influence, demonstrating that explicit registration is not required in the proposed method.
3.4 Stability to the Number of Components Used
Instantiation for two HCM patients (Subject 21 and Subject 27) was calculated along all time frames with a varying number of components used (), as shown in Fig. 8a and Fig. 8d. It can be seen that the accuracy of KPLSR is less sensitive to this parameter  the number of components used  than that of PLSR, as the standard deviations of KPLSR are less than that of PLSR at most time frames. Two time frames (time frame of Patient 21, time frame of Patient 27) are shown with the mean distance errors at different numbers of components used in Fig. 8b and Fig. 8e. Two instantiation examples colored by the distance errors are shown in Fig. 8c and Fig. 8f. The error is distributed evenly over the mesh.
3.5 Performance at Boundary Time Frames
The mean distance errors for shape instantiation along all time frames are shown for two selected subjects (Subject (asymptomatic) and Subject (HCM)) in Fig. 9a. The PLSR errors show large peaks near systole (time frame 10 for Subject 6, time frame 11 for Subject 19) and diastole (time frame and for Subject 6, time frame and for Subject 19) while KPLSR errors show smaller increasing errors at these boundary time frames. It can be concluded that KPLSR has better performance at boundary time frames than PLSR.
3.6 Accuracy of Dynamic Shape Instantiation
Mean distance errors of PLSR and KPLSR and the shape variation of two digital phantom livers and one porcine liver are shown along all time frames in Fig. 10. For the two digital livers, KPLSR achieved much lower errors at the time frames where PLSR showed high peaks. For the porcine data, the accuracy of KPLSR is higher than that of PLSR at most time frames. For both digital phantom and porcine liver studies, the mean distance error of KPLSR is much lower than the shape variation at most time frames. The peaks for KPLSR (time frames 1819 in Fig. 10b and time frames 12 in Fig. 10c) were caused by boundary time frames. The higher accuracy of KPLSR for the two digital livers is not as obvious as that for the porcine liver due to the design and linear deformation of the digital phantom.
Eight patients with metastatic tumors were used for instantiation validation with the mean distance errors of PLSR and KPLSR and the shape variation shown along all time frames in Fig. 11. For most of the time frames and patients, KPLSR achieved much more accurate instantiation results than those of PLSR. The mean distance errors of KPLSR were also much lower than the shape variation. The higher errors of KPLSR (time frames 2930 for P1, time frames 13 and 29 for P4, time frame 22 for P5, time frame 14 for P7) were caused by boundary time frames.
Shape instantiation of 27 RVs was validated with the mean distance errors of PLSR and KPLSR and the shape variation shown in Fig. 9b; the standard deviation in the graphs was from the error variation along different time frames. Overall, KPLSR achieves both lower mean and standard deviation errors in the instantiation than PLSR for all subjects. The error achieved by KPLSR was also much lower than the shape variation for all subjects. The similar results between patients also demonstrate the availability of using one approximate optimal scan plane  the horizontal (fourchamber) longaxis plane for all RVs in this paper.
For optimal scan plane determination, the number of informative vertices was determined as of the total number of vertices in each test mesh, the parameter was fixed at 0.0001, was set at as we are targeting a single scan plane, and the parameter was set as the number of informative vertices. For 3D shape instantiation, with the exception of the test for stability to the number of components used, all tests were validated with the number of components used for PLSR optimized between while that for KPLSR was empirically set between . Overall, KPLSR achieved better accuracy at a higher number of components used than PLSR. The Gaussian ratio parameter of KPLSR was selected empirically.
Experiments were performed in MATLAB on an Intel(R) Core(TM) i74790 CPU @3.60Hz computer. The training took approximately for one component deflation; the number of component deflations is the number of components used. The prediction or shape instantiation took approximately .
4 Discussion
In this paper, SPCA was applied instead of PCA to determine the informative vertices to find the optimal scan plane. We expect that nearby points on the surface of organs will tend to move dependently in a similar fashion. This is because the movement of one cell will cause the movement of its nearby cells due to the connectivity of tissues. the sparse informative vertices determined by SPCA and the clustered informative vertices determined by PCA could illustrate the ability of SPCA to derive principal components from unrelated original variables and hence select the true, unrelated informative vertices. However, from our experiments, the overall trend of the informative vertices selected by PCA was shown to be similar to the trend determined by SPCA. It is more reasonable to conclude that SPCA facilitates the determination of the optimal scan plane more clearly and quickly than PCA rather than more accurately in this case. Setting a higher number of informative vertices when applying PCA could also achieve a good scan plane.
In practical applications, the calculated optimal scan plane is not always accessible. The robustness of the proposed KPLSRbased 3D shape instantiation to local scan plan deviations ensures the adjustment of the scan plane for better accessibility and visibility in practical clinical scenarios. The optimal scan plane for the RV, which will be used directly for future patients, was determined by analyzing the pattern of the optimal scan planes for 27 RVs. This method of determining the optimal scan plane could be adopted for other anatomies which share similar deformation and shape across patients. For anatomy such as the metastatic liver which has significantly different deformation and shape between patients, the optimal scan plane has to be determined for each patient individually.
The registration between the preoperative 3D SSM and synchronized 2D SSM is no longer required in this paper. The validation on a liver phantom experiment with both registered and nonregistered predictors showed that the accuracy of KPLSR was not influenced by this. The removal of explicit registration will decrease the workload for clinicians significantly when applying the proposed method in practice. It was proved that KPLSR had much higher stability to the number of components used than PLSR. This is important during practical applications in case of the use of a suboptimal setting of this parameter. KPLSR also had better processing at boundary time frames than PLSR though the errors of KPLSR at boundary time frames are still higher than at other normal time frames. This boundary limitation corresponded to more time frames for the liver data than the cardiac data, as the SOFA framework generated meshes at the first few and last few time frames with very small shape variations which were usually less than . In practical applications, always including the time frames at maximum inhalation and exhalation or at systole and diastole in the training data is highly recommended.
As preoperative 4D volumes are not typically acquired for livers, FEM was applied to simulate the meshes between the inspiration and expiration. FEM or any other methods which could simulate the physical organ motion can thus be used to generate preoperative 4D volumes when transferring the proposed framework onto other target anatomies whose dynamic motion is difficult to gate.
In general, three kinds of data are needed to apply the proposed 3D shape instantiation: the 3D SSM for learning, the 2D SSM for learning, and the 2D intraoperative images for prediction. Synchronization is needed between the learning 3D SSM and the learning 2D SSM while registration is needed between the learning 2D SSM and the 2D intraoperative images for prediction. The 4D volume used for constructing the learning 3D SSM was scanned preoperatively while the 2D images used for constructing the learning 2D SSM could be scanned preoperatively or intraoperatively, as the learning only takes a few seconds. In practical applications, for organs whose motion could be gated easily, i.e. the RV, the synchronization between the learning 3D SSM and the learning 2D SSM could be achieved through dynamic motion gating, i.e. electrocardiogram (ECG) gating or respiratory gating. The registration between the learning 2D SSM and the intraoperative 2D images for prediction could be achieved by setting the scan machine at the same scan position. For organs whose motion is difficult to gate, i.e. the liver, FEM or other available methods which could simulate the 3D volumes at different time positions could be used to collect the learning 3D SSM and to slice for the learning 2D SSM. The registration between the learning 2D SSM and the intraoperative 2D images for prediction could be achieved by setting the scan machine to the same scan position as that used to slice the learning 2D SSM.
Two digital livers, one porcine liver, and eight metastatic livers were used to illustrate the applicability of the proposed method on livers. As well, 27 RVs were used in our validation with real 2D MR images as the predictors, which demonstrates the potential value of the proposed method in practical operations. Even with only a single scan plane, a mean distance error of about was achieved for the RV. This error was comparable to the mean accuracy in (Gao et al., 2012) and (Huang et al., 2009) which were approximately and for patients and animals, respectively. The computation time for prediction () demonstrates the realtime ability of the proposed method.
5 Conclusions
In conclusion, a realtime and registrationfree framework for dynamic shape instantiation which is generalizable to multiple anatomies is proposed in this paper. SPCA is applied to select the unrelated and real informative vertices from a preoperative 3D SSM, which facilitates a more clear and quick determination for the optimal scan plane. KPLSR is used to improve the accuracy and robustness of the instantiation. For anatomies like the RV, the optimal scan plane only needs to be determined once and then can be used in subsequent interventions. The detailed experiments performed for the removal of explicit registration, the stability to the number of components used, and the performance at boundary time frames covers the issues which may occur during practical applications. FEM extends the application of the framework to anatomies like the liver, whose motion is difficult to gate. The patientspecific learning removes the restrictions on the applicable anatomy. This paper sets the basis for applying the proposed framework to other interventional procedures involving dynamic anatomies.
6 Acknowledgements
We would like to thank Dr Karim Lekadir and Dr Robert Merrifield for supplying the cardiac data, Dr Maria Hawkins and Dr Diana Tait for the liver patient data, and Dr Mirna Lerotic for her assistance with the finite element simulations. This research was partly supported by the Engineering and Physical Sciences Research Council UK (EP/L020688/1).
References
References
 Ablitt et al. (2004) Ablitt, N. A., Gao, J., Keegan, J., Stegger, L., Firmin, D. N., & Yang, G.Z. (2004). Predictive cardiac motion modeling and correction with partial least squares regression. IEEE transactions on medical imaging, 23, 1315–1324.
 Allard et al. (2007) Allard, J., Cotin, S., Faure, F., Bensoussan, P.J., Poyer, F., Duriez, C., Delingette, H., & Grisoni, L. (2007). Sofa  an open source framework for medical simulation. In MMVR 15Medicine Meets Virtual Reality (pp. 13–18). IOP Press volume 125.
 Baka et al. (2011) Baka, N., Kaptein, B., de Bruijne, M., van Walsum, T., Giphart, J., Niessen, W. J., & Lelieveldt, B. P. (2011). 2D3D shape reconstruction of the distal femur from stereo Xray imaging using statistical shape models. Medical image analysis, 15, 840–850.
 Barratt et al. (2008) Barratt, D. C., Chan, C. S., Edwards, P. J., Penney, G. P., Slomczykowski, M., Carter, T. J., & Hawkes, D. J. (2008). Instantiation and registration of statistical shape models of the femur and pelvis using 3D ultrasound imaging. Medical image analysis, 12, 358–374.
 Cadima & Jolliffe (1995) Cadima, J., & Jolliffe, I. T. (1995). Loading and correlations in the interpretation of principle compenents. Journal of Applied Statistics, 22, 203–214.
 Cignoni et al. (2008) Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., & Ranzuglia, G. (2008). MeshLab: an opensource mesh processing tool. In V. Scarano, R. D. Chiara, & U. Erra (Eds.), Eurographics Italian Chapter Conference. The Eurographics Association. doi:10.2312/LocalChapterEvents/ItalChap/ItalianChapConf2008/129136.
 Cool et al. (2006) Cool, D., Downey, D., Izawa, J., Chin, J., & Fenster, A. (2006). 3D prostate model formation from nonparallel 2D ultrasound biopsy images. Medical Image Analysis, 10, 875–887.
 Cootes et al. (1995) Cootes, T. F., Taylor, C. J., Cooper, D. H., & Graham, J. (1995). Active shape modelstheir training and application. Computer vision and image understanding, 61, 38–59.
 De Jong (1993) De Jong, S. (1993). SIMPLS: an alternative approach to partial least squares regression. Chemometrics and intelligent laboratory systems, 18, 251–263.
 Duan et al. (2015) Duan, F., Huang, D., Tian, Y., Lu, K., Wu, Z., & Zhou, M. (2015). 3D face reconstruction from skull by regression modeling in shape parameter spaces. Neurocomputing, 151, 674–682.
 Filippi et al. (2008) Filippi, S., Motyl, B., & Bandera, C. (2008). Analysis of existing methods for 3D modelling of femurs starting from two orthogonal images and development of a script for a commercial software package. Computer methods and programs in biomedicine, 89, 76–82.
 Frangi et al. (2002) Frangi, A. F., Rueckert, D., Schnabel, J. A., & Niessen, W. J. (2002). Automatic construction of multipleobject threedimensional statistical shape models: Application to cardiac modeling. IEEE transactions on medical imaging, 21, 1151–1166.
 Frank & Friedman (1993) Frank, L. E., & Friedman, J. H. (1993). A statistical view of some chemometrics regression tools. Technometrics, 35, 109–135.
 Gao et al. (2012) Gao, G., Penney, G., Ma, Y., Gogin, N., Cathier, P., Arujuna, A., Morton, G., Caulfield, D., Gill, J., Rinaldi, C. A., Hancock, J., Redwood, S., Thomas, M., Razavi, R., Gijsbers, G., & Rhode, K. (2012). Registration of 3D transesophageal echocardiography to Xray fluoroscopy using imagebased probe tracking. Medical image analysis, 16, 38–49.
 Gegenfurtner (1992) Gegenfurtner, K. R. (1992). Praxis: Brent’s algorithm for function minimization. Behavior Research Methods, Instruments, & Computers, 24, 560–564.
 Geiger et al. (1995) Geiger, D., Gupta, A., Costa, L. A., & Vlontzos, J. (1995). Dynamic programming for detecting, tracking, and matching deformable contours. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17, 294–302.
 Geuzaine & Remacle (2009) Geuzaine, C., & Remacle, J.F. (2009). Gmsh: A 3D finite element mesh generator with builtin preand postprocessing facilities. International Journal for Numerical Methods in Engineering, 79, 1309–1331.
 Huang et al. (2009) Huang, X., Moore, J., Guiraudon, G., Jones, D. L., Bainbridge, D., Ren, J., & Peters, T. M. (2009). Dynamic 2D ultrasound and 3D CT image registration of the beating heart. IEEE transactions on medical imaging, 28, 1179–1189.
 Jolliffe (2002) Jolliffe, I. (2002). Principal component analysis. Wiley Online Library.
 Jolliffe (1995) Jolliffe, I. T. (1995). Rotation of principal components: choice of normalization constraints. Journal of Applied Statistics, 22, 29–35.
 Jolliffe et al. (2003) Jolliffe, I. T., Trendafilov, N. T., & Uddin, M. (2003). A modified principal component technique based on the lasso. Journal of computational and Graphical Statistics, 12, 531–547.
 Karade & Ravi (2015) Karade, V., & Ravi, B. (2015). 3D femur model reconstruction from biplane Xray images: a novel method based on laplacian surface deformation. International journal of computer assisted radiology and surgery, 10, 473–485.
 Koh et al. (2011) Koh, K., Kim, Y. H., Kim, K., & Park, W. M. (2011). Reconstruction of patientspecific femurs using Xray and sparse CT images. Computers in biology and medicine, 41, 421–426.
 Kroon (2016) Kroon, D.J. (2016). Shape context based corresponding point models. https://uk.mathworks.com/matlabcentral/fileexchange/30845shapecontextbasedcorrespondingpointmodels. Accessed: 20160220.
 Lee et al. (2010) Lee, S.L., Chung, A., Lerotic, M., Hawkins, M., Tait, D., & Yang, G.Z. (2010). Dynamic shape instantiation for intraoperative guidance. In International Conference on Medical Image Computing and ComputerAssisted Intervention (pp. 69–76). Springer.
 Lee et al. (2005) Lee, S.L., Horkaew, P., Caspersz, W., Darzi, A., & Yang, G.Z. (2005). Assessment of shape variation of the levator ani with optimal scan planning and statistical shape modeling. Journal of computer assisted tomography, 29, 154–162.
 Lerotic & Lee (2010) Lerotic, M., & Lee, S.L. (2010). A multimodal silicone phantom for robotic surgical training and simulation. In The Hamlyn Symposium on Medical Robotics, London, UK (pp. 65–66).
 Lerotic et al. (2009) Lerotic, M., Lee, S.L., Keegan, J., & Yang, G.Z. (2009). Image constrained finite element modelling for realtime surgical simulation and guidance. In Biomedical Imaging: From Nano to Macro, 2009. ISBI’09. IEEE International Symposium on (pp. 1063–1066). IEEE.
 Manu (2016) Manu (2016). nonrigidicp. https://uk.mathworks.com/matlabcentral/fileexchange/41396nonrigidicp. Accessed: 20160220.
 Mercier et al. (2005) Mercier, L., Langø, T., Lindseth, F., & Collins, L. D. (2005). A review of calibration techniques for freehand 3D ultrasound systems. Ultrasound in medicine & biology, 31, 143–165.
 Prager et al. (1998) Prager, R. W., Rohling, R. N., Gee, A., & Berman, L. (1998). Rapid calibration for 3D freehand ultrasound. Ultrasound in medicine & biology, 24, 855–869.
 Rajamani et al. (2007) Rajamani, K. T., Styner, M. A., Talib, H., Zheng, G., Nolte, L. P., & Ballester, M. A. G. (2007). Statistical deformable bone models for robust 3D surface extrapolation from sparse data. Medical Image Analysis, 11, 99–109.
 Razavi et al. (2003) Razavi, R., Hill, D. L., Keevil, S. F., Miquel, M. E., Muthurangu, V., Hegde, S., Rhode, K., Barnett, M., van Vaals, J., Hawkes, D. J., & Baker, E. (2003). Cardiac catheterisation guided by MRI in children and adults with congenital heart disease. The Lancet, 362, 1877–1882.

Rosipal & Krämer (2006)
Rosipal, R., & Krämer, N.
(2006).
Overview and recent advances in partial least
squares.
In
Subspace, latent structure and feature selection
(pp. 34–51). Springer. 
Rosipal & Trejo (2001)
Rosipal, R., & Trejo, L. J.
(2001).
Kernel partial least squares regression in
reproducing kernel hilbert space.
Journal of machine learning research
, 2, 97–123.  Saeed et al. (2005) Saeed, M., Saloner, D., Weber, O., Martin, A., Henk, C., & Higgins, C. (2005). MRI in guiding and assessing intramyocardial therapy. European radiology, 15, 851–863.
 Segars et al. (2010) Segars, W., Sturgeon, G., Mendonca, S., Grimes, J., & Tsui, B. M. (2010). 4D XCAT phantom for multimodality imaging research. Medical physics, 37, 4902–4915.
 Sjöstrand et al. (2012) Sjöstrand, K., Clemmensen, L. H., Larsen, R., & Ersbøll, B. (2012). SpaSM: A matlab toolbox for sparse statistical modeling. http://www2.imm.dtu.dk/projects/spasm/. Accessed: 20171125.
 Vines (2000) Vines, S. (2000). Simple principal components. Journal of the Royal Statistical Society: Series C (Applied Statistics), 49, 441–451.
 Yeh et al. (2002) Yeh, W.C., Li, P.C., Jeng, Y.M., Hsu, H.C., Kuo, P.L., Li, M.L., Yang, P.M., & Lee, P. H. (2002). Elastic modulus measurements of human liver and correlation with pathology. Ultrasound in medicine & biology, 28, 467–474.
 Zou & Hastie (2005) Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67, 301–320.
 Zou et al. (2006) Zou, H., Hastie, T., & Tibshirani, R. (2006). Sparse principal component analysis. Journal of computational and graphical statistics, 15, 265–286.