Spatiotemporal-Aware Augmented Reality: Redefining HCI in Image-Guided Therapy

Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the redefinition of different human-machine interfaces. We demonstrate that this AR paradigm is generic, and can benefit a wide variety of procedures. Our system achieved an error of 4.76±2.91 mm for placing K-wire in a fracture management procedure, and yielded errors of 1.57±1.16^∘ and 1.46±1.00^∘ in the abduction and anteversion angles, respectively, for total hip arthroplasty. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes.



There are no comments yet.


page 1

page 2

page 3

page 4

page 5

page 6

page 9


Closing the Calibration Loop: An Inside-out-tracking Paradigm for Augmented Reality in Orthopedic Surgery

In percutaneous orthopedic interventions the surgeon attempts to reduce ...

Towards Augmented Reality-driven Human-City Interaction: Current Research and Future Challenges

Interaction design for Augmented Reality (AR) is gaining increasing atte...

Distributed Augmented Reality with 3D Lung Dynamics -- A Planning Tool Concept

Augmented Reality (AR) systems add visual information to the world by us...

3D Augmented Reality-Assisted CT-Guided Interventions: System Design and Preclinical Trial on an Abdominal Phantom using HoloLens 2

Background: Out-of-plane lesions pose challenges for CT-guided intervent...

Patient Specific Biomechanics Are Clinically Significant In Accurate Computer Aided Surgical Image Guidance

Augmented Reality is used in Image Guided surgery (AR IG) to fuse surgic...

Plan in 2D, execute in 3D: An augmented reality solution for cup placement in total hip arthroplasty

Reproducibly achieving proper implant alignment is a critical step in to...

Towards Predictions of the Image Quality of Experience for Augmented Reality Scenarios

Augmented Reality (AR) devices are commonly head-worn to overlay context...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Interventional image guidance is widely adopted across multiple disciplines of minimally-invasive and percutaneous therapies [1, 2, 3, 4]. Despite its importance in providing anatomy-level updates, visualization of images and interaction with the intra-operative data are inefficient, thus requiring extensive experience to properly associate the content of the image with the patient anatomy. These challenges become evident in interventions that require the surgeon to navigate wires and catheters through critical structures under excessive radiation, such as in fracture or endovascular repairs.

Surgical navigation and robotic systems are developed to support surgery with localization and execution of well-defined tasks [5, 6, 7, 8]. Though these systems increase the accuracy, their complex setup and explicit tracking nature may overburden the surgical workflow and consequently impede their acceptance in clinical routines [9]. Image-based navigation alleviates the requirements for external tracking, though depends strongly on pre-operative data which become outdated when the anatomy is altered during the surgery [10, 11].

Fig. 1: The augmented user interacts with the X-ray images within their viewing frustums (A-C). Corresponding AR views are shown in D-F.

As the surgical expectancy increases, the communication between the surgeon, crew, and the information becomes an important concern. Ineffective communication leads to increase of surgery time, radiation, and frustration to a point where in fluoroscopy-guided procedures, instead of the X-ray technician, the surgeons may reposition the scanners to ensure the task-defined views are optimal [12, 13].

To bridge the inefficiency gaps in surgical workflows, researchers have investigated the importance of human factor considerations in improving the usability of surgical data [14, 15]. Recent works focused on facilitating the unmet interaction needs by introducing touch-less mechanisms such as gaze, foot, or voice commands [16, 17, 18]. We believe the high stakes of surgery necessitates efficient interaction between all actors in the operating room i.e. surgeon, anesthesiologist and staff to communicate and access information. This demands user-centric designs that can also accommodate fluid movement of information and surgical inference across the entire team.

Augmented reality (AR) solutions have gained popularity in computer-integrated surgeries, as they can provide intuitive visualizations of medical data directly at patients’ site. Early works on surgical AR focused largely on multi-modal fusion of information and provided display-based overlays [19, 20, 21]. Subsequently, AR enabled the utilization of pre- and intra-interventional 3D data during therapies [22, 23, 24, 25].

The emergence of commercially available optical-see through head-mounted display (HMD) systems has led to development of AR solutions for various image-guided surgical disciplines, including percutaneous vertebroplasty, kyphoplasty, lumbar facet joint injection, orthopedic fracture management, bone cancer treatment, total hip arthroplasty (THA), interlocking nailing, cardiovascular surgeries, and surgical education [26, 27, 28, 29, 30, 31, 32, 33, 15].

AR has served as image viewer that directly displays the data at the operative site using virtual fluoroscopy monitors, hence eliminating the conventional off-axis visualization through static monitors [34, 35]. Moreover, AR is used to provide navigational information during interventions [36, 37, 38]. These systems often rely on tracking of external markers, which require line-of-sight and invasive implantation into patients tissue that can hinder their usability. Andress et al. suggested a flexible marker-based surgical AR methodology which only required the marker to appear in the X-ray beam during the image acquisition, and was removed immediately after [39]. Recent inside-out localization strategies in AR have greatly favored the fluid workflow over explicit navigation, and have proved effective in eliminating the need for external markers [40, 41, 42].

Fig. 2: Transformation chain of the spatially-aware AR system

This manuscript introduces the methodology and usability of a novel spatially-aware concept that enables immediate interaction with the medical data and promotes team approach where all stake-holders share a unified AR experience and communicate effectively. Our methodology exploits the viewing frustum of the imaging devices and human observers in the operating room (Fig. 1), and provides an engaging and immersive experience for the surgical team [42]. We showcase this solution in two high volume orthopedic procedures, i.e. K-wire placement in fracture care surgery, and acetabular cup placement in THA.

Ii Methodology

Our main contributions are the collaborative AR concepts using spatiotemporal-aware flying frustums [42] that enable intra-operative planning, define new workflows, support surgical crew, enhance the communication between surgeon and data, and enable intuitive documentation of the surgery for training purposes. We present the methodology for the realization of these concepts in the remainder of this section.

Ii-a Spatial-Awareness for AR

Visual data from cameras contain a wealth of information that can be used for simultaneous localization and mapping (SLAM). Visual SLAM is an important ingredient in our AR-based interaction recipe that enables co-localization of augmented users and the imaging device, which we will refer to as imaging observer, in a shared operating room environment. The relative pose between two frames and using the environment map

can be estimated by minimizing the following reprojection function:


where and are corresponding features in images and , and is the projection operator.

In this setting, all users are localized with respect to a common spatial anchor in the operating room. The first member joining the shared experience will establish the anchor, i.e. OR coordinate system, and every other member of the AR session will share their pose in a master-slave configuration with respect to this OR frame [40]. This relation is shown as in Fig. 2.

Ii-B Imaging Observer

C-arm scanners offer fluorscopic imaging capabilities for a wide range of less-invasive therapeutic areas. To seamlessly integrate this imaging device into our interactive AR paradigm, we augment the scanner with a rigidly attached visual tracker, that observers the structures in the OR environment, and communicates spatial information to all users. The materialization of this imaging observer system requires a co-calibration between the visual tracker on the scanner, and the X-ray source [41]. The constant transformation that explains the calibration is denoted as in Fig. 2.

To estimate , we formulate an over-determined system of equations as follows:


IR denotes the frame of an external tracker that is used to track the motion of the C-arm source as it undergoes different motion at times and . It is important to note that the IR tracker is only used for this one-time and offline co-calibration, and it is not used intra-operatively. By re-arranging Eq. 2, we formulate the problem in the form of as presented in Eq. 3, such that , and and represent the relative motion of the X-ray source and the SLAM capable visual sensor on the gantry, respectively.


Rotation and translation components of the hand-eye problem are disentangled and computed separately as:


We estimate the rotation parameters using unit quaternion representation as . Given that a unit quaternion

is formed by a vector

and a scalar such that , we re-write the rotation component in Eq. 4 using the quaternion product rule as:


Re-arranging the above formulation yields:


which is then solved in a constrained optimization fashion as:


where . After the rotation parameters are computed, the translation vector is estimated in a least-squares setting: .

Fig. 3: Multiple flying frustum are rendered at their corresponding 3D pose. The applications running on the computer and the HMD allows the users to replay various acquisition during or after the intervention.

Ii-C Geometry-Awareness for AR

In this section, we describe the underlying geometry that allows us to combine the content of 2D X-ray images, directly with the 3D spatial information we computed in Sec. II-A and II-B. To this end, we explicitly model the viewable region of the X-ray camera, known as the flying frustum [42], and allow interaction with images within their geometries. It is important to note that, the flying frustum refers to the full pyramid of vision (Fig. 2

), and is different than the truncated pyramids used in the computer graphics community. Despite the similarities in formulation, the conventional frustum model in graphics only applies to reflective images, and cannot accommodate the transmission model used in fluoroscopy. Therefore, we extend the perspective pinhole camera model that is commonly used in the computer vision community 


Fig. 4: The augmented projections allow us to exploit the geometry in AR and plan surgical tools in relation to patient anatomy. The misaligned virtual drill in A is repositioned until it appears inside the desired structure in all the frustums (B).

In our paradigm, users can move the images within their frustums on a virtual plane known as the near plane, between X-ray source and detector (referred to as the far plane), while they remain a valid image of the same anatomy. This interaction enables the users to intersect the images with corresponding anatomies, and intuitively observe 2D-image-to-3D-anatomy associations. Additionally, the imaging technologists which operate the scanner, can align the scanner with a desired frustum that is decided by the surgeon.

A flying frustum is defined using the following model:


where refers to the distance to the near plane, is the focal length, is the matrix of intrinsic parameters, and . The parameter is controlled by the user, such that when , the X-ray image is directly displayed at the detector scale. It is worth mentioning that, with conventional frustum models, the near plane can only take values smaller than the far plane, which is not the case in our representation.

Fig. 5: AR interface to plan trajectories on the X-ray acquisitions
Fig. 6: Each point in a frustum image corresponds to a ray passing through the landmark in 3D, and connecting the source and detector of the C-arm. Intersection of two rays recovers the 3D point and renders it directly on the patient (A-B). Similarly, annotation of lines in each frustum, corresponds to a plane in 3D. The intersection of these planes restores the 3D planning trajectory, and renders it in AR such that it travels through the corresponding anatomical structure (C-D).

For each 2D point , where is the domain of all acquired images, the corresponding point in the frustum domain is scaled by a factor as . Finally, the 3D pose of the interactive image in the frustum is defined as:


where . Fig. 3 demonstrates multiple flying frustums, each rendered given their respective 3D pose.

Fig. 7: In THA, abduction and anteversion angles of the acetabular implant are defined with respect to the anterior pelvic plane.
Fig. 8: K-wire placement in a fracture reduction procedure

Ii-D Planning Using Flying Frustums

Flying frustums discussed in Sec. II-A-II-C embed sufficient 3D and 2D information that enable interventional planning for the placement of surgical tools. In this section we introduce two distinct approaches for intra-operative planning.

In the first method, virtual tools are manipulated in 3D by the user, and simultaneously projected onto the X-ray images of all valid frustums. A point , where is the domain of all 3D points on a virtual tool, is projected onto the frustum as .

In an exemplary case shown in Fig. 4, the virtual drill is rotated and translated until it passes through a desired structure (e.g. through a bone canal) in all frustums. An alignment consensus in all frustums is the equivalent of the alignment of the virtual 3D tool with the imaged anatomy in 3D.

The second planning approach requires 2D interaction on the frustum X-ray images. In this setting, for each selected landmark on a frustum image (Fig. 5), a 3D ray connecting the C-arm source and the target landmark will be rendered into the AR scene. As illustrated in Fig. 6, the intersection of two rays from a corresponding landmark in two images reconstruct the 3D landmark. Each ray is defined via two elements: i) the position of the C-arm X-ray source , and ii) the unit direction vector from the source to the annotated landmark in the frustum. We estimate the closest point to the rays corresponding to each landmark via a least-squares minimization strategy as follows:


Similarly, two points on a frustum defining the entry and the exit points of a drilling trajectory, associate to two rays and in 3D. These two rays span a plane in 3D as shown in Fig. 6. The intersection of the planes corresponding to the same entry and exit points on frustums and form a 3D line that passes through the desired entry and exit points on the patient anatomy.

Our first approach requires a more complex interaction with the augmented surgical implant using the degrees-of-freedom, however generalizes to arbitrary structures beyond linear annotations, such as the curved plates used for internal fixations.

Ii-E Surgical Workflow Integration

Intra-operative planning and execution with the flying frustums support can be used in various fluoroscopy-guided procedures. In THA, the critical points defining the anterior pelvic plane (APP) can be each identified on X-ray images. Given APP, a virtual acetabular implant and a rigidly attached impactor are rendered in AR with their desired orientation that is calculated with respect to APP. Likewise, the translational component of the cup implant is identified by defining the center of the patient acetabulum on corresponding fluoroscopic images. These relations are shown in Fig. 7.

Another exemplary image-guided procedure is the placement of screws and K-wires during fracture management. As shown in Fig. 8, AR provides support for placement of K-wires using the trajectory planning on the corresponding frustums.

Iii Experimental results

Iii-a System Setup

Our system comprises an ARCADIS Orbic 3D C-arm (Siemens Healthineers, Forchheim, Germany) as an intra-operative X-ray device that automatically computes the cumulative area dose for each session. The immersive AR solution was built using the Unity cross-platform game engine (Unity Technologies, San Francisco, CA, US) and was deployed to an optical-see-through HMD, the Microsoft HoloLens (Microsoft, Redmond, WA). To jointly co-localize the augmented surgeon and the C-arm scanner, a second HoloLens device with inside-out SLAM capabilities was attached near the X-ray detector. The two HMDs shared their spatial anchor, a rich feature reference region in the common environment, over a wireless local network, allowing them to remain synchronized and establish spatial awareness. This connection was enabled through a TCP-based sharing service running on an Alienware (Dell, Round Rock, TX, US) laptop server with an Intel i7-7700HQ CPU, NVIDIA GTX 1070 graphics card, 16 GB RAM, and Windows 10 operating system. The X-ray images from C-arm were transmitted to the server computer over a direct Ethernet connection, and then uploaded to the HMD.

Iii-B System Calibration

To solve the hand-eye calibration problem in Eq. 3, different pairs of corresponding poses were recorded from the visual tracker on the C-arm as well as an external infrared tracking system that tracked the C-arm source. Fig. 9 presents the error for this offline calibration step given different sampling for the pose pairs.

The localization quality of the SLAM-based visual tracking system on the scanner, i.e. the HMD on the scanner, was compared to a ground-truth provided by an external tracker. We measured rotational errors of with a norm of and translational errors of  mm with a norm of  mm along the axes, respectively [42].

Fig. 9:

Mean and standard deviation of the translational and rotational errors for the hand eye calibration step are shown in the left and right plots, respectively. For each

number of pose pairs shown on the horizontal axis (except the last column which considers all the available data), the experiments were repeated times by each time sampling poses from the total available poses and computing the hand-eye calibration.

Iii-C Experiments

Eight orthopedic surgeons and residents from Johns Hopkins Hospital participated in pre-clinical user studies and performed two surgically relevant tasks while utilizing interactive flying frustums in an immersive AR environment.

In the first procedure, we focused on the correct placement of a K-wire to repair complex fractures. To emulate the K-wire placement through the superior pubic ramus (acetabulum arc), we used radiopaque cubic phantoms, as seen in Fig. 10-C. For direct comparison, we used the same setup that was used by Fischer et al. [22]. Each cube consisted of a stiff, lightweight, and non radiopaque methylene bisphenyl diisocyanate (MDI) foam. Since the superior pubic ramus is a tubular bone with a diameter of approximately  mm, we used a thin aluminium mesh filled with MDI that was placed inside each cube and served as the bone phantom. The two ends of the tubular structures were complemented with a rubber radiopaque ring. Each subject was asked to place a K-wire with a diameter of  mm through the tubular phantom using a surgical drill (Stryker Corporation, Kalamazoo, MI, US).

Fig. 10: A-B are the X-ray images of the cubic phantom shown in C. In D-E, the X-ray images of the same phantom are shown after a K-wire was successfully inserted inside the tube. F is the CBCT scan of the phantom which was acquired for verification. Due to metal artifacts, the tube does not exhibit strong contrast.

For the second procedure, we constructed a total hip arthroplasty mock setup by using a radiopaque pelvis phantom with a magnetic acetabulum to fixate the acetabular cup (Fig. 11). For direct comparison, we adopted the same experimental setup that was suggested by Alexander et al. [44]. The cup was attached to a straight cylindrical acetabular trialing impactor (Smith & Nephew, London, UK) allowing the operator to guide the cup. Since the ideal orientation of the implant is unknown, we use abduction and anteversion angles that lie in a safe zone defined by landmarks on the pelvis as described in [45].

Fig. 11: In A the setup of the C-arm, pelvic phantom, and the acetabular cup are shown. B is a close-up view of the phantom with an empty acetabular socket and a magnet for holding the implant in position. Image C shows the impactor while it is placed by a surgeon during the experiment, and D shows the successfully placed cup in the acetabulum.

Initially, each surgeon received a brief introduction to the Microsoft Hololens, preparing them to properly mount and use the HMD. To further instruct them on our AR application, pre-recorded training X-ray images were loaded onto their HMD, allowing them to become familiar with the interface, planning procedure, and the interaction mechanism using hand gestures.

After the required planning images were acquired by the proctors, each surgeon planned their respective procedure in AR and performed the drilling task into the cube or placed the acetabular component into the pelvis. During the procedure, they were explicitly allowed to order as many X-ray shots from any perspective that they considered necessary.

We recorded the planing time, the time it took them to execute the procedure, number of fluoroscopic acquisitions, and the cumulative radiation dose as it was measured by the scanner. Finally, for the verification and accuracy measurement, we acquired a 3D cone-beam CT (CBCT) scan of the phantoms with their respective implants.

Planning Time (sec)
Execution Time (sec)
# X-ray images 2
Dose (cGY(cm2))
Error (mm)
TABLE I: Outcome from K-wire insertion for participants
Planning Time (sec)
Execution Time (sec)
# X-ray images
Dose (cGY(cm2))
Abduction error ()
Anteversion error ()
TABLE II: Outcome from the placement of the acetabular implant for participants

Iii-D Results

Tables I and II comprise the performance of every participant in the experiments. Table I contains the measurements for the K-wire insertion, and Table II presents the procedural outcome for the acetabular cup placement. We separate the interventional time measurements into i) planning time, the time it took each surgeon to plan their procedure in AR, and ii) execution time, determining the duration of the insertion/placement of the instruments. Furthermore, we recorded the number of X-ray acquisitions and the respective dose for each user. Finally, to assess the overall performances, we computed the average distance of the K-wire from the center of the tube at the entry and the exit surface of the tubular structure, and the abduction and anteversion angles of the acetabular implant, based on standard guidelines.

Table III compares the K-wire insertion results of our immersive AR system with a previous non-immersive AR system [22] as well as the standard operating procedure (SOP) using conventional fluoroscopic guidance. Combining the planning and execution times, the AR procedure took on average  sec versus the  sec during SOP. Fig. 12 depicts this comparison. On average, the surgeons used fluoroscopic shots with a combined dose of  (cGY(cm2)) per user and committed an insertion error of  mm. The associated standard deviation (SD) values are presented in Table V.

MEAN Time # X-ray Dose Error
(sec) images (cGY(cm2)) (mm)
NI-AR [22]
TABLE III: Comparison of AR, non-immersive AR (NI-AR), and SOP for K-wire insertion. We denote the time for the AR procedure as (), with being the planning time and the execution time.
Fig. 12: The plots present the execution time and total radiation dose during K-wire insertion using the AR supported approach and SOP. On the leftmost plot, the blue boxplot is the execution time with AR, whereas the orange boxplot is the total time including the planning phase. The green lines show the mean values for each of the groups.
MEAN Time # X-ray Dose Abd. Ant.
(sec) images (cGY(cm2)) () ()
NI-AR [44]
TABLE IV: Performance during acetabular implant placement using AR, NI-AR, and SOP. We denote the time for the AR procedure as (), with being the planning and the execution time.

In Table IV we present the outcome for the acetabular cup placement procedure with the immersive AR system, comparing it to a previous non-immersive AR application [44] and SOP. Using SOP, it took surgeons on average  sec to place the cup and under AR a combined time of  sec was achieved. For the AR setup, we acquired X-ray images with an average dose of  (cGY(cm2)) per surgeon, whereas fluoroscopic images with a dose of  (cGY(cm2)) were acquired during SOP. With the AR system, the average errors were and for the abduction and anteversion angles, respectively. Under SOP the respective angles were and . Figs. 13 and 14 present the outcome with respect to time, radiation dose, and individual rotational measures for the acetabular cup placement experiments using AR and SOP. The corresponding SD are listed in Table VI. The immersive AR results show an SD of respectively sec and  sec for planning and execution time, for the number of X-ray images, for the dose, for the abduction error and in the anteversion error.

Fig. 13: Comparison of time and total radiation dose during cup placement with AR and SOP approaches. The orange boxplot represents the total time including the planning time. The red (

) denote outliers, where in the leftmost plot the top sign belongs to the orange boxplot, and the bottom (

) to the blue plot.
SD Time # X-ray Dose Error
(sec) images (cGY(cm2)) (mm)
TABLE V: The respective standard deviation values, relating to Table III.
SD Time # X-ray Dose Abd. Ant.
(sec) images (cGY(cm2)) () ()
TABLE VI: The respective standard deviation values, relating to Table IV.
Fig. 14: Anteversion and abduction angles after acetabular cup placement using AR support and SOP

Iv Discussion

We evaluated our spatially-aware AR system in two clinically relevant procedures, i) the placement of K-wires through tubular structures for fracture repair tasks, and ii) placement of acetabular components into the hip socket for total hip arthroplasty. We selected these two high volume procedures among the many other applications which can be enabled by our interactive AR system, as they each represent a class of common orientational alignment and localization tasks that are prevalent across different fields of image-guided surgery.

For the K-wire insertion procedure, the immersive AR system performed substantially faster than the conventional SOP, yielding less than a fifth of the time (Fig. 12). Table III demonstrates a detailed comparison of our system, not only with the SOP as an established baseline, but also with a previously presented non-immersive mixed reality method based on RGBD sensing and intra-operative CBCT imaging [22].

With the AR system every surgeon used exactly X-ray images, which were the images required for procedure planning. Despite explicitly allowing them to take as many radiographs as they desire, no one of the surgeons requested additional X-ray images. As mentioned above, during SOP, surgeons inserted the K-wires with an average of fluoroscopic images and with an average dose of  cGY(cm), compared to cGY(cm), which were used during the AR procedures. The RGBD-CBCT system in [22, 23] yielded on average X-rays, although it required a pre-procedural CBCT scan of the phantom, inducing the higher radiation dose of cGY(cm).

Finally, evaluating the outcome of the procedure with regard to the drilling error, AR ( mm) outperforms RGBD-CBCT ( mm), both being marginally worse than SOP ( mm). Considering that we only instructed the surgeons to drill through the tube and not precisely through the center of the tube, we regard these difference as negligible. It is important to note that, our AR system performed similar to the conventional X-ray method in terms of accuracy, while reducing time by a factor of , number of fluoroscopic acquisitions by a factor of , and the radiation dose by a factor of .

The same trend is observed with the measurements for the placement of the acetabular cup, demonstrating the effectiveness of our AR system, we compare it against SOP and a NI-AR system as presented in [44]. As shown in Fig. 13, the execution time is considerably lower using AR; even when combining planning and execution time it took the surgeons  sec, which is nearly half of the  sec that they needed under SOP and less than the  sec with NI-AR. Furthermore, the number of fluoroscopic images were reduced; every surgeon used exactly images, which are again merely the images required for planning. This resulted in an average dose of  cGY(cm), which is lower than with SOP, where the surgeons used an average of radiographs with an average dose of  cGY(cm), and with NI-AR where one pre-procedure CBCT lead to a dose of  cGY(cm). The objective of this procedure was to achieve abduction and anteversion angles of and , respectively, which lie in the clinical safe-zone [45]. The respective errors are shown in Table IV and Fig. 14. The outcome distinctly displays a more accurate cup placement using the spatially-aware immersive AR system () compared to the SOP (), compared to the NI-AR system () the abduction error is slightly less, whereas the anteversion error is marginally higher ().

For both procedures the deployment of our AR system lead to a comparable or higher accuracy, fewer X-ray images with a consequently lower radiation dose. For the total time, it has to be noted that our planning time does not include the recording of the X-ray images that were necessary to plan the procedure. This step however, as shown in [46], can be fully automated, resulting in an immediate availability of the fluoroscopic images.

V Conclusion and Future Work

We presented the embodiment of a novel interaction concept based on spatiotemporal-aware AR. For the two orthopedic use cases presented in this manuscript, our immersive AR system demonstrated improvements in time, number of X-ray acquisitions, radiation dose, and outcome during cup placement.

The spatiotemporal awareness inherent in AR overhauls the ill-posed communication between the surgeon, staff, and information; e.g. Fig. 15 shows the potential role of flying frustums and AR in effectively communicating desired X-ray views to the technician, eliminating unfavorable views and reducing the staff burnout.

Fig. 15: Visualization of a target frustum (A) allows the C-arm operator to align the current C-arm frustum with the surgeon’s desired perspective (B) and eliminate the waste of time and radiation during fluoro hunting. This concept is an example of the capabilities of interactive frustums on moving information between different stake holders in the OR, i.e. surgeon, patient, X-ray technician, staff, etc.

An application of AR outside the OR is ”surgical replay”, where the residents can review the surgery, accompanied with its temporal and spatial information including all the X-ray acquisitions and optical point-clouds from the patient site. This enables the medical trainees to identify distinct actions that were taken by the experienced surgeon based upon each image. Access to such 3D post-operative analysis has the potential to dramatically improve the quality of surgical education.

Fig. 16: Spatial and temporal information from the surgery can be recorded and reviewed after surgery.

Direct visualization of X-ray images within their corresponding viewing frustums delivers intuition that effectively unites the content of the 2D image with the 3D imaged anatomy. In this setting, images from various perspectives can be grouped within their frustums to form multi- or extended-view representations of the anatomy. The interlocked frustums shown in Fig. 17 are examples for such visualization concept, that can particularly benefit interventions where leg-length discrepancies or malrotations in tibio- and lateral/distal-femoral angles are major concerns.

Fig. 17: Interlocking of multiple X-ray frustums enables visualization of large anatomical structures.

Though our solution delivers spatial awareness, it should not be regarded as a surgical navigation system. This is because marker-less tracking, currently, cannot deliver the level of accuracy achieved by marker-based surgical navigation or robotic systems. Our solution is merely an advanced visualization platform that enhances the interaction across the surgical ecosystem and promotes effective collaboration and team approach.

The widespread adoption of human-centered AR in interventional routines requires careful considerations regarding surgeons’ experience. The interaction with the virtual content using intuitive hand gestures, and resolving the perceptual ambiguities between real and virtual that occur due to vergence-accommodation conflict, can greatly contribute to the acceptance of AR technologies [47]

. Furthermore, development of artificial intelligence strategies can

i) create semantic understanding from the surgical environment and augment surgeon’s intelligence, and ii) enhance the spatial mapping and co-localization, thus improving the stability of marker-less AR systems. Lastly, future integration of eye tracking systems into HMDs can circumvent the internal eye-to-display calibration, adjust the rendering based on each user, and replace unnecessary gestures with gaze-based interaction.

As shown in this paper, the introduction of new technology requires a user-centric design for its full integration into the clinical workflow. The ultimate goal may be the discovery of new surgical workflows enabled by the introduction of novel technology. This goal could only be achieved with a close and extended partnership between surgeons and technical experts, as well as the full integration of the advanced technology in the broad spectrum of surgical teaching, training, planning, workflow, and documentation.


  • [1] J. Siewerdsen, D. Moseley, S. Burch, S. Bisland, A. Bogaards, B. Wilson, and D. Jaffray, “Volume ct with a flat-panel detector on a mobile, isocentric c-arm: Pre-clinical investigation in guidance of minimally invasive surgery,” Medical physics, vol. 32, no. 1, pp. 241–254, 2005.
  • [2] J. S. Hott, V. R. Deshmukh, J. D. Klopfenstein, V. K. Sonntag, C. A. Dickman, R. F. Spetzler, and S. M. Papadopoulos, “Intraoperative iso-c c-arm navigation in craniospinal surgery: the first 60 cases,” Neurosurgery, vol. 54, no. 5, pp. 1131–1137, 2004.
  • [3] D. L. Miller, E. Vanó, G. Bartal, S. Balter, R. Dixon, R. Padovani, B. Schueler, J. F. Cardella, and T. De Baère, “Occupational radiation protection in interventional radiology: a joint guideline of the cardiovascular and interventional radiology society of europe and the society of interventional radiology,” Cardiovascular and interventional radiology, vol. 33, no. 2, pp. 230–239, 2010.
  • [4] N. Theocharopoulos, K. Perisinakis, J. Damilakis, G. Papadokostakis, A. Hadjipavlou, and N. Gourtsoyiannis, “Occupational exposure from common fluoroscopic projections used in orthopaedic surgery,” JBJS, vol. 85, no. 9, pp. 1698–1703, 2003.
  • [5] C. W. Kim, Y.-P. Lee, W. Taylor, A. Oygar, and W. K. Kim, “Use of navigation-assisted fluoroscopy to decrease radiation exposure during minimally invasive spine surgery,” The Spine Journal, vol. 8, no. 4, pp. 584–590, 2008.
  • [6] U. K. Kakarla, A. S. Little, S. W. Chang, V. K. Sonntag, and N. Theodore, “Placement of percutaneous thoracic pedicle screws using neuronavigation,” World neurosurgery, vol. 74, no. 6, pp. 606–610, 2010.
  • [7] N. R. Crawford, N. Theodore, and M. A. Foster, “Surgical robot platform,” Oct. 10 2017, uS Patent 9,782,229.
  • [8] T. Yi, V. Ramchandran, J. H. Siewerdsen, and A. Uneri, “Robotic drill guide positioning using known-component 3d–2d image registration,” Journal of Medical Imaging, vol. 5, no. 2, p. 021212, 2018.
  • [9] L. Joskowicz and E. J. Hazan, “Computer aided orthopaedic surgery: incremental shift or paradigm change?” 2016.
  • [10]

    R. B. Grupp, R. Hegeman, R. Murphy, C. Alexander, Y. Otake, B. McArthur, M. Armand, and R. H. Taylor, “Pose estimation of periacetabular osteotomy fragments with intraoperative x-ray navigation,”

    IEEE Transactions on Biomedical Engineering, 2019.
  • [11] J. Goerres, A. Uneri, M. Jacobson, B. Ramsay, T. De Silva, M. Ketcha, R. Han, A. Manbachi, S. Vogt, G. Kleinszig et al., “Planning, guidance, and quality assurance of pelvic screw placement using deformable image registration,” Physics in Medicine & Biology, vol. 62, no. 23, p. 9018, 2017.
  • [12] M. Synowitz and J. Kiwit, “Surgeon’s radiation exposure during percutaneous vertebroplasty,” Journal of Neurosurgery: Spine, vol. 4, no. 2, pp. 106–109, 2006.
  • [13] A. Starr, A. Jones, C. Reinert, and D. Borer, “Preliminary results and complications following limited open reduction and percutaneous screw fixation of displaced fractures of the acetabulum.” Injury, vol. 32, pp. SA45–50, 2001.
  • [14] K. Aagaard, B. S. Laursen, B. S. Rasmussen, and E. E. Sørensen, “Interaction between nurse anesthetists and patients in a highly technological environment,” Journal of PeriAnesthesia Nursing, vol. 32, no. 5, pp. 453–463, 2017.
  • [15] C. Laverdière, J. Corban, J. Khoury, S. M. Ge, J. Schupbach, E. J. Harvey, R. Reindl, and P. A. Martineau, “Augmented reality in orthopaedics: a systematic review and a window on future possibilities,” The Bone & Joint Journal, vol. 101, no. 12, pp. 1479–1488, 2019.
  • [16] B. Hatscher, M. Luz, L. E. Nacke, N. Elkmann, V. Müller, and C. Hansen, “Gazetap: towards hands-free interaction in the operating room,” in Proceedings of the 19th ACM international conference on multimodal interaction.   ACM, 2017, pp. 243–251.
  • [17] A. Mewes, B. Hensen, F. Wacker, and C. Hansen, “Touchless interaction with software in interventional radiology and surgery: a systematic literature review,” International journal of computer assisted radiology and surgery, vol. 12, no. 2, pp. 291–305, 2017.
  • [18] M. Ma, P. Fallavollita, S. Habert, S. Weidert, and N. Navab, “Device-and system-independent personal touchless user interface for operating rooms: One personal ui to control all displays in an operating room.” International journal of computer assisted radiology and surgery, vol. 11, no. 6, pp. 853–861, 2016.
  • [19] Y. Sato, M. Nakamoto, Y. Tamaki, T. Sasama, I. Sakita, Y. Nakajima, M. Monden, and S. Tamura, “Image guidance of breast cancer surgery using 3-d ultrasound images and augmented reality visualization,” IEEE Transactions on Medical Imaging, vol. 17, no. 5, pp. 681–693, 1998.
  • [20] N. Navab, A. Bani-Kashemi, and M. Mitschke, “Merging visible and invisible: Two camera-augmented mobile c-arm (camc) applications,” in Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99).   IEEE, 1999, pp. 134–141.
  • [21] N. Navab, S.-M. Heining, and J. Traub, “Camera augmented mobile c-arm (camc): calibration, accuracy study, and clinical applications,” IEEE transactions on medical imaging, vol. 29, no. 7, pp. 1412–1423, 2009.
  • [22] M. Fischer, B. Fuerst, S. C. Lee, J. Fotouhi, S. Habert, S. Weidert, E. Euler, G. Osgood, and N. Navab, “Preclinical usability study of multiple augmented reality concepts for k-wire placement,” International journal of computer assisted radiology and surgery, vol. 11, no. 6, pp. 1007–1014, 2016.
  • [23] J. Fotouhi, B. Fuerst, S. C. Lee, M. Keicher, M. Fischer, S. Weidert, E. Euler, N. Navab, and G. Osgood, “Interventional 3d augmented reality for orthopedic and trauma surgery,” in 16th Annual Meeting of the Int. Society for Computer Assisted Orthopedic Surgery (CAOS), 2016.
  • [24] E. Tucker, J. Fotouhi, M. Unberath, S. C. Lee, B. Fuerst, A. Johnson, M. Armand, G. M. Osgood, and N. Navab, “Towards clinical translation of augmented orthopedic surgery: from pre-op ct to intra-op x-ray via rgbd sensing,” in Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications, vol. 10579.   International Society for Optics and Photonics, 2018, p. 105790J.
  • [25] J. Fotouhi, C. P. Alexander, M. Unberath, G. Taylor, S. C. Lee, B. Fuerst, A. Johnson, G. M. Osgood, R. H. Taylor, H. Khanuja et al., “Plan in 2-d, execute in 3-d: an augmented reality solution for cup placement in total hip arthroplasty,” Journal of Medical Imaging, vol. 5, no. 2, p. 021205, 2018.
  • [26] F. Müller, S. Roner, F. Liebmann, J. M. Spirig, P. Fürnstahl, and M. Farshad, “Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3d imaging,” The Spine Journal, 2019.
  • [27] C. A. Agten, C. Dennler, A. B. Rosskopf, L. Jaberg, C. W. Pfirrmann, and M. Farshad, “Augmented reality–guided lumbar facet joint injections,” Investigative radiology, vol. 53, no. 8, pp. 495–498, 2018.
  • [28] J. T. Gibby, S. A. Swenson, S. Cvetko, R. Rao, and R. Javan, “Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography,” International journal of computer assisted radiology and surgery, vol. 14, no. 3, pp. 525–535, 2019.
  • [29] B. van Duren, K. Sugand, R. Wescott, R. Carrington, and A. Hart, “Augmented reality fluoroscopy simulation of the guide-wire insertion in dhs surgery: A proof of concept study,” Medical engineering & physics, vol. 55, pp. 52–59, 2018.
  • [30] H. S. Cho, M. S. Park, S. Gupta, I. Han, H.-S. Kim, H. Choi, and J. Hong, “Can augmented reality be helpful in pelvic bone cancer surgery? an in vitro study,” Clinical Orthopaedics and Related Research®, vol. 476, no. 9, pp. 1719–1725, 2018.
  • [31] H. Ogawa, S. Hasegawa, S. Tsukada, and M. Matsubara, “A pilot study of augmented reality technology applied to the acetabular cup placement during total hip arthroplasty,” The Journal of arthroplasty, vol. 33, no. 6, pp. 1833–1837, 2018.
  • [32] H. Brun, R. Bugge, L. Suther, S. Birkeland, R. Kumar, E. Pelanis, and O. Elle, “Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease,” European Heart Journal-Cardiovascular Imaging, 2018.
  • [33] P. E. Pelargos, D. T. Nagasawa, C. Lagman, S. Tenn, J. V. Demos, S. J. Lee, T. T. Bui, N. E. Barnette, N. S. Bhatt, N. Ung et al., “Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery,” Journal of Clinical Neuroscience, vol. 35, pp. 1–4, 2017.
  • [34] G. Deib, A. Johnson, M. Unberath, K. Yu, S. Andress, L. Qian, G. Osgood, N. Navab, F. Hui, and P. Gailloud, “Image guided percutaneous spine procedures using an optical see-through head mounted display: proof of concept and rationale,” Journal of neurointerventional surgery, vol. 10, no. 12, pp. 1187–1191, 2018.
  • [35] P. C. Chimenti and D. J. Mitten, “Google glass as an alternative to standard fluoroscopic visualization for percutaneous fixation of hand fractures: a pilot study,” Plastic and reconstructive surgery, vol. 136, no. 2, pp. 328–330, 2015.
  • [36] R. Moreta-Martinez, D. García-Mato, M. García-Sevilla, R. Pérez-Mañanes, J. Calvo-Haro, and J. Pascau, “Augmented reality in computer-assisted interventions based on patient-specific 3d printed reference,” Healthcare technology letters, vol. 5, no. 5, pp. 162–166, 2018.
  • [37] J. W. Meulstee, J. Nijsink, R. Schreurs, L. M. Verhamme, T. Xi, H. H. Delye, W. A. Borstlap, and T. J. Maal, “Toward holographic-guided surgery,” Surgical innovation, vol. 26, no. 1, pp. 86–94, 2019.
  • [38] L. Ma, Z. Zhao, B. Zhang, W. Jiang, L. Fu, X. Zhang, and H. Liao, “Three-dimensional augmented reality surgical navigation with hybrid optical and electromagnetic tracking for distal intramedullary nail interlocking,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 14, no. 4, p. e1909, 2018.
  • [39] S. Andress, A. Johnson, M. Unberath, A. F. Winkler, K. Yu, J. Fotouhi, S. Weidert, G. Osgood, and N. Navab, “On-the-fly augmented reality for orthopedic surgery using a multimodal fiducial,” Journal of Medical Imaging, vol. 5, no. 2, p. 021209, 2018.
  • [40] J. Hajek, M. Unberath, J. Fotouhi, B. Bier, S. C. Lee, G. Osgood, A. Maier, M. Armand, and N. Navab, “Closing the calibration loop: an inside-out-tracking paradigm for augmented reality in orthopedic surgery,” in International Conference on Medical Image Computing and Computer-Assisted Intervention.   Springer, 2018, pp. 299–306.
  • [41] J. Fotouhi, M. Unberath, T. Song, J. Hajek, S. C. Lee, B. Bier, A. Maier, G. Osgood, M. Armand, and N. Navab, “Co-localized augmented human and x-ray observers in collaborative surgical ecosystem,” International journal of computer assisted radiology and surgery, vol. 14, no. 9, pp. 1553–1563, 2019.
  • [42]

    J. Fotouhi, M. Unberath, T. Song, W. Gu, A. Johnson, G. Osgood, M. Armand, and N. Navab, “Interactive flying frustums (iffs): spatially aware surgical data visualization,”

    International journal of computer assisted radiology and surgery, pp. 1–10, 2019.
  • [43] R. Hartley and A. Zisserman, Multiple view geometry in computer vision.   Cambridge university press, 2003.
  • [44] C. Alexander, A. E. Loeb, J. Fotouhi, N. Navab, M. Armand, and H. S. Khanuja, “Augmented reality for acetabular component placement in direct anterior total hip arthroplasty,” The Journal of Arthroplasty, 2020.
  • [45] G. E. Lewinnek, J. Lewis, R. Tarr, C. Compere, and J. Zimmerman, “Dislocations after total hip-replacement arthroplasties.” The Journal of bone and joint surgery. American volume, vol. 60, no. 2, pp. 217–220, 1978.
  • [46] L. Qian, M. Unberath, K. Yu, B. Fuerst, A. Johnson, N. Navab, and G. Osgood, “Towards virtual monitors for image guided interventions-real-time streaming to optical see-through head-mounted displays,” arXiv preprint arXiv:1710.00808, 2017.
  • [47] J. Fotouhi, T. Song, A. Mehrfard, G. Taylor, A. Martin-Gomez, B. Fuerst, M. Armand, M. Unberath, and N. Navab, “Reflective-ar display: An interaction methodology for virtual-real alignment in medical robotics,” arXiv preprint arXiv:1907.10138, 2019.