From individual to population: Challenges in Medical Visualization

06/06/2012 ∙ by Charl P. Botha, et al. ∙ Association for Computing Machinery Stony Brook University Otto-von-Guericke-Universität Magdeburg Delft University of Technology 0

In this paper, we first give a high-level overview of medical visualization development over the past 30 years, focusing on key developments and the trends that they represent. During this discussion, we will refer to a number of key papers that we have also arranged on the medical visualization research timeline. Based on the overview and our observations of the field, we then identify and discuss the medical visualization research challenges that we foresee for the coming decade.



There are no comments yet.


page 5

page 13

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Since the advent of magnetic resonance imaging (MRI) and computed tomography (CT) scanners around the early seventies, and the consequent ubiquitousness of medical volume data, medical visualization has undergone significant development and is now a primary branch of Visualization. It finds application in diagnosis, for example virtual colonoscopy, in treatment, for example surgical planning and guidance, and in medical research, for example visualization of diffusion tensor imaging data. Although the field of medical visualization only established itself with this name in the late eighties 

preim_visualization_2007 , we shall see in the next section that already in the seventies there were published examples of computer-generated images, based on medical data and used for medical applications.

During the past decades, medical image acquisition technology has undergone continuous and rapid development. It is now possible to acquire much more complex data than ever before. For example, in High Angular Resolution Diffusion Imaging (HARDI), forty or more diffusion-weighted volumes are acquired in order to calculate and visualize water diffusion and, indirectly, structural neural connections in the brain tuch_high_2002

. In fMRI-based full brain connectivity, time-based correlation of neural activity is indirectly measured between all pairs of voxels in the brain, thus giving insight into the functional neural network 

greicius_functional_2003 . Moreover, the questions that users attempt to answer using medical visualization have also become significantly more complex.

1978 - Sunguroff and Greenberg: CT and 3D surfaces for diagnosis and radiotherapy planning sunguroff_computer_1978-1 .

1983 - Vannier et al.: 3D surfaces from CT for planning of craniofacial surgery vannier_three_1983 .

1986 - Hohne and Bernstein: Shading 3D Images from CT using gray-level gradients hohne_shading_1986 .

1987 - Lorensen and Cline: Marching Cubes lorensen_marching_1987 .

1988 - Levoy publishes Direct Volume Rendering paper in May levoy_display_1988 , Drebin et al. in August drebin_volume_1988 .

Multi-modal volume rendering by Höhne et al. hohne_display_1988 .

1993 - Altobelli et al.: Predictive simulation in surgical planning altobelli_computer-assisted_1993 . Gerig et al.: Vessel visualization barrett_symbolic_1993 .

1994 - Basser et al.: Diffusion Tensor Imaging basser_mr_1994 .

1995 - Hong et al.: 3D Virtual Colonoscopy lichan_hong_3d_1995 .

1996 - Behrens et al. et al.: Visualization of Dynamic Contrast-Enhanced MRI mammography data (time-varying) behrens_computer-assisted_1996 .

1998 - Basser et al.: DTI Tractography basser_fiber-tractography_1998 ; basser_vivo_2000 .

2000 - Ebert and Rheingans - Medical Volume Illustration ebert_volume_2000 .

2001 - Tory et al.: Multi-timepoint MRI tory_4d_2001 .

2003 - Krüger and Westermann: GPU raycasting kruger_acceleration_2003 .

2007 - Blaas et al.: Multi-Field Medical Visualization blaas_interactive_2007 .

2008 - Wang et al.: LifeLines2 - multi-subject electronic health records wang_aligning_2008 .

2010 - Steenwijk et al.: Cohort studies - multi-subject imaging and metadata steenwijk_integrated_2010 .
Figure 1: Timeline with a subset of medical visualization papers showing the progression from scalar volume datasets through time-dependent data to multi-field and finally multi-subject datasets. This timeline is by no means complete, instead attempting to show a representative sample of papers that represent various trends in the development of the field.

In this paper, we first give a high-level overview of medical visualization development over the past 30 years, focusing on key developments and the trends that they represent. During this discussion, we will refer to a number of key papers that we have also arranged on the medical visualization research timeline shown in figure 1. Based on the overview and our observations of the field, we then identify and discuss the medical visualization research challenges that we foresee for the coming decade.

2 Thirty-year Overview of Medical Visualization

Already in 1978, Sunguroff and Greenberg published their work on the visualization of 3D surfaces from CT data for diagnosis, as well as a visual radio-therapy planning system, also based on CT data sunguroff_computer_1978-1 . Five years later, Vannier et al. published their results developing a system for the computer-based pre-operative planning of craniofacial surgery vannier_three_1983 . By this time, they had already used and evaluated their surgical planning system in treating 200 patients. The system was based on the extraction and visualization of 3D hard and soft tissue surfaces from CT data. Through the integration of an industrial CAD application, it was also possible to perform detailed 3D measurements on the extracted surfaces.

2.1 Practical and Multi-modal Volume Visualization

In 1986, Hohne and Bernstein proposed using the gray-level gradient to perform shading of surfaces rendered from 3D CT data hohne_shading_1986 . In 1987, Lorensen and Cline published the now famous Marching Cubes isosurface extraction algorithm, which enabled the fast and practical extraction of 3D isosurfaces from real-world medical data. In the year thereafter, Levoy introduced the idea of volume raycasting in May levoy_display_1988 , and Drebin et al. in August drebin_volume_1988 . Although medical volume visualization was possible before these publications, as witnessed by a number of publications, previous techniques were either not as fast or yielded less convincing results. With the introduction of Marching Cubes and volume raycasting, volume visualization became a core business of visualization and medical visualization for the years to come.

Up to this point, research had focused on uni-modality data, primarily CT. However, already in 1988 the first multi-modal volume rendering paper was published by Höhne et al.

, in which they demonstrated the registration and combined visualization of CT and MRI. A great deal of work has been done since then on the theory and applications of multi-modal volume visualization. The first notable example is the work of Cai and Sakas in 1999 where they classified voxel-voxel multi-modal volume rendering techniques according to the volume rendering pipeline stage where they take place 

cai_data_1999 . The three classes are image level, where two volume renderings are combined pixel-by-pixel, accumulation level, where looked up samples along the ray are combined, and illumination model level, where the illumination model is adapted to process two volume samples directly. The second example we mention is a convincing application of multi-modal volume rendering for the planning of neurosurgical interventions, where MRI, CT, fMRI, PET and DSA data are all combined in an interactive but high quality visualization for the the planning of brain tumor resection beyer_high-quality_2007 .

2.2 Therapy Planning, Predictive Simulation, and Diagnosis

Therapy planning was one of the first real applications of medical visualization and remains important to this day. In 1993, Altobelli et al. published their work on using CT data to visualize the possible outcome of complicated craniofacial surgery altobelli_computer-assisted_1993 . By manually repositioning soft tissue fragments based on the bony surfaces under them, in certain cases taking into account bone-skin motion ratios from literature, the expected outcome of a craniofacial procedure could be visualized. Although still rudimentary, this could be considered one of the earliest cases of predictive or outcome simulation integrated with visualization for surgical planning. The idea of predictive simulation, or predictive medicine, was further explored by Taylor et al. for cardiovascular surgery taylor_predictive_1999 .

With the introduction of virtual colonoscopy (VC) in 1995 lichan_hong_3d_1995 , medical visualization also gained diagnosis as an important medical application, namely screening for colon cancer. VC combines CT scanning and volume visualization technologies. The patient abdomen is imaged in a few seconds by a multi-slice CT scanner. A 3D model of the colon is then reconstructed from the scan by automatically segmenting the colon and employing “electronic cleansing” of the colon for computer-based removal of the residual material. The physician then interactively navigates through the volume rendered virtual colon employing camera control mechanisms, customized tools for 3D measurements, “virtual biopsy” to interrogate suspicious regions, and “painting” to support 100% inspection of the colon surface hong_virtual_1997 . VC is rapidly gaining popularity and is poised to become the procedure of choice in lieu of the conventional optical colonoscopy for mass screening for colon polyps – the precursor of colorectal cancer. Unlike optical colonoscopy, VC is patient friendly, fast, non-invasive, more accurate, and cost-effective procedure for mass screening for colorectal cancer.

VC technologies gave rise to the computer-aided detection (CAD) of polyps, where polyps are detected automatically by integrating volume rendering, conformal colon flattening, clustering, and “virtual biopsy” analysis. Along with the reviewing physician, CAD provides a second pair of “eyes” for locating polyps hong_pipeline_2006 . This work was also the basis for many other virtual endoscopy systems, such as virtual bronchoscopy, virtual cystoscopy, and virtual angioscopy. A careful integration of image analysis (e.g., segmentation, skeletonization), with efficient rendering (e.g., occlusion culling) and interaction (e.g., camera control based on predefined paths) are major ingredients of such systems bartz_virtual_2005 ; hong_virtual_1997 .

2.3 Multi-field Data

Diffusion Tensor Imaging, or DTI, is an MRI-based acquisition modality, introduced in 1994 by Basser et al., that yields symmetric diffusion tensors as its native measurement quantity basser_mr_1994 . The tensors represent the local diffusion of water molecules, and hence indirectly indicate the presence and orientation of fibrous structures, such as neural fiber bundles or muscle fibers. Already in this first paper, the authors employed 3D glyphs to visualize the eigensystems represented by the tensors, as shown in figure 2.

Basser and his colleagues were also some of the first to extract and visualize fiber-tract trajectories from DTI data of the brain basser_vivo_2000 ; basser_fiber-tractography_1998 , thus linking together the point diffusion measurements to get an impression of the global connective structures in the brain. With DTI it was encouraging to see that the first visualization efforts were initiated by the scientists developing this new scanning modality themselves. Early work by the visualization community includes tensor lines for tractography weinstein_tensorlines:_1999 and direct volume rendering of DTI data kindlmann_hue-balls_1999 ; kindlmann_strategies_2000 .

Figure 2: Effective diffusion ellipsoid glyphs from a 2D region of interest of a DTI dataset of a cat brain. Image taken from basser_mr_1994 .

Importantly, DTI serves as one of the first examples of natively multi-field medical data, that is medical data with multiple parameters defined over the same spatio-temporal domain. The advent of DTI initiated a whole body of medical visualization research dedicated to the question of how best to visually represent and interact with diffusion tensor data in particular and multi-field medical data in general. The 2007 paper by Blaas et al. presented a visual analysis-inspired solution to this problem based on linked physical and feature space views blaas_interactive_2007 .

2.4 Time-varying Data

Time-varying medical volume data visualization made its entrance in 1996 with work by Behrens

et al. on supporting the examination of Dynamic Contrast-Enhanced MRI mammography data with the display of parameter maps, the selection of regions of interest (ROIs), the calculation of time-intensity curves (TICs), and the quantitative analysis of these curves behrens_computer-assisted_1996 . In 2001, Tory et al. presented methods for visualizing multi-timepoint (1 month interval) MRI data of a multiple sclerosis (MS) patient, where the goal was to study the evolution of brain white matter lesions over time tory_4d_2001 . Methods used included glyphs, multiple isosurfaces, direct volume rendering and animation. Coto et al. applied multiple coupled views, including linked cursors and brushing on 3D renderings and scatterplots, to dynamic contrast-enhanced MRI (DCE-MRI) mammography data coto_mammoexplorer:_2005 .

2.5 Illustrative Visualization

Illustrative visualization is primarily motivated by the attempt to create renditions that consider perceptual capabilities of humans. As an example, humans infer information about shapes not only from realistic shading but also from appropriate hatching and from outlines that support the mental separation of nearby objects rendered in similar colours.

Illustrative visualization is related to the term Non-Photorealistic Rendering in computer graphics, or NPR for short. The term NPR was used since around 1990 when the seminal paper of Saito et al. clearly illustrated that complex 3D shapes could be rendered more comprehensible by using certain feature lines saito_comprehensible_1990 . Compared to NPR, illustrative visualization is the more focused term that covers rendering techniques serving clear visualization goals, namely to convey shape information efficiently. In medical visualization, either surfaces or volume data are rendered in illustrative styles. For illustrative volume rendering, the term volume illustration was introduced by Ebert et al. in 2000 ebert_volume_2000 . Boundary enhancement based on gradient approximation csebfalvi_fast_2001 and curvature-based transfer functions kindlmann_curvature-based_2003 are landmarks in illustrative medical visualization. Tietjen et al. applied silhouettes and other feature lines for various scenarios in liver surgery planning tietjen_combining_2005

. Besides silhouettes, stippling and probably even more hatching has great potential to reveal details of shapes 

interrante_enhancing_1995 .

Later, Bruckner et al. made a number of important contributions that support depth and shape perception with adapted transfer functions. In particular, they considered the peculiarities of interactive exploration of 3D datasets and elaborated on the idea of preserving essential context information bruckner_illustrative_2005 ; bruckner_illustrative_2006 ; bruckner_enhancing_2007 . These and later refinements are integrated in the VolumeShop-system that is publicly available and used by several research groups

2.6 Multi-subject Data

Medical visualization has also started to work on the problem of dealing with multi-subject data. These are datasets that include measurements, including imaging, of more than one subject. The goal is to be able to extract patterns that affect sub-groups of the whole collection, for example to explore which aspects of the data correlate with a specific disease outcome. Examples of this type of work include LifeLines2, an information visualization approach to visualize and compare multiple patient histories or electronic medical records wang_aligning_2008 . More recently, work has been done on the interactive visualization of the multi-subject and mixed modality datasets acquired by medical cohort studies steenwijk_integrated_2010 . In these studies, mixed modality data, including imaging, genetics, blood measurements and so on, is acquired from a group of subjects in order to understand, diagnose or predict the clinical outcome of that group. Steenwijk et al. demonstrated that it was possible to create a highly interactive coupled view visualization interface, integrating both information and scientific visualization techniques, with which patterns, and also hypotheses, could be extracted from the whole data collection.

3 Challenges in Medical Visualization

3.1 Advances in Data Acquisition

Toshiba’s 320-slice CT scanner, the Aquilion One, was introduced in 2007. It is able to acquire five 320 slice volumes per second hsiao_ct_2010 and can thus image a beating heart. Rapid and continuous advances in the dynamic nature and sheer magnitude of data in this and other mainstream medical imaging necessitates improvements to existing techniques in terms of computational and perceptual scalability.

High Angular Resolution Diffusion Imaging (HARDI) tuch_high_2002 and Diffusion Spectrum Imaging (DSI) hagmann_understanding_2006 datasets contain hundreds of diffusion-weighted volumes describing the diffusion of water molecules and hence indirectly the orientation of directed structures such as neural fiber bundles or muscle fibers. This is a rather extreme example of multi-field medical data that is becoming more relevant in both medical research and clinical application. Completely new visual metaphors are required to cope with the highly multi-variate and three-dimensional data of diffusion weighted imaging in particular and many other new imaging modalities in general.

Molecular imaging enables the in vivo imaging of biochemical processes at macroscopic level, meaning that, for example, pathological processes can be studied and followed over time in the same subject long before large-scale anatomical changes occur. Examples are bioluminescence (BLI) and fluorescence (FLI) imaging, two molecular imaging modalities that enable the in vivo imaging of gene expression. Molecular imaging yields datasets that vary greatly in scale, sensitivity, spatial-temporal embedding and in the phenomena that can be imaged. Each permutation brings with it new domain-specific questions and visualization challenges. Up to now, most of the visualization research has been focused on small animal imaging kok_integrated_2007 ; kok_articulated_2010 , but due to its great diagnostic potential, molecular imaging will see increased application in humans.

The integration of microscopy imaging is an essential task for the future, where data handling, interaction facilities but also more classical rendering tasks such as transfer function design become essential. With more and more large scale and 3D microscopy data available, there are many opportunities for visualization researchers. Recent examples include techniques for interactively visualizing large-scale biomedical image stacks demonstrated on datasets of up to 160 gigapixels won-ki_jeong_interactive_2010 and tools for the interactive segmentation and visualization of large-scale 3D neuroscience datasets, demonstrated on a 43 gigabyte electron microscopy volume dataset of the hippocampus jeong_ssecrett_2011 .

With these examples, we hope to have illustrated that advances in image acquisition are continuous, and due to the increasing demands of modern society are accelerating. Each new advance in imaging brings potentially greater magnitudes and exotic new types of data leading to new challenges for medical visualization.

3.2 Heterogeneous Display and Computing Devices

Mobile devices, in particular the Apple products iPad and iPhone, are extremely popular among medical doctors and indeed solve some serious problems of desktop devices in routine clinical use. In particular, bedside use of patient data is an essential use case for medical doctors of various disciplines.

Meanwhile several mobile devices are equipped with powerful graphics cards and using the OpenGL ES (Embedded Systems) standard, they are able to provide high-quality interactive rendering. Although the performance still tails that of modern desktop devices, slicing medical volume data and 3D rendering is feasible moser_interactive_2008 .

The rapid and widespread use of mobile devices also made gesture input popular. In particular, multi-touch interaction is considered an intuitive interaction since many potential users know a variety of gestures from their everyday activities with smart phones. Therefore multitouch interaction is also incorporated in large displays in medical use, e.g. the Digital Lightbox111 by BrainLab and the multi-touch table of Lundström et al. lundstrom_multi-touch_2011 .

3.3 Interactive Image Segmentation

Image segmentation is important in clinical practice, for example, diagnosis and therapy planning, and also in image-based medical research. In these applications, segmentation is complicated by the great deal of variation in image acquisition, pathology and anatomy. Furthermore, in matters of diagnosis or therapy planning, the accuracy of the segmentation can be critical. It comes as no surprise that user interaction is often required to help ensure the quality of the results, by initializing an image processing method, checking the accuracy of the result or to correct a segmentation olabarriaga_interaction_2001 .

A well-known interactive segmentation technique is the live-wire or intelligent scissors mortensen_intelligent_1995 . These ideas were later extended and applied to medical images falcao_user-steered_1998 . More recently, visualization has been applied to the challenge of explicitly dealing with the uncertainty inherent in interactive 3D segmentation prassni_uncertainty-aware_2010 ; saad_exploration_2010 .

Medical visualization research often combines elements of image analysis, graphics and interaction, and is thus ideally equipped to address the challenge of developing and validating effective interactive segmentation approaches for widespread use in medical research and practice.

3.4 Topological Methods

Topological data representation has played an important role in medical visualization, since it can allow us to segment specific features such as human organs and bones from the input medical datasets systematically and identify their spatial relationships consistently. Actually, such topological concepts have been already introduced in the earlier stage of medical visualization. For example, the 3D surface of a human cochlea was reconstructed from a series of 2D CT cross-sectional images by identifying correct correspondence between the cross-sectional contours shinagawa_constructing_reeb-1991 .

Topological approaches have also been extended to analyze 3D medical volume data. Contour trees bajaj_contour_spectrum_1997 have been employed for designing transfer functions in order to illuminate human organs and bones systematically since the associated anatomical structure can be effectively captured as topological skeletons of isosurfaces weber_topology_controlled_2007 . Spatial relationships between bones and the position of aneurysm were successfully extracted respectively from CT and angiographic datasets using a variant of contour tree chan_relation_aware_2008 . Interesting features in medical volume data can be visually analyzed using an optimal camera path, which can be obtained by referring to the topological structure of human organs takahashi_previewing_volume_2011 (see figure 3). Topological methods are now being developed for visualizing multi-variate and high-dimensional datasets, and thus potentially for analyzing tensor fields obtained through DT-MRI, multi-subject data in group fMRI studies, and time-varying data measured by high-speed CTs.

Figure 3: Previewing a sheep heart volume along the optimal camera path. Image taken from takahashi_previewing_volume_2011 .

3.5 Integration of Simulation Models

In their 1999 predictive medicine paper, Taylor et al. argued that surgical planning should not only address questions of surgical approach but also of the expected outcome, for example predicted future states such as the efficacy of a treatment option or the performance of an implant taylor_predictive_1999 . Medical visualization approaches become significantly more valuable when enhanced with simulation models that help to predict the outcome of a disease process or therapeutic procedure, or that enrich measured data with expected physiological phenomena. Examples besides the blood flow simulations of Taylor et al. include interactive skeletal range of motion krekel_interactive_2006 and biomechanical stress dick_stress_2009 simulation models for implant planning in orthopedics and nasal airflow simulation for reconstructive rhinosurgery zachow_visual_2009 .

The integration of these predictive models, although potentially valuable, brings with it new challenges. The addition of potentially complex and dynamic simulation output data to existing visualizations requires new visual representation techniques. Furthermore, for the simulation results to be maximally useful, the models should be tightly coupled to and steered by the user’s interaction with the medical visualization. Finally, most simulations yield data with a certain degree of inherent uncertainty. The role of this uncertainty should be fully explored and it should be carefully but explicitly represented as an integral part of the visualization.

3.6 Mappings and Reformations

In 2002, the American Heart Association proposed a standardised segmentation and accompanying 2D bull’s eye plot (see figure 4) of the myocardium, or heart muscle, of the left heart ventricle cerqueira_standardized_2002 . This 2D plot is a simple but great example of reducing complex 3D data to a standardized 2D representation that greatly facilitates the interpretation of that data. Another well-known example is that of curved planar reformation, or CPR, where volume data is sampled along a curved plane following the trajectory of a blood vessel or other tubular structure, thus enabling the study of the vessel and its surroundings with the minimum of interaction kanitsar_cpr:_2002 . Other good examples of reformation can also be found in brain flattening fischl_cortical_1999 and colon unfolding vilanova_nonlinear_2001 .

Figure 4: On the left, the American Heart Association standardized 2D bull’s eye plot (BEP) of the left ventricle of the heart cerqueira_standardized_2002 . On the right, the volumetric bull’s eye plot, a modernized version by Termeer et al. termeer_covicad:_2007 .

Recently, the idea of intelligently reformatting or mapping 3D data was further explored by Neugebauer et al. with aneurysm maps for the visualization of complex blood flow simulation data on the inside surfaces of aneurysms neugebauer_map_2009 and by Rieder et al. with their tumor maps for the post-operative assessment of radiofrequency ablation therapy rieder_visual_2010 . These types of reformations and mappings entail that more effort has to be put into carefully designing simplified, usually 2D, representations of complex 3D data, as opposed to for example the relatively straight-forward projection of volume data. The resultant visualization, if done right, requires less or no interaction and by definition avoids a number of problems inherent in 3D representations ware_designing_2001 .

3.7 Illustrative Visualization in Medicine

For a long time, it was not possible to apply illustrative visualization techniques in practice due to performance constraints. With advances in graphics hardware and algorithms, such as GPU raycasting kruger_acceleration_2003 , it is now feasible from a computational standpoint. Now that computational problems have been largely solved, illustrative visualization approaches have to be finetuned and evaluated for diagnostic and treatment planning purposes. Recent examples of such work include the simulation of crepuscular rays for tumor accessibility planning khlebnikov_crepuscular_2011 and multi-modal illustrative volume rendering for neurosurgical tumor treatment rieder_interactive_2008 .

Illustrative medical visualization becomes increasingly important when visualizations become more complex and multi-modal, integrating functional (measured and simulated), anatomical information and for example surgical instruments. Illustration techniques enable visual representations to be simplified intelligently by the visualization designer, whilst still communicating as much information as possible. An example of this is the work of Zachow et al. on the visualization of nasal air flow simulation where boundary enhancement was used as an illustrative technique to convey the simulated flow and the anatomy simultaneously zachow_visual_2009 .

3.8 Hyper-realism

Analogous to the case of illustrative visualization, the rapid development in graphics hardware and algorithms has now enabled the interactive rendering of medical imaging datasets with physically-based lighting kroes_exposure_2012 . Figure 5 shows an example of such a visualization. These techniques make possible the simulation of an arbitrary number of arbitrarily shaped and textured lights, real shadows, a realistic camera model with lens and aperture, and so forth, all at interactive rates.

Figure 5: Two examples of interactive visualizations made with the volume renderer of Kroes et al. kroes_exposure_2012 . Through the use of GPUs, physically-based lighting has become possible in an interactive volume rendering setting, enabling increased realism through soft shadows, depth of field and in this case mixed phase function and BRDF surface scattering.

These techniques enable not only photo-realism, but also a technical form of hyper-realism in art, where it is possible to enhance visualizations with additional realistic detail in order to better convey information. Whilst there are strong indications that for example global illumination and shadows can have a positive effect on task performance in normal volume rendering lindemann_about_2011 , the possibilities and value of hyper-realistic effects in medical visualization need to be explored.

3.9 Visual Analysis in Healthcare

Visual analysis is becoming an essential component of medical visualization due to the rapidly growing role and availability of complex multi-dimensional, time-varying, mixed-modality, simulation and multi-subject datasets. In our view, the magnitude and especially the heterogeneity of the data necessitate the use of visual analysis techniques.

Existing examples involving time-varying data include the work of Coto et al. on DCE-MRI mammography coto_mammoexplorer:_2005 and Oeltze et al. on perfusion data in general and MR perfusion of the brain in particular oeltze_interactive_2007 . Blaas et al. applied visual analysis techniques to multi-modal medical data, whilst Zachow et al. focused on nasal airflow simulation data combined with anatomical information zachow_visual_2009 .

There is great potential for visual analysis in medical visualization, with clinical applications including advanced diagnosis and medical research and, even more importantly, treatment planning and evaluation, e.g. radio therapy planning and post-chemotherapy evaluation. The new Visual Analysis in Healthcare (VAHC) workshops that were held at IEEE VisWeek in 2010 and 2011 underline the emerging importance of this research direction.

3.10 Population Imaging

In population imaging, medical image data and other measurements are acquired of a large group of subjects, typically more than one thousand, over a longer period, typically years, in order to study the onset and progression of disease, general aging effects, and so forth in larger groups of people. Examples include the Rotterdam Scan Study focusing on neuro-degeneration de_leeuw_prevalence_2001 and the Study of Health In Pomerania (SHIP) focusing on general health john_study_2001 .

This application domain is an extreme example of multi-subject medical visualization discussed in section 2, integrating large quantities of heterogeneous, multi-modal and multi-timepoint data acquired of a large group of subjects. The scientists running these studies usually do not formulate strictly-defined hypotheses beforehand, instead opting for meticulous data acquisition, followed by an extended period of analysis in order to extract patterns and hypotheses from the data. Recently, Steenwijk et al. set the first steps for the visualization of population imaging by applying visual analysis techniques to cohort study imaging data steenwijk_integrated_2010 . The extreme heterogeneity and magnitude of the data, coupled with the explorative nature of the research, renders this a promising long-term application domain for visual analysis and medical visualization.

4 Conclusions

In this chapter, we gave a compact overview of the history of medical visualization research, spanning the past 30 years. Based on this history and on our own observations working in the field, we then identified and discussed the research challenges of the coming decade.

Our discussion of classic medical visualization problems related to efficient and high quality display of one

static dataset was brief. We devoted more space to data that change over time, to the integration of anatomy with simulation and finally to cohort studies. We refer to problems where such time-dependent and high-dimensional data are employed as

MedVis 2.0 problems. While the classic problems are – from an application perspective – solved, there are many research opportunities in MedVis 2.0 problems. These data are significantly more difficult to analyze, to process and to visualize. Time-dependent MRI data, e.g., exhibit all artifacts of static MRI data but a number of additional artifacts, e.g. due to motion. Integrated analysis and visualization is a key feature of MedVis 2.0 solutions. In general, successful solutions to these problems require a considerably deeper understanding of the medical background and thus favor close collaborations with medical doctors over merely having access to medical data.


  • (1) Altobelli, D.E., Kikinis, R., Mulliken, J.B., Cline, H., Lorensen, W., Jolesz, F.: Computer-assisted three-dimensional planning in craniofacial surgery. Plastic and Reconstructive Surgery 92(4), 576–585; discussion 586–587 (1993). PMID: 8356120
  • (2) Bajaj, C.L., Pascucci, V., Schikore, D.R.: The contour spectrum. In: IEEE Conference on Visualization, 1997. Visualization ’97. Proceedings, pp. 167–173. IEEE (1997). DOI 10.1109/VISUAL.1997.663875
  • (3) Bartz, D.: Virtual endoscopy in research and clinical practice. Computer Graphics Forum 24(1), 111–126 (2005). DOI 10.1111/j.1467-8659.2005.00831.x
  • (4) Basser, P.: Fiber-tractography via diffusion tensor MRI (DT-MRI). In: Proceedings of the 6th Annual Meeting ISMRM, Sydney, Australia, vol. 1226 (1998)
  • (5) Basser, P., Mattiello, J., LeBihan, D.: MR diffusion tensor spectroscopy and imaging. Biophysical Journal 66(1), 259–267 (1994). DOI 10.1016/S0006-3495(94)80775-1
  • (6) Basser, P.J., Pajevic, S., Pierpaoli, C., Duda, J., Aldroubi, A.: In vivo fiber tractography using DT-MRI data. Magnetic Resonance in Medicine: Official Journal of the Society of Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine 44(4), 625–632 (2000). PMID: 11025519
  • (7) Behrens, U., Teubner, J., Evertsz, C., Walz, M., Jürgens, H., Peitgen, H.O.: Computer-assisted dynamic evaluation of contrast-enhanced-MRI. In: Proc. of Computer Assisted Radiology, pp. 362–367 (1996)
  • (8) Beyer, J., Hadwiger, M., Wolfsberger, S., Buhler, K.: High-quality multimodal volume rendering for preoperative planning of neurosurgical interventions. Visualization and Computer Graphics, IEEE Transactions on 13(6), 1696 –1703 (2007). DOI 10.1109/TVCG.2007.70560
  • (9) Blaas, J., Botha, C.P., Post, F.H.: Interactive visualization of multi-field medical data using linked physical and feature-space views. In: K. Museth, A. Ynnerman, T. Möller (eds.) Proc. Eurographics / IEEE-VGTC EuroVis, p. 123–130 (2007)
  • (10) Bruckner, S., Grimm, S., Kanitsar, A., Gr?ller, M.E.: Illustrative context-preserving exploration of volume data. IEEE Transactions on Visualization and Computer Graphics 12(6), 1559–1569 (2006). DOI
  • (11) Bruckner, S., Grimm, S., Kanitsar, A., Gröller, M.: Illustrative context-preserving volume rendering. In: Proceedings of EUROVIS, vol. 2005, p. 69–76 (2005)
  • (12) Bruckner, S., Gröller, E.: Enhancing depth-perception with flexible volumetric halos. IEEE Transactions on Visualization and Computer Graphics 13(6), 1344–1351 (2007). DOI
  • (13) Cai, W., Sakas, G.: Data intermixing and multi-volume rendering. Computer Graphics Forum 18(3), 359–368 (1999). DOI 10.1111/1467-8659.00356
  • (14) Cerqueira, M.D., Weissman, N.J., Dilsizian, V., Jacobs, A.K., Kaul, S., Laskey, W.K., Pennell, D.J., Rumberger, J.A., Ryan, T., Verani, M.S.: Standardized myocardial segmentation and nomenclature for tomographic imaging of the heart. Circulation 105(4), 539 –542 (2002). DOI 10.1161/hc0402.102975
  • (15) Chan, M.Y., Qu, H., Chung, K.K., Mak, W.H., Wu, Y.: Relation-aware volume exploration pipeline. IEEE Transactions on Visualization and Computer Graphics 14(6), 1683–1690 (2008). DOI 10.1109/TVCG.2008.159
  • (16) Coto, E., Grimm, S., Bruckner, S., Gröller, E., Kanitsar, A., Rodriguez, O.: MammoExplorer: an advanced CAD application for breast DCE-MRI. Proc. Vision, Modelling, and Visualization (VMV) p. 91–98 (2005)
  • (17) Csébfalvi, B., Mroz, L., Hauser, H., König, A., Gröller, E.: Fast visualization of object contours by Non-Photorealistic volume rendering. Computer Graphics Forum 20(3), 452–460 (2001). DOI 10.1111/1467-8659.00538
  • (18) Dick, C., Georgii, J., Burgkart, R., Westermann, R.: Stress tensor field visualization for implant planning in orthopedics. IEEE Transactions on Visualization and Computer Graphics 15(6), 1399–1406 (2009). DOI 10.1109/TVCG.2009.184
  • (19) Drebin, R.A., Carpenter, L., Hanrahan, P.: Volume rendering. SIGGRAPH Comput. Graph. 22(4), 65–74 (1988). DOI 10.1145/378456.378484
  • (20) Ebert, D., Rheingans, P.: Volume illustration: non-photorealistic rendering of volume models. In: Proceedings of the conference on Visualization ’00, VIS ’00, p. 195–202. IEEE Computer Society Press, Los Alamitos, CA, USA (2000)
  • (21) Falcão, A.X., Udupa, J.K., Samarasekera, S., Sharma, S., Hirsch, B.E., Lotufo, R.d.A.: User-steered image segmentation paradigms: Live wire and live lane. Graphical Models and Image Processing 60(4), 233–260 (1998). DOI 10.1006/gmip.1998.0475
  • (22) Fischl, B., Sereno, M.I., Dale, A.M.: Cortical surface-based analysis: II: inflation, flattening, and a surface-based coordinate system. NeuroImage 9(2), 195–207 (1999). DOI 10.1006/nimg.1998.0396
  • (23) Gerig, G., Koller, T., Székely, G., Brechbühler, C., Kübler, O.: Symbolic description of 3-d structures applied to cerebral vessel tree obtained from MR angiography volume data. In: H.H. Barrett, A.F. Gmitro (eds.) Information Processing in Medical Imaging, vol. 687, pp. 94–111. Springer-Verlag, Berlin/Heidelberg (1993)
  • (24) Greicius, M.D., Krasnow, B., Reiss, A.L., Menon, V.: Functional connectivity in the resting brain: A network analysis of the default mode hypothesis. Proceedings of the National Academy of Sciences 100(1), 253 –258 (2003). DOI 10.1073/pnas.0135058100
  • (25) Hagmann, P., Jonasson, L., Maeder, P., Thiran, J.P., Wedeen, V.J., Meuli, R.: Understanding diffusion MR imaging techniques: from scalar diffusion-weighted imaging to diffusion tensor imaging and beyond. Radiographics: A Review Publication of the Radiological Society of North America, Inc 26 Suppl 1, S205–223 (2006). DOI 10.1148/rg.26si065510. PMID: 17050517
  • (26) Höhne, K.H., Bernstein, R.: Shading 3D-Images from CT using gray-level gradients. IEEE Transactions on Medical Imaging 5, 45–47 (1986). DOI 10.1109/TMI.1986.4307738
  • (27) Höhne, K.H., Bomans, M., Tiede, U., Riemer, M.: Display of multiple 3D-objects using the generalized voxel-model. In: Medical Imaging II, Part B, Proc. SPIE 914, pp. 850–854. Newport Beach (1988)
  • (28) Hong, L., Kaufman, A., Wei, Y.C., Viswambharan, A., Wax, M., Liang, Z.: 3D virtual colonoscopy. In: Biomedical Visualization, 1995. Proceedings., pp. 26–32, 83. IEEE (1995). DOI 10.1109/BIOVIS.1995.528702
  • (29) Hong, L., Muraki, S., Kaufman, A., Bartz, D., He, T.: Virtual voyage: interactive navigation in the human colon. In: Proceedings of the 24th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’97, p. 27–34. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1997). DOI 10.1145/258734.258750
  • (30) Hong, W., Qiu, F., Kaufman, A.: A pipeline for computer aided polyp detection. IEEE Transactions on Visualization and Computer Graphics 12(5), 861 –868 (2006). DOI 10.1109/TVCG.2006.112
  • (31) Hsiao, E.M., Rybicki, F.J., Steigner, M.: CT coronary angiography: 256-slice and 320-detector row scanners. Current cardiology reports 12(1), 68–75 (2010). DOI 10.1007/s11886-009-0075-z. PMID: 20425186 PMCID: 2893879
  • (32) Interrante, V., Fuchs, H., Pizer, S.: Enhancing transparent skin surfaces with ridge and valley lines. In: IEEE Conference on Visualization, 1995. Visualization ’95. Proceedings, pp. 52–59, 438. IEEE (1995). DOI 10.1109/VISUAL.1995.480795
  • (33) Jeong, W., Beyer, J., Hadwiger, M., Blue, R., Law, C., Vazquez, A., Reid, C., Lichtman, J., Pfister, H.: SSECRETT and NeuroTrace: interactive visualization and anaysis tools for large-scale neuroscience datasets. IEEE Computer Graphics and Applications PP(99), 1–1 (2011). DOI 10.1109/MCG.2011.33
  • (34) Jeong, W.K., Schneider, J., Turney, S.G., Faulkner-Jones, B.E., Meyer, D., Westermann, R., Reid, R.C., Lichtman, J., Pfister, H.: Interactive histology of large-scale biomedical image stacks. IEEE Transactions on Visualization and Computer Graphics 16(6), 1386–1395 (2010). DOI 10.1109/TVCG.2010.168
  • (35) John, U., Hensel, E., Lüdemann, J., Piek, M., Sauer, S., Adam, C., Born, G., Alte, D., Greiser, E., Haertel, U., Hense, H.W., Haerting, J., Willich, S., Kessler, C.: Study of health in pomerania (SHIP): a health examination survey in an east german region: Objectives and design. Social and Preventive Medicine 46, 186–194 (2001). DOI 10.1007/BF01324255
  • (36) Kanitsar, A., Fleischmann, D., Wegenkittl, R., Felkel, P., Gröller, M.E.: CPR: curved planar reformation. In: Proceedings of the conference on Visualization ’02, VIS ’02, p. 37–44. IEEE Computer Society, Washington, DC, USA (2002)
  • (37) Khlebnikov, R., Kainz, B., Muehl, J., Schmalstieg, D.: Crepuscular rays for tumor accessibility planning. IEEE Transactions on Visualization and Computer Graphics 17(12), 2163–2172 (2011). DOI 10.1109/TVCG.2011.184
  • (38) Kindlmann, G., Weinstein, D.: Hue-balls and lit-tensors for direct volume rendering of diffusion tensor fields. In: Proceedings of the conference on Visualization ’99: celebrating ten years, VIS ’99, p. 183–189. IEEE Computer Society Press, Los Alamitos, CA, USA (1999)
  • (39) Kindlmann, G., Weinstein, D., Hart, D.: Strategies for direct volume rendering of diffusion tensor fields. IEEE Transactions on Visualization and Computer Graphics 6(2), 124–138 (2000). DOI 10.1109/2945.856994
  • (40) Kindlmann, G., Whitaker, R., Tasdizen, T., Moller, T.: Curvature-based transfer functions for direct volume rendering: Methods and applications. In: Proceedings of the 14th IEEE Visualization 2003 (VIS’03), VIS ’03, p. 67–. IEEE Computer Society, Washington, DC, USA (2003). DOI 10.1109/VISUAL.2003.1250414
  • (41) Kok, P., Baiker, M., Hendriks, E.A., Post, F.H., Dijkstra, J., Löwik, C.W., Lelieveldt, B.P., Botha, C.P.: Articulated planar reformation for change visualization in small animal imaging. IEEE Transactions on Visualization and Computer Graphics 16(6), 1396–1404 (2010). DOI 10.1109/TVCG.2010.134
  • (42) Kok, P., Dijkstra, J., Botha, C.P., Post, F.H., Kaijzel, E., Que, I., Lowik, C., Reiber, J., Lelieveldt, B.P.F.: Integrated visualization of multi-angle bioluminescence imaging and micro CT. In: K.R. Cleary, M.I. Miga (eds.) Proc. SPIE Medical Imaging 2007, vol. 6509 (2007)
  • (43) Krekel, P.R., Botha, C.P., Valstar, E.R., Bruin, P.W.d., Rozing, P.M., Post, F.H.: Interactive simulation and comparative visualisation of the bone-determined range of motion of the human shoulder. In: T. Schulze, G. Horton, B. Preim, S. Schlechtweg (eds.) Proceedings of Simulation and Visualization, p. 275–288. SCS Publishing House Erlangen (2006). Best Paper Award.
  • (44) Kroes, T., Post, F.H., Botha, C.P.: Exposure render: An interactive photo-realistic volume rendering framework. PLoS ONE (2012)
  • (45) Krüger, J., Westermann, R.: Acceleration techniques for GPU-based volume rendering. In: Visualization Conference, IEEE, p. 38. IEEE Computer Society, Los Alamitos, CA, USA (2003). DOI
  • (46) de Leeuw, F.E., de Groot, J.C., Achten, E., Oudkerk, M., Ramos, L.M.P., Heijboer, R., Hofman, A., Jolles, J., van Gijn, J., Breteler, M.M.B.: Prevalence of cerebral white matter lesions in elderly people: a population based magnetic resonance imaging study. the rotterdam scan study. Journal of Neurology, Neurosurgery & Psychiatry 70(1), 9 –14 (2001). DOI 10.1136/jnnp.70.1.9
  • (47) Levoy, M.: Display of surfaces from volume data. IEEE Computer Graphics and Applications 8(3), 29–37 (1988). DOI 10.1109/38.511
  • (48) Lindemann, F., Ropinski, T.: About the influence of illumination models on image comprehension in direct volume rendering. IEEE Transactions on Visualization and Computer Graphics 17(12), 1922–1931 (2011). DOI 10.1109/TVCG.2011.161
  • (49) Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3D surface construction algorithm. SIGGRAPH Comput. Graph. 21(4), 163–169 (1987). DOI 10.1145/37402.37422
  • (50) Lundström, C., Rydell, T., Forsell, C., Persson, A., Ynnerman, A.: Multi-touch table system for medical visualization: application to orthopedic surgery planning. IEEE Transactions on Visualization and Computer Graphics 17(12), 1775–1784 (2011). DOI 10.1109/TVCG.2011.224. PMID: 22034294
  • (51) Mortensen, E.N., Barrett, W.A.: Intelligent scissors for image composition. In: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, SIGGRAPH ’95, p. 191–198. ACM, New York, NY, USA (1995). DOI 10.1145/218380.218442
  • (52) Moser, M., Weiskopf, D.: Interactive volume rendering on mobile devices. Vision, modeling, and visualization 2008: proceedings, October 8-10, 2008, Konstanz, Germany p. 217 (2008)
  • (53) Neugebauer, M., Gasteiger, R., Beuing, O., Diehl, V., Skalej, M., Preim, B.: Map displays for the analysis of scalar data on cerebral aneurysm surfaces. Computer Graphics Forum 28(3), 895–902 (2009). DOI 10.1111/j.1467-8659.2009.01459.x
  • (54) Oeltze, S., Doleisch, H., Hauser, H., Muigg, P., Preim, B.: Interactive visual analysis of perfusion data. IEEE Transactions on Visualization and Computer Graphics 13(6), 1392–1399 (2007). DOI
  • (55) Olabarriaga, S., Smeulders, A.: Interaction in the segmentation of medical images: A survey. Medical Image Analysis 5(2), 127–142 (2001). DOI 10.1016/S1361-8415(00)00041-4
  • (56) Prassni, J.S., Ropinski, T., Hinrichs, K.: Uncertainty-aware guided volume segmentation. IEEE Transactions on Visualization and Computer Graphics 16(6), 1358–1365 (2010). DOI 10.1109/TVCG.2010.208
  • (57) Preim, B., Bartz, D.: Visualization in Medicine. Morgan Kaufmann (2007)
  • (58) Rieder, C., Ritter, F., Raspe, M., Peitgen, H.: Interactive visualization of multimodal volume data for neurosurgical tumor treatment. In: Computer Graphics Forum, vol. 27, p. 1055–1062 (2008)
  • (59) Rieder, C., Weihusen, A., Schumann, C., Zidowitz, S., Peitgen, H.: Visual support for interactive Post-Interventional assessment of radiofrequency ablation therapy. Computer Graphics Forum 29(3), 1093–1102 (2010). DOI 10.1111/j.1467-8659.2009.01665.x
  • (60) Saad, A., Hamarneh, G., Mo?ller, T.: Exploration and visualization of segmentation uncertainty using shape and appearance prior information. IEEE Transactions on Visualization and Computer Graphics 16(6), 1366–1375 (2010). DOI 10.1109/TVCG.2010.152
  • (61) Saito, T., Takahashi, T.: Comprehensible rendering of 3-d shapes. SIGGRAPH Comput. Graph. 24(4), 197–206 (1990). DOI 10.1145/97880.97901
  • (62) Shinagawa, Y., Kunii, T.L.: Constructing a Reeb graph automatically from cross sections. IEEE Computer Graphics and Applications 11(6), 44–51 (1991). DOI 10.1109/38.103393
  • (63) Steenwijk, M.D., Milles, J., Buchem, M.A., Reiber, J.H., Botha, C.P.: Integrated visual analysis for heterogeneous datasets in cohort studies. In: IEEE VisWeek Workshop on Visual Analytics in Health Care (2010)
  • (64) Sunguroff, A., Greenberg, D.: Computer generated images for medical applications. In: Proceedings of the 5th annual conference on Computer graphics and interactive techniques, SIGGRAPH ’78, p. 196–202. ACM, New York, NY, USA (1978). DOI 10.1145/800248.807390
  • (65) Takahashi, S., Fujishiro, I., Takeshima, Y., Bi, C.: Previewing volume decomposition through optimal viewpoints. In: Scientific Visualization: Interactions, Features, Metaphors (2011)
  • (66) Taylor, C., Draney, M., Ku, J., Parker, D., Steele, B., Wang, K., Zarins, C.: Predictive medicine: computational techniques in therapeutic decision-making. Computer Aided Surgery 4(5), 231–247 (1999)
  • (67) Termeer, M., Bescos, J.O., Breeuwer, M., Vilanova, A., Gerritsen, F., Groller, M.E.: CoViCAD: comprehensive visualization of coronary artery disease. IEEE Transactions on Visualization and Computer Graphics 13(6), 1632–1639 (2007). DOI 10.1109/TVCG.2007.70550
  • (68) Tietjen, C., Isenberg, T., Preim, B.: Combining silhouettes, surface, and volume rendering for surgery education and planning. In: IEEE/Eurographics Symposium on Visualization (EuroVis), p. 303–310 (2005)
  • (69) Tory, M., Röber, N., Möller, T., Celler, A., Atkins, M.S.: 4D space-time techniques: a medical imaging case study. In: Proc. IEEE Visualization 2001, p. 473–476. IEEE Computer Society, Washington, DC, USA (2001)
  • (70) Tuch, D.S., Reese, T.G., Wiegell, M.R., Makris, N., Belliveau, J.W., Wedeen, V.J.: High angular resolution diffusion imaging reveals intravoxel white matter fiber heterogeneity. Magnetic Resonance in Medicine 48(4), 577–582 (2002). DOI 10.1002/mrm.10268
  • (71) Vannier, M.W., Marsh, J.L., Warren, J.O.: Three dimensional computer graphics for craniofacial surgical planning and evaluation. SIGGRAPH Comput. Graph. 17(3), 263–273 (1983). DOI 10.1145/964967.801157
  • (72) Vilanova, A., Wegenkittl, R., König, A., Gröller, E.: Nonlinear virtual colon unfolding. In: Proceedings of the conference on Visualization ’01, VIS ’01, p. 411–420. IEEE Computer Society, Washington, DC, USA (2001)
  • (73) Wang, T.D., Plaisant, C., Quinn, A.J., Stanchak, R., Murphy, S., Shneiderman, B.: Aligning temporal data by sentinel events: discovering patterns in electronic health records. In: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI ’08, p. 457–466. ACM, New York, NY, USA (2008). DOI 10.1145/1357054.1357129
  • (74) Ware, C.: Designing with a 2 1/2-d attitude. Information Design Journal 10(3), 258–65 (2001). DOI 10.1075/idj.10.3.07war
  • (75) Weber, G.H., Dillard, S.E., Carr, H., Pascucci, V., Hamann, B.: Topology-controlled volume rendering. IEEE Transactions on Visualization and Computer Graphics 13(2), 330–341 (2007). DOI 10.1109/TVCG.2007.47
  • (76) Weinstein, D., Kindlmann, G., Lundberg, E.: Tensorlines: advection-diffusion based propagation through diffusion tensor fields. In: Proceedings of the conference on Visualization ’99: celebrating ten years, VIS ’99, p. 249–253. IEEE Computer Society Press, Los Alamitos, CA, USA (1999)
  • (77) Zachow, S., Muigg, P., Hildebrandt, T., Doleisch, H., Hege, H.C.: Visual exploration of nasal airflow. IEEE Transactions on Visualization and Computer Graphics 15(6), 1407–1414 (2009). DOI