Antara: An Interactive 3D Volume Rendering and Visualization Framework

12/11/2018
by   Pratik Kalshetti, et al.
0

The goal of 3D visualization is to provide the user with an intuitive interface which enables him to explore the 3D data in an interactive manner. The aim of the exploration is to identify and analyze anomalies or to give proof of the non-anomaly of the visualized organic structures. For 3D Medical Data, Magnetic Resonance Images (MRI) has been used. To create the 3D model, we used the Direct Volume Rendering technique. In the input 3D data, we have x, y and z coordinates and an intensity value for each voxel. The 3D data is used by Volume Ray Casting to compute 2D projections from 3D volumetric data sets. In ray casting, a ray of light is made to pass through the volume data. The interaction of each voxel with this ray is used to assign RGB and alpha values for every voxel in the volume. As a result, we are able to generate the 3D model of the region of interest using the 3D data. The 3D model is interactive, thus enabling us to visualize the different layers of the 3D volume by adjusting the transfer function.

READ FULL TEXT VIEW PDF

Authors

page 1

page 6

page 7

page 8

page 9

09/28/2021

Common Artifacts in Volume Rendering

Direct Volume Rendering is a popular and powerful visualization method f...
02/21/2018

Medical Volume Reconstruction Techniques

Medical visualization is the use of computers to create 3D images from m...
08/13/2020

Interactive volume illumination of slice-based ray casting

Volume rendering always plays an important role in the field of medical ...
11/19/2021

DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering

DIVeR builds on the key ideas of NeRF and its variants – density models ...
08/19/2020

Deep Volumetric Ambient Occlusion

We present a novel deep learning based technique for volumetric ambient ...
11/29/2017

A Novel Image-centric Approach Towards Direct Volume Rendering

Transfer Function (TF) generation is a fundamental problem in Direct Vol...
07/08/2014

Visualization of Large Volumetric Multi-Channel Microscopy Data Streams on Standard PCs

Background: Visualization of multi-channel microscopy data plays a vital...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In medical field, visualizing the internal structure of body is very important for proper medical diagnosis [7]. To aid doctors in visualizing the internal body parts, scanning methods like MRI (Magnetic Resonance Imaging), X-ray and CT (Computed Tomography) scan are used. Output of such methods are generally one or more grayscale 2D images[15]. It is difficult to visualize the internal structure with the help of these 2D images.

To solve this problem, we propose an interactive 3D volume rendering and visualization framework Antara (Sanskrit: Interior) from the data obtained using MRI scan as shown in Fig. 1. This has been achieved using Volume Rendering [11]. Volume rendering is an important graphics and visualization technique. A rendered volume can be used for displaying not only surfaces of a model but also the intricate details contained within. In order to achieve our goal, we chose Direct Volume Rendering (DVR) technique for 3D visualization. It helps to generate an interactive 3D model which can be altered to show surfaces having different intensities[9]. DVR methods generate images of a 3D volumetric data set without explicitly extracting geometric surfaces from the data. These techniques use an optical model to map data values to optical properties, such as color and opacity. During rendering, these optical properties are accumulated along each viewing ray to form an image of the data. This helps in generating accurate images of the part being visualized [2].

Figure 1: Extracting 3D bone from MRI of knee.

2 Literature Survey

Most of the fundamental volume visualization algorithms can be divided into two categories: DVR algorithms and Surface Fitting (SF) algorithms. SF algorithms fit (usually planar) surface primitives such as polygons or patches to constant-value contour surfaces in volumetric data-sets. The user begins by choosing a particular threshold value and the geometric primitives are then automatically fit to the high contrast contours in the volume that match the threshold. Cells whose corner values are all above the chosen threshold or all below the threshold are removed and thus have no effect on the final image. SF algorithms constitute few of the algorithms like opaque cubes, marching cubes, dividing cubes and marching tetrahedra[4]. SF methods are typically faster than DVR methods since SF methods only traverse the volume once to extract the surfaces. After extracting the surfaces, rendering hardware and other rendering methods can be used to quickly render the surface primitives each time the user changes a viewing or lighting parameter. Changing the SF threshold value is time consuming because it requires that all of the cells be revisited to extract a new set of surface primitives. Also, SF methods suffer from several problems such as occasional false positive and negative surface pieces and incorrect handling of small features and branches in the data. Also, it only displays surfaces which meet a threshold density and will only show the surface that is located closest to the imaginary viewer.

DVR methods perform mapping of elements directly into the screen without using geometric primitives as an intermediate representation. DVR algorithms include approaches such as ray-casting, integration methods, splatting, and V-buffer rendering. In DVR method, transparency and colors are used to allow a better representation of the volume to be depicted in a single image. DVR retains all of the volume data during rendering, and thus all the acquired voxels may contribute to each rendered image. Spatial gradients is computed prior to or during rendering, and these gradients, as well as the relative contributions of each voxel, is adjusted to produce images of surfaces. Thus volume rendering can produce images of surfaces. Therefore, if volume rendering of medical data sets is inexpensive and fast enough, surface rendering will no longer be utilized.

Figure 2: Overall framework of Antara.

Ray-casting[12] is the popular technique of volume visualization for the production of high quality images. It conducts an image-order traversal of the image plane pixels, finding a color and opacity for each. The color tuple and opacity values are added to the pixel using a weighting formula and a step is taken along the ray to the next re-sample point. In ray casting, rays continue in a straight line until the opacity encountered by the ray sums to unity or the ray exits the rear of the volume. No shadows or reflections are generated in ray-casting. Because the magnitude of the gradient is usually a good indication of the strength of an iso-surface within a cell, the result is a large color contribution to the pixel when the ray encounters a new substance in the volume[8]. Ray-casting is CPU-intensive, but the images show the entire data-set, not just a collection of thin surfaces as in SF. 3D models were created using Ray casting algorithm [10] which is a volume rendering technique. Volume rendering techniques are computationally intensive, whereas the ray tracing algorithm [18, 3, 1], is faster in terms of running time and avoiding redundant computations.

The important objectives in visualizing medical data is high quality rendering and provide real time interaction. This paper highlights the use of ray tracing for volume rendering in the medical domain for getting an insight into the interior stucture of the organs of the body using DVR, thus producing high quality rendered image focusing even minute anomalies in the bones and tissues. Also the real time interaction is achieved with the recusive formulation of the rendering equation. Another novel highlight of the paper is the metric used for analysing the result of the volume rendering. The idea is an extension of the image segmentation metric used in 2D images, thus opening gates to use such metrics for 3D data.

3 System Overview

The overall approach comprises of the modules explained in Fig. 2. Representing a surface contained within a volumetric data set using geometric primitives can be useful in many applications, however, there are several main drawbacks to this approach [14]. First, geometric primitives can only approximate surfaces contained within the original data. Adequate approximations may require an excessive amount of geometric primitives. Therefore, a trade-off must be made between accuracy and space requirements. Second, since only a surface representation is used, much of the information contained within the data is lost during the rendering process [17]. For example, in CT scanned data useful information is contained not only on the surfaces, but within the data as well. Also, amorphous phenomena, such as clouds, fog, and fire cannot be adequately represented using surfaces, and therefore must have a volumetric representation, and must be displayed using volume rendering techniques.

3.1 Volumetric Function Interpolation

The volume grid only defines the value of some measured property at discrete locations in space. If one requires the value of at an off-grid location

, a process called interpolation must be employed to estimate the unknown value from the known grid samples

. We have used trilinear interpolation in this case. It can be written as 3 stages of 7 linear interpolations, since the filter function is separable in higher dimensions. The first 4 linear interpolations are along :

(1)

Using these results, 2 linear interpolations along follow:

(2)

and, interpolation along yields the result:

(3)

Here the are the distances (cell of size of 13 (empirically determined)) of the sample at from the lower left rear voxel in the cell containing the sample point.

Optical models for direct volume rendering view the volume as a cloud of particles. Light from a source can either be scattered or absorbed by particles. The particles in an actual cloud occlude incoming light, as well as add their own glow as shown in Fig. 3.

Figure 3: Optical model used for characterizing the optic properties of the volume.

3.2 Direct Volume Rendering

Volume rendering or DVR is the process of creating a 2D image directly from 3D volumetric data, hence it is often called direct volume rendering[5]. We have used an image-order technique. It includes the following operations:

  1. Cast rays from the image pixels, sampling the grid at discrete locations along their paths.

  2. Obtain the samples via interpolation.

  3. The interpolated samples are further processed to simulate the light transport within a volumetric medium.

The fundamental element in DVR is the volume rendering equation. It is obtained by using an absorption and emission model for particles. The equation is formulated as partial differential equation which is then solved by Ordinary Differential Equation (ODE) solver. Using the Riemann approximation for the integral present in the solution of the partial differential equation, we get the discretized version of the same. The discrete version of the rendering equation

[11] at the distance of the image plane is given by:

(4)

where is the background light, and is the opacity and color value associated the sampled volume data points assuming sampled data points.

Note that the first term in Eq. 4 captures absorption whereas the latter one focuses on the emission. The above equation requires redundant computation for each of the data points which can be resolved by using recursive implementation called the Front-to-Back Compositing which is defined in Eq. 5 and Eq. 6.

(5)

This formulation makes it possible to implement the approach and perform the process in real time.

(6)

Here, and denote the color value before and after any iteration and similarly and denote the opacity value before and after any iteration.

3.3 Raycasting

In nature a light source is always required to visualize any object. Raycasting algorithm is used to determine the color and opacity associated with each voxel present in volume data due to the light source[13]. It is achieved by appling following operation on each voxel [12]:

  1. Cast a ray into the volume

  2. Linearly interpolate data values from voxel corners

  3. Convert data values to optical properties viz. color and opacity

  4. Composite the optical properties

  5. Return the final color to assign to the image plane pixel

The above steps are shown in Fig. 4.

Figure 4: Illustration of the ray casting technique.

3.4 Shading

We have explored various shading models and found Phong Illumination Model (see Eq. 7) best suitable for the purpose of obtaining the color value associated with each sampled data value. This is further emphasised in Sec. 4.5. Here, the normal is computed using central difference method to find the gradient.

(7)

where is the color of the sample ; , , are the light’s ambient, diffuse, and specular components, is the normal at sample ,

is the vector from sample point to eye,

is the light vector from the sample to light source, is the reflection vector and is the shininess. Now once obtained the color value, the next task is to obtain the opacity value which can be obtained from the opacity function.

Figure 5: Effect of varying opacity on rendered objects.

3.5 Opacity Function

This stage computes the opacity value of different data points within the sampled volume. The user provides the opacity value with the help of a transfer function defined as where is the opacity value for all the data points with intensity value of . The rationale behind providing this user interaction is to clearly view the region of interest by setting its opacity to a larger value. Finally, the compositing values obtained, thus forms the pixel values for the image plane which projects the 3D volume data onto a 2D plane.

Figure 6: Illustration of Antara’s rendering process. (I)-(IV): effect of varying opacity values of different voxels.

An illustration of this stage is shown in Fig. 5. Consider two objects in the volume data. The pyramid is blue in color and the sphere is red. When viewed from the bold face of the cuboid as shown, only the sphere is rendered on the image plane since the opacity value for red intensity data points is set to a high value whereas that for blue data points is set to a low value.

3.6 Rendering Algorithm

The overall idea is depicted in Fig. 6. Consider a volume data consisting of four data points with intensity values being red(a), yellow(b), blue(c) and green(d). The blue and red points lie at same depth() while green and yellow lie at a different depth() and the former depth being nearer to the screen. The red and the yellow data point lie on the same ray along the camera to a pixel on the image plane. The opacity function shows the opacity values for each of the intensity values present in the volume data. Since the red and yellow data points are projected on the same pixel on the image, the final value at the pixel is a weighted combination of these two values where the weights are determined by the opacity function which is modified by the user.

The steps have been illustrated below:

  1. For full opacity , intensities , and are visible on the image plane. Intensity has no effect because it is behind (see Fig. 6(I)).

  2. Zero opacity value for and hides them completely in the image plane and intensity (see Fig. 6(II)).

  3. Setting , a mixture of both the intensities is visible on the image plane (see Fig. 6(III)).

  4. When , varying has no significance as it is obstructed completely by (see Fig. 6(IV)).

4 Experimental Results

4.1 Dataset

The dataset used for evaluation of the proposed approach is the volume data obtained via MRI and CT of 5 human organs (brain, foot, knee, tooth and heart). The ground truth is created by manually segmenting the bone region from each of the slices(images) from the scan and then validated by a medical expert. Also we have scans of an engine and a car for proving the proposed algorithm’s generalization capability in other domains.

4.2 Qualitative Analysis

Figure 7: Result on different datasets: (a) Brain CT, (b) Foot CT, (c) Knee MRI, (d) Tooth CT, (e) Heart CT.

Antara’s performance on different datasets is shown in Fig. 7. The colors in the figure are chosen for better visualization. Changing the opacity function for different intensities varies the rendered object, thus giving an insight into the different structures as per the material characteristics. In Fig. 7, the outer structure is obtained by giving high opacity values to them and gradually decreasing this value for the outer structures helps in visualizing the inner structure. The obtained results can this aid in understanding the hidden structure and thus find flaws in it. This will help in better surgery as per the patient needs. The proposed method achieves the objective of high accuracy required in the visualization of medical images, which can render non geometrical shapes with precision. The removal of tissues yields exact visualization of bones, which is possible due to their relative intensity difference.

4.3 Generalization Capability

Antara not only works well with medical data but also works on data from other domains. In Fig. 8 data from automobile sector is worked upon by the proposed algorithm and shows high quality result giving insight into internal structures. This data includes a porche car scan and an engine volume data. This aids in detecting flaws in the internal structure of the machine. This shows that Antara has far reaching effects on different disciplines like city architecture plans, electrical systems and industrial machines.

Figure 8: Result on automobile volume data.

4.4 Quantitative Analysis

The qualitative analysis clearly depicted the efficacy of Antara in the previous section. Quantitative analysis, in terms of accuracy of the proposed algorithm, provides a better analysis of the technique. To evaluate the performance of our segmentation approach, we have used the Dice similarity coefficient index as a measure to quantify the accuracy of the obtained results, where the ground truth has been generated manually and later validated by the doctor.

The adopted technique is a novel method for analysing the volume data. The genesis of this technique can be traced back from the fact that dice coefficient is a technique widely used in validation of segmentation of medical images. Extending this idea, and with the thought that the volume data in MRI is a formed by slice wise images of a particular organ, the rendered image can then be considered as a result obtained by segmenting the corresponding slice from input volume data(MRI slices). The Dice coefficient is given by:

(8)

where is the segmented result of Antara and is the manually segmented ground truth.

Figure 9: Quantitative Evaluation.

As shown in Fig. 9, the dice coefficient of a slice obtained from the rendered volume when compared with the ground truth slice containing only the bone results into a value close to 1 for almost all the slices. Thus, this technique provides higher accuracy results which are essential in the medical domain. The average dice similarity coefficient index obtained over the entire data set used, was found to be .

This value clearly proves the efficiency of the results generated by the proposed approach and hence making it suitable for using in applications where high precision is required. This principle of measuring the result of a 3D data using the methods available for 2D data is in itself an innovation in the analysis phase in the domain of computer graphics.

4.5 Effect of shading and shininess

Shading plays an important role in visualizing a scene. Human vision uses shading as a cue to form position and depth. Total handling of light is very expensive and shading model gives a good approximation of what would really happen much less expensively. The two less expensive techniques (i.e. real time interactive) widely used are Phong model and the Cel shading model. The result obtained by Phong model helps better visualizing the data points that are occluded from the scanning direction as compared to the Cel model. As shown in Fig. 10 the encircled region is not highlighted clearly in the Cel Model, while it is shaded properly to visualize it appropriately.

Phong Model Cel Model
Figure 10: Comparison between Phong and Cel Model.

The importance of incorporating all the three lights viz. ambient, diffused and specular is clear from Fig. 11, where the rendered image by using various combination of these lights is shown. All these images proves that none of these lighting models when used alone or pairwise provide reallistic results, thus making it a necessity to use all the three lights together to provide high quality realistic result.

Figure 11: Effect of shading on the rendered object.

Shininess is the coefficient of specular reflectivity associated with the surface of an object being rendered. Appropriate coefficient needs to be chosen as per the application. All the results shown throughout have been generated with the value of 60 for the coefficient. The variation of shininess has been shown visually in Fig. 12.

Figure 12: Effect of varying shininess values on rendering.

The rendered image in Fig. 13 with all the lighting components included and a value of for the shininess coefficient gives an accurate result obtained in real time. The results are generated from the MRI scan of the knee when viewed from different orthogonal planes. The color is chosen for better visualization. The recommended value for shininess is 60 using the Phong Shading model with all the three light components for the medical dataset.

Axial Coronal Sagittal
Figure 13: Different Views of Knee.

5 Conclusions

Antara works on the basis of differences in the intensities of the components of the 3D data. It is a multi-purpose platform for the analysis of various types of patient data. E.g., analysis of bones as well as tissues, detection of tumors, etc. Antara also works for non-medical domain also (e.g. automobiles). There is a dearth of such low cost, open-source software available in the market for analyzing 3D models of medical images. Such softwares can be used for image guided surgery and can be put to other applications in the medical image analysis field. In future, we are planning to do a organ or tissue specific rendering. Techniques such as [6] can be combined with Antara to let the doctors perform a virtual test surgery.

References

  • [1] Parker, Steven and Parker, Michael and Livnat, Yarden and Sloan, Peter-Pike and Hansen, Charles and Shirley, Peter (2005) Interactive ray tracing for volume visualization In:SIGGRAPH
  • [2] Raman, Siva P and Horton, Karen M and Fishman, Elliot K (2013) MDCT evaluation of ureteral tumors: advantages of 3D reconstruction and volume visualization In: American Journal of Roentgenology
  • [3] Levoy, Marc (1990) Efficient ray tracing of volume data In: ACM TOG
  • [4] Okuyan, Erhan and Güdükbay, Uğur Direct volume rendering of unstructured tetrahedral meshes using CUDA and OpenMP In: The Journal of Supercomputing
  • [5] Drebin, Robert A and Carpenter, Loren and Hanrahan, Pat Volume rendering In: ACM SCG
  • [6] Shrivastava, Prateek and Das, Sukhendu (2014) Particle Coding for Meshfree Cutting of Deformable Assets, In:ICVGIP
  • [7]

    Post, Frits H and Nielson, Gregory and Bonneau, Georges-Pierre (2012) Data visualization: the state of the art In: Springer Science & Business Media.

  • [8] Soler, Luc and Nicolau, Stephane and Pessaux, Patrick and Mutter, Didier and Marescaux, Jacques (2014) Real-time 3D image reconstruction guidance in liver resection surgery In: Hepatobiliary surgery and nutrition.
  • [9] Höhne, Karl Heinz and Bomans, Michael and Pommert, Andreas and Riemer, Martin and Schiers, Carsten and Tiede, Ulf and Wiebecke, Gunnar (1990) 3D visualization of tomographic volume data using the generalized voxel model In: Visual Computing.
  • [10] Adeshina, AM and Hashim, Rathiah and Khalid, Noor Elaiza Abdul and Abidin, Siti ZZ (2012) Hardware-accelerated raycasting: Towards an effective brain MRI visualization In: arXiv:1211.5595.
  • [11] Kajiya, James T(1986) The rendering equation In: SIGGRAPH.
  • [12] Ray, Harvey and Pfister, Hanspeter and Silver, Deborah and Cook, Todd A (1999) Ray casting architectures for volume visualization In: IEEE CGA.
  • [13] J. Meyer-Spradow and T. Ropinski and J. Mensmann and K. Hinrich (2009) Voreen: A Rapid-Prototyping Environment for Ray-Casting-Based Volume Visualizations In: IEEE T-VCG.
  • [14] Nelly Gordillo and Eduard Montseny and Pilar Sobrevill (2013) State of the art survey on {MRI} brain tumor segmentatio In: Magnetic Resonance Imaging.
  • [15] R. A. Robb and C. Barillo (1989) Interactive display and analysis of 3-D medical images In: IEEE T-MI.
  • [16] Levoy, Marc (1990) Efficient Ray Tracing of Volume Data In: ACM TOG.
  • [17] Elvins, T. Todd (1992) A Survey of Algorithms for Volume Visualization In: SIGGRAPH Comput. Graph.
  • [18] Hachisuka, Toshiya and Jarosz, Wojciech and Weistroffer, Richard Peter and Dale, Kevin and Humphreys, Greg and Zwicker, Matthias and Jensen, Henrik Wann (2008) Multidimensional adaptive sampling and reconstruction for ray tracing In: ACM TOG.