Unwind: Interactive Fish Straightening

The ScanAllFish project is a large-scale effort to scan all the world's 33,100 known species of fishes. It has already generated thousands of volumetric CT scans of fish species which are available on open access platforms such as the Open Science Framework. To achieve a scanning rate required for a project of this magnitude, many specimens are grouped together into a single tube and scanned all at once. The resulting data contain many fish which are often bent and twisted to fit into the scanner. Our system, Unwind, is a novel interactive visualization and processing tool which extracts, unbends, and untwists volumetric images of fish with minimal user interaction. Our approach enables scientists to interactively unwarp these volumes to remove the undesired torque and bending using a piecewise-linear skeleton extracted by averaging isosurfaces of a harmonic function connecting the head and tail of each fish. The result is a volumetric dataset of a individual, straight fish in a canonical pose defined by the marine biologist expert user. We have developed Unwind in collaboration with a team of marine biologists. Our system has been deployed in their labs, and is presently being used for dataset construction, biomechanical analysis, and the generation of figures for scientific publication.



There are no comments yet.


page 1

page 5

page 6

page 7

page 8

page 10

page 11


Volumetric Data Exploration with Machine Learning-Aided Visualization in Neutron Science

Recent advancements in neutron and x-ray sources, instrumentation and da...

Interactive Modelling of Volumetric Musculoskeletal Anatomy

We present a new approach for modelling musculoskeletal anatomy. Unlike ...

A method for automatic forensic facial reconstruction based on dense statistics of soft tissue thickness

In this paper, we present a method for automated estimation of a human f...

BigDataViewer: Interactive Visualization and Image Processing for Terabyte Data Sets

The increasingly popular light sheet microscopy techniques generate very...

A Survey of Spline-based Volumetric Data Modeling Framework and Its Applications

The rapid advances in 3D scanning and acquisition techniques have given ...

Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets

We present a learning method for predicting animation skeletons for inpu...

CT-GAN: Malicious Tampering of 3D Medical Imagery using Deep Learning

In 2018, clinics and hospitals were hit with numerous attacks leading to...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Related Work

Our approach combines techniques from geometry processing, to estimate the initial deformation, with rendering and deformations techniques developed within the visualization community. Here, we give an overview of the most closely related works, and we refer to 

[3, 19] for a complete overview.

Skeletonization via Discrete Maps.

We review here the skeletonization works applicable in our setting, and we refer an interested reader to [45] for a detailed overview.

Our skeleton construction is based on a harmonic volumetric parametrization [47, 26] constructed from a pair of user-provided landmarks. The isosurfaces of the scalar function are averaged to find points in the center of the fish: this construction is inspired by the hexahedral method proposed in [15] and the tubolar parametrization proposed in [29]. It is important to observe that the parametrization induced by these functions is not bijective [37], and it is thus not a proper foliation [4]: however, this is not a problem in our case since we use it only to compute an approximate skeleton. We opted for this skeleton extraction procedure since it allows users to intuitively and interactively control the skeleton, which is mandatory to make our system able to process challenging datasets.

Volumetric Deformation.

This problem has been heavily studied in both in the context of (1) volumetric parametrization, where a energy minimizing, quasi-static solution is found via numerical optimization, (2) in physical simulation, where the focus is on modeling dynamics effects, and (3) in space warping techniques, where a explicit reparametrization of the space is used to warp an object. Since we are not interested in dynamics (we only need to deform the volume once), we only review the parametrization and free form deformation literature, and we refer an interested reader to [49, 30, 34, 41, 18] for an overview of dynamic physical deformation techniques.

Volumetric Parametrization.

A convex approximation of the space of bounded distortion (and thus inversion-free) maps has been proposed in [27, 21], allowing to efficiently generate these maps both in 2 and 3 dimensions. These methods do not require a starting point, but they might fail to find a valid solution in challenging cases. A different approach, guaranteed to work but requiring a valid map has been proposed in [17, 12]: the idea is to evolve the initial map to minimize a desired cost function, while never leaving the valid space of locally injective maps. Many variants of this construction have been proposed, either enriching existing deformation energies with a barrier function [38], or directly minimizing energies that diverge when elements degenerate [42]. Specific numerical methods to minimize these energies have been proposed, including coordinate descent [17, 14], quasi-newton approaches [42, 22, 36, 40, 7], and Newton [38] methods. A last category of methods [13, 35] produces an initial guess separating all triangles and rotating them into the UV space, and then stitches them together using Newton descent. However, all these methods are computationally intensive, and not suitable for interactively deforming high-resolution CT scans.

Space Warping.

Closed-form volumetric deformations have been defined using lattices [39, 1, 8] or other parametrizations [5]. While not directly minimizing for geometric distortion, they have the major advantage of being directly usable in a volumetric rendering pipeline, enabling to render in real-time the deformed volume. Our approach is also directly usable in a real-time volumetric rendering pipeline and uses a keyframed skeletal parametrization to define the deformation.

Volume wires [46] uses a skeleton to define a free form deformation, parametrizing it with values attached to the skeleton itself. Volume wires relies on a computationally intensive evaluation which prevents its use in a real-time rendering systems. Our method shares the idea of using a skeleton to parametrize the deformation, while providing detailed deformation control using keyframes, an algorithm to automatically estimates an initial deformation, and being specifically tailored to be used in an interactive volumetric rendering system.

Interactive Applications.

Many variants of the previous volumetric deformation techniques have been used in interactive applications in the visualization community: since a complete overview is beyond the scope of this work, we limit our review to the most closely related works, and we refer an interested reader to the surveys by Sun et al [44] for visual analytics, and Liu et al. [28] for information visualization techniques.

Closely related to this work, Correa et al. [10] introduced an interactive visual approach to deform images as well as volumetric data based on a set of user-defined control points. Here, the deformation is controlled based on the movement of these control points. Even though their formulation introduced distortions in other regions of the data, since their focus was on illustrative applications and volume exploration and visualization, such distortions were acceptable since they were occluded in the visualization. On the other hand, our goal is to generate data that will be further analyzed by the marine biologists. It is therefore necessary that such distortions are avoided. Such distortions are common in other approaches as well that perform volume deformation with the focus on exploration and/or animations [9, 23].

There have also been visual approaches that target generating illustrations with the focus on medical data [31, 25], in particular, generating views when the covering surface is “cut open”. Since these approaches distort the data that is deformed, they not suitable for our work. Nakao et al. [32, 33] proposed an interactive volume deformation, also catered towards medical applications, which deforms the volume based on a proxy geometry that approximated the volume. However, the proxy geometry itself is computed in a preprocessing phase, which makes the combined pipeline not interactive.

Directly related to CT scans of fishes, in our previous work we proposed TopoAngler [2] that combines a topology based approach with a visual framework to help users segment fishes from the CT data. TopoAngler is used as a preprocessing step in this work, to extract a segmented fish (Section 3)

2 Cylindrical Deformation

Figure 1: An illustration of our volumetric deformation: The density at a point in the straight volume is determined by the cross sectional plane centered at , where is a piecewise linear curve sweeping the spine of the fish and is a frame centered at defining the orientation of the fish at a given position. and are specified at a set of keyframes

(the blue squares) and interpolated linearly in between.

In this section we introduce the underlying mathematical framework of our approach to straighten bent and twisted fishes. Without loss of generality, we assume that fishes are straightened one at a time. If more than one fish is present in the scanned volume, we isolate individual fishes using TopoAngler [2]. The key idea in our approach is to identify a deformation function , which transforms an axis aligned bounding box (which will contain the straight fish), into a deformed version of the fish. This function, being the inverse of the deformation that the fish underwent, will be then used to recover the straight fish.

More concretely, we are interested in a mapping, , which deforms a straight cylindrical region into a deformed one such that the central axis of the is mapped to a curve , which is aligned with the body of the twisted fish. Furthermore, for every coordinate in , we define a rotation matrix which maps points off the central axis to points in . The set of rotation matrices captures both the “bends” that the fish underwent as well as the “twists” (or torsion) in the fish. Figure 1 illustrates one such deformation function .

Thus, to parameterize , we require a space curve and a continuous field of rotation matrices , allowing us to write:


Intuitively, the direction along in points along the skeleton (central axis) of the fish, while the and directions define the right and up directions of the fish respectively. Thus, allows us to undo any bending in the fish while and allow us to remove torsion.

In our setting, we use an arclength parameterized piecewise linear curve for . Note that the arclength parametrization ensures that the mapping will be close to an isometry independently on the speed of the parametrization of . This is important since it allows the user to freely change the parametrization speed by adding additional keyframes, without introducing unwanted distortion (Section 3). Specifically, can be parameterized by vertices . Letting , we can write explicitly as:




To define , we define orthonormal coordinate frames at each vertex of . For to be continuously defined at all points along , we identify the unit normals, , with points on a sphere and spherically interpolate the directions between adjacent :


As we show next, the parameters of the deformation function are initially estimated by our system, and optionally interactively refined by the user in a second stage to compute the final straightened fish.

3 Interactive Fish Straightening

Unwind uses the cylindrical deformation (Equation 1) to assist users in straightening deformed fishes. This system divides the entire process into 5 steps:

  1. Identify the sub-volume of the input corresponding to the fish.

  2. Compute the harmonic function used for estimating the deformation function .

  3. Estimate the parameters of the deformation function .

  4. (Optional) Refine the parameters of the deformation function .

  5. Export the straightened fish.

We now describe in detail each of these steps and the associated visual interface of our system. Each step comprises of its own set of visualization widgets, and the user can move forward and back between the different steps. The entire workflow is illustrated in Figure Unwind: Interactive Fish Straightening and in the accompanying video. Note that the user input is used to guide this process during the different steps.

3.1 Identify Sub-Volume

As mentioned earlier, the input to our system is a 3D volume (structured volumetric grid) containing the scan of a single fish. Due to the long scanning time, the ScanAllFish project participant always scan tens of fishes in the same scanning volume: we use topology-based segmentation tool TopoAngler [2] to extract single exemplars. This approach first generates a hierarchical segmentation based on the join tree of the input data, and then allows the user to interactively choose the simplification to be performed and select the segmented sub-volumes corresponding to the fish. We refer the reader to [2] for a detailed description of this process.

3.2 Compute Harmonic Function

We first approximate the fish by a smooth curve in order to estimate the parameters of the deformation function . This curve is computed as the set of centroids of level sets of an harmonic scalar field defined on the volume, following a technique similar to [15].


Different techniques could be used to compute the harmonic function, and we opted for a finite element method due to its efficiency, simplicity, and robustness. While it is possible to use directly the voxel grid as a space discretization, this would be prohibitively expensive on the high resolution CT scan. Downsampling the image is a possibility, but it would lose the high-frequency details and risk to lead to disconnected components in the thin regions of the fish. We therefore opt for an adaptive tetrahedral mesh, generated using an implementation of Isosurface Stuffing [24], which strikes a good balance between boundary approximation and computational efficiency. While unlikely, it is possible that the generated tetrahedral mesh is made of multiple disconnected components, either due to lack of resolution, or due to the TopoAngler segmentation. To ensure there is only a single connected component, we inflate the voxel grid until all the connected components are merged using [6].

User-Provided Extrema.

Figure 2: The user selects two extrema on the endpoints of the fish by clicking on the extracted tetrahedral mesh

The extrema of the harmonic function, used as boundary conditions, are provide by the user with an end-point selection widget allowing the user to select points on the segmented fish (see Figure 2). These points corresponds to the head and tail of the fish, making it straightforward for the user do this selection.

The two endpoints are then used to compute a discrete harmonic function with Dirichlet boundary conditions setting the head vertex, and tail vertex as a source and sink:


Here, is the discrete Laplace-Beltrami Operator [3], and is a scalar field defined at each vertex of the tetrahedral mesh. While solutions to Equation 6 produce fields whose level sets trace out a reasonable skeleton approximation, the spacing between level sets is not uniform. To remedy this issue, we resample the curve using a fixed spacing between vertices in ambient space. To ensure the resampling process does not discard details, we use a sampling width which is half the size of the smallest segment in the curve traced out by the level sets of the harmonic solution.

Figure 3: Estimated skeletons using the centroids of level sets: The red curve shows the piece-wise linear curve where the vertices are 100 centroids of the level sets uniformly sampled on the scalar function (Equation 6 defined on the tetrahedral mesh. The purple curve is the result of applying 50 smoothing steps the red curve.

3.3 Estimating Parameters and

To generate an initial estimate for the piecewise linear curve, , we sample level sets of at isovalues uniformly spread on . We then compute the centroids, , of these level sets. While the piecewise linear curve whose vertices are the ’s traces a curve approximating the bend of the fish, the curve itself might be noisy due to the complex boundary geometry. We thus apply iterations of Laplacian smoothing (replacing every vertex with the average of its 2 neighbours) to smooth the curve. Figure 3 compares two curves before and after smoothing. Specifically, if and are consecutive vertices of the curve, one iteration of smoothing can be written as:


Once we have vertices , we then compute orthogonal coordinate frames , where . First we compute at each of the using central differences on the interior and one sided differences at the boundary:

Figure 4: Computing a minimal set of parameters by subdivision: The black dotted line illustrates the initial estimated parameters and , using the method described in Section 3.3. We compute the gray prism by connecting two squares on the and planes. We progressively subdivide the prism at vertices until the subdivided prisms fully contains all the . This refinement procedure yields a new, reduced set of parameters and .

We then compute and by projecting the and axes into the plane defined by . If such a projection degenerates, we repeat this procedure with the and axes as well as the and axes (one of them has to succeed since the skeleton is not degenerate by construction):


The result is a piecewise linear curve with vertices and orthonormal bases at each vertex. Note that the number of level set samples, , and smoothing iterations, are user-tunable parameters. The default is and and our users did not change them in any of their experiments.

Computing a Minimal Set of Parameters.

Having a large numbers of parameters can become cumbersome to a user when refining the deformation (see Section 3.4 below). Thus, while we could use the parameters and for the deformation, we opt instead to compute a minimal set of parameters and , , which agree with the ’s and ’s.

To compute these new parameters, we select a radius and construct a prism whose bases are squares with side lengths . Each base is centered at and and is oriented to lie in the planes and with the sides aligned with , and , . Figure 4(a) shows a 2D illustration of the initial configuration for an example curve.

Then, while the prism does not fully contain the vertices, , we subdivide it by first choosing a vertex and frame , and then splitting a prism into two with a base centered at and aligned with . Figure 4(b)–(e) illustrates this subdivision procedure.

The resulting and

are the vertices and coordinate frames of the subdivision location used to construct the prism. The radius hyperparameter,

is user selectable: A larger radius will yield a coarser approximation, and a smaller radius will yield a finer one. By default we set to 10 voxel-widths, and our users did not adjust it in any of their experiments.

Figure 5 shows an example of the initial estimated deformation on a fish scan.

Input Default Skeleton User Adjusted Skeleton

Figure 5: The deformation automatically estimated from the input (top) captures the majority of the distortion (middle) and can be further refined adding additional keyframes (bottom).

Handling Disconnected Components.

Figure 6: In cases where the segmentation outputs disconnected components, we recover a curve approximating the skeleton by selecting endpoints on each connected component. In the figure, the user selects points 1, 2, 3, and 4 and the system estimates a curve between 1 and 2 and another between 3 and 4 which are joined by a line connecting 2 and 3.

There are cases where the components corresponding to a fish are significantly far apart that the dilation operation performed when identifying the sub-volume corresponding to the fish is not sufficient to merge the two components (see Section 4.1 for details). To handle such a scenario, we allow the user to select the end points of the different components in order, compute the harmonic function and estimate the deformation function parameters individually for each component, and use the ordered end points to merge the different curves into a single one (Figure 6).

3.4 Deformation Refinement

We allow the user to interactively edit and refine the deformation parameters. The user interface for this step comprises 3 widgets (Figure 7): a 2D view for editing in a cross section (Figure 7, left), a 3D view showing the effect of the edits in real time (Figure 7, right), and a control widget providing buttons and sliders to help the user perform the deformation (Figure 7, bottom).

Control Widget.

This widget provides the user with a slider (Figure 7, 1) that controls the parameter along the parametric curve . Users can use this to move along this curve to identify locations to edit the deformation. Buttons at either end of the slider allow the user to skip to parameters corresponding to the parameter vertices, . The widget also provides options for the user to add new vertices, and frames, , as well as delete existing vertices and key frames. Alternatively, any interaction in 2D view will automatically add a new vertex and key frame along the curve. Additionally, this widget enables the user to view and change the transfer function of the rendered volume and includes controls to recenter the camera and toggle between the straight and deformed views.

Figure 7: Our user interface for performing interactive deformations: The view is split into 3 parts: Left: the user sees a 2D cross section of the volume. Right: the user sees the straightened fish updated in real time. Bottom: the user sees a set of controls for editing the deformation. We have also labelled several import UI components: 1. A slider to control the parameter along ; 2. Vertices to adjust the size of the prism bases of ; 3. A point allowing the user to change the position of in the plane of ; 4. The axes and projected into the plane .

2D View.

This view shows a cross section of the volume in the plane orthogonal to corresponding to the currently selected parameter in the control widget. Visual cues are overlaid over this cross section to allow user modify the different deformation parameters. These cues include:

  1. A box corresponding to the region of space in the input which will be deformed to generate the output (Figure 7, 2). This represents the the prism base corresponding to the current parameter vertex . The user can adjust the size of the bounding prism, , by dragging the corners of this box.

  2. A point corresponding to the position of in the 2D plane orthogonal to (Figure 7, 3). Dragging this point allows the user to change change the position of vertices on the parametric curve.

  3. The directions in the 2D plane. The user can also rotate the and vectors around by holding shift and dragging (Figure 7, 4). This feature allows the user to align and with the principal directions of the fish, thus allowing for the removal of any torsion which may be present in the twisted input.

3D View.

Depending on the option selected in the control widget, the 3D view visualizes in real time, either the straightened fish or the deformed bounding cage and curve . The former option allows the user to receive real-time feedback on how their edits affect the output, while the latter view allows the user to visualize how well their curve approximates the skeleton of the fish. The straightened volume is obtained by sampling the deformation function after each update to its parameters. This is accomplished by mapping a regular lattice (which is used for the volume rendering) to the input volume which is then sampled using trilinear interpolation. To provide real time feedback at interactive rates, we perform this sampling using the fragment shader as part of the rendering pipeline.

3.5 Real-Time Rendering and Export

Our system, similarly to existing volume deformation pipelines [5], enables real-time rendering of the warped volume, allowing the user to instantaneously see the result of their actions. To render deformations in real time, our system computes a straight volume by evaluating (Equation 1) along cross sections in . We implement this evaluation in an OpenGL shader which renders cross sections along into a volume texture. The texture size is determined by the prisms described in Section 3.3.

Once the user is satisfied with modeled deformation, the straightened volume can be exported onto disk. We use the same procedure for exporting as we do for real time rendering. To preserve the correct dimensions during export, the depth (-direction) of the volume is set to the arclength of the linear curve . The width and height ( and directions) are set to maintain the same aspect ratio as the prisms. The user can optionally edit the size of the exported volume.

In addition to exporting the volume, our application allows the user to save a session to disk and reload it later for further editing or inspection. This feature is important to ensure provenance, enabling to store a direct mapping between the straightened fish and the original RAW volumetric dataset.

4 Results

The results reported here were generated over a period of 4 weeks in the Tytell Laboratory in the Biology Department at Tufts University while processing examplars for the ScanAllFish project. Users used Unwind to straighten fishes on a workstation with a Intel Xeon CPU E5-1607 v4 @ 3.10GHz, 32 GB RAM and a NVIDIA GTX-1080-Ti GPU. In the supplemental material, we include raw screencasts of the editing sessions for all the results presented in the paper. Our reference implementation and sample datasets can be downloaded from our GitHub website https://github.com/fwilliams/unwind: binaries are also provided for Windows and Linux.

Synthetic Evaluation.

Input (L2-error 5.54)

4 Keyframes (L2-error 3.09)

6 Keyframes (L2-error 2.40)

8 Keyframes (L2-error 1.78)

10 Keyframes (L2-error 1.64)

12 Keyframes (L2-error 1.57)

14 Keyframes (L2-error 1.57)

16 Keyframes (L2-error 1.54)

Ground Truth

Figure 8: Straightening a synthetic example of a cylinder bent into the shape of a sine wave. As the user adds new keyframes, the result more closely approximates the ground truth cylinder. We also show the L2 distance between each volume and the ground truth. Each distance is normalized by the length of the ground truth volume.
Phase Average Time %
File Selection 0m19s 5.3%
Data Loading and Contour Tree 1m27s 24.4%
TopoAngler Segmentation 0m29s 8.1%
Mesh Extraction 0m03s 0.1%
Endpoint Selection 0m07s 1.9%
Straightening 3m30s 58.8%
User Interaction 4m27s 74.8%
Processing 1m31s 25.2%
Total 5m57s 100%
Table 1: Average time taken by one expert user took to straighten 15 fishes. The top part of the table shows the time taken in each phase as well as loading times between phases. The bottom part shows how much of the time was spent by the user and now much time the system spent performing computations.

Input Output

Figure 9: This scan was purposefully twisted to evaluate the performance of our system. In spite of the fact that the scan contains large twists as well as multiple fishes, our system produces a straightened volume for a single exemplar.

We performed a synthetic experiments to verify that our system works on analytic deformations. In Figure 8, we show how a cylinder whose medial axis is a sinusoid, can be deformed back to a straight line. We use a different number of keyframes, to show the effect of the refinement on the final deformation. We also compute the distances (normalized by the length of the ground truth volume) between the reconstructed volume and the ground truth. In all user experiments, the deformations are acceptable for the applications in the ScanAllFish project.

Baseline Comparison.

Twisted Scan Straight Scan Ours

Figure 10: Here we scan a fish in both a twisted (top) and straightened (middle) pose. We compare the straightened scan with the scan produced by our method.

The motivation for our project is the avoidance of the practical hurdle of scanning each fish individually in a CT scan, enabling to scan tens of fishes in the same session. However, our post-processing procedure will lead to results of unavoidably lower quality than a direct scanning in the straight pose. To evaluate this error, we performed an experiment: we scanned a fish individually in a straight pose, and then together with other fishes, and then processed it with our pipeline. The two results are indistinguishable (Figure 10) , suggesting that the loss in quality introduced by our algorithm is negligible, and heavily out-weighted by the massive reduction of scanning time.

Timings and Performances.

Our system is interactive, requiring only a few seconds to transition between the different processing phases. The large majority of the time is spent in the segmentation phase, and in the generation of the tetrahedral mesh and solution of the linear system to compute the skeleton. A breakdown of the timings (averaged over all the experiments in the paper) is provided in Table 1. The overall (user + processing) average processing time per fish is less than 6 minutes. In comparison, our collaborators estimate that it takes an experienced biologist 30 minutes to prepare, label, photograph, and pack the tube for scanning, and another 4-12 hours for the actual scan to complete. Our system thus adds a minor overhead to the overall acquisition pipeline.

Showcase of Results.

Figures 9, 11, 12, 13 demonstrate the capabilities of our system under a variety of challenging situations. These cases were selected and straightened by a marine-biologist expert user to evaluate the capabilities of our system for their applications. Further examples are shown in the Appendix (Section A.2).

Input Output

Figure 11: Despite missing parts of the scan, our system can produce a straightened exemplar.

Input Output

Figure 12: This fish was purposefully bent to an extreme position to evaluate the capabilities of our system in the presence of large deformations .

Input Output

Figure 13: Our system can handle a wide variety of fish geometries.

4.1 Expert Feedback

The overall feedback we received from our collaborators was positive with them finding the software extremely useful. They found it intuitive and easy to use. In fact, it took one of our collaborators only “5 to 8 minutes” to explain the working of the tool to a student, who was then able to use it independently. This has now become a staple application for them and has been integrated as part of their daily workflow.

One of the primary advantages they found with straightening fish was when doing morphometrics analysis, especially when trying to look at the fish in a specific anatomical plane. As one of our collaborators mentions: “Before, we had to have versions of the scan rotated at different angles to measure things in a somewhat straight line. Now we can just straighten a single scan which saves a lot of time, storage space, and confusion with different versions of files”. Our collaborator went on add that in addition to the analysis, our tool also significantly helps save time when preparing results for communication as the following quote testifies: “it makes creating figures a lot easier. We don’t have to spend hours looking for the perfect scan, we can just fix any one we have”.

However there were some difficulties that were discovered as the software was being used. Cases were discovered where there was a significant gap between parts of the fish, due to which the dilation performed was not sufficient to create a single connected component. Such cases are possible if the skeleton of a fish is not necessarily connected (i.e., they might be connected via other tissues that have lower density than the bone), of the scanned fish is broken. We then had to then tweak both our interface, as well as the way the central axis of the fish is computed to handle such scenarios (see Section 3.2). There were also some interface requests, in particular, regarding the supported file formats. Currently, our software supports importing data from a stack of .bmp files, which is the output from the CT scanning software, and exports the straightened fish as a raw binary image. Given that users can also have processed data in other formats (tiff stack, DICOM stack, or nrrd files, etc.), having the having the ability to handle these formats will make it easier and faster to work with the tool.

Based on one of the suggestions for next steps, we will be looking into also doing the reverse—bend a fish to specific angles. This can be greatly useful for a variety of studies as well. For example, marine biologists also work with scans of fish with muscle fibers stained. In such cases, they would like to take a straight fish, and then see what the difference in muscle fiber length would look like in a bent fish.

We attach the unmodified transcript of our interview in the additional material (Section A.3).

5 Limitations and Concluding Remarks

We introduced a novel system for supporting the creation of a large encyclopedia of CT scans of fishes, providing a user-assisted procedure to undo the unwanted deformation introduced in the scanning process. We demonstrated the utility of our system, evaluating quantitatively the error introduced in baseline comparisons, and qualitatively over real-world scans. The system is available as open source, and it is in active use in our collaborator’s labs.


Our system has a high GPU memory requirement, which requires a high-end graphics workstation to run. Reducing this requirement is important to foster the applicability of this system and enable it to be used by the community at large. Another limitation of our system is that for exemplars with high distortion, the initial deformation estimation can be inaccurate, requiring more user interaction than for other exemplars (Figure 5). Much of the distortion is a result of the assumption that there is a straight line between the fish’s snout and its tail. Fish skulls are highly complex and extremely diverse across species. One way to fix these large deformations would be to define the skull of the fish prior to the de-warping process. If we constrain slices between the front and back of the skull, it is likely to prevent most of the distortion and improve the efficiency of the workflow.

Future Work.

We believe that, after a several months of usage, we will have access to sufficient data to replace the original deformation estimation with a data-drive model, trained on the data manually processed by the users. We are also porting our application to WebAssembly, to run directly in a web browser and making it easier to deploy in biological labs.

An additional, and surprising, application for this system is strategically bending fish that were scanned straight. Many scientists research fish muscle morphology and how local shape change during locomotion contributes to swimming kinematics. To do this, they stain specimens with iodine before scanning so that the muscle fibers are radio-opaque just like the skeleton. A modified version of Unwind could be developed to warp scanned fish to a shape they might attain during swimming, and look at the change in length of the muscle fibers.


  • [1] A. H. Barr. Global and local deformations of solid primitives. SIGGRAPH Computer Graphics, 18(3):21–30, Jan. 1984. doi: 10 . 1145/964965 . 808573
  • [2] A. Bock, H. Doraiswamy, A. Summers, and C. Silva. Topoangler: Interactive topology-based extraction of fishes. IEEE Transactions on Visualization and Computer Graphics, 24(1):812–821, 2018. doi: 10 . 1109/TVCG . 2017 . 2743980
  • [3] M. Botsch, L. Kobbelt, M. Pauly, P. Alliez, and B. uno Levy. Polygon Mesh Processing. AK Peters, 2010.
  • [4] M. Campen, C. T. Silva, and D. Zorin. Bijective maps from simplicial foliations. ACM Transactions on Graphics, 35(4):74:1–74:15, 2016. doi: 10 . 1145/2897824 . 2925890
  • [5] M. Chen, D. Silver, A. S. Winter, V. Singh, and N. Cornea. Spatial transfer functions: A unified approach to specifying deformation in volume modeling and animation. In Proceedings of the Workshop on Volume Graphics, pp. 35–44, 2003. doi: 10 . 1145/827051 . 827056
  • [6] Z. Chen, D. Panozzo, and J. Dumas. Half-space power diagrams and discrete surface offsets. CoRR, abs/1804.08968, 2018.
  • [7] S. Claici, M. Bessmeltsev, S. Schaefer, and J. Solomon. Isometry‐aware preconditioning for mesh parameterization. Computer Graphics Forum, 36(5):37–47, 2017. doi: 10 . 1111/cgf . 13243
  • [8] S. Coquillart. Extended free-form deformation: A sculpturing tool for 3d geometric modeling. SIGGRAPH Computer Graphics, 24(4):187–196, 1990. doi: 10 . 1145/97880 . 97900
  • [9] C. Correa, D. Silver, and M. Chen. Illustrative deformation for data exploration. IEEE Transactions on Visualization and Computer Graphics, 13(6):1320–1327, 2007. doi: 10 . 1109/TVCG . 2007 . 70565
  • [10] C. D. Correa, D. Silver, and M. Chen. Volume deformation via scattered data interpolation. In Volume Graphics, pp. 91–98, 2007.
  • [11] R. Cross. New 3d scanning campaign will reveal 20,000 animals in stunning detail. doi: doi:10 . 1126/science . aap7604
  • [12] P. Degener, J. Meseth, and R. Klein. An adaptable surface parameterization method. In Proceedings of the 12th International Meshing Roundtable, pp. 201–213, 2003.
  • [13] X.-M. Fu and Y. Liu. Computing inversion-free mappings by simplex assembly. ACM Transactions on Graphics, 35(6):216:1–216:12, 2016. doi: 10 . 1145/2980179 . 2980231
  • [14] X.-M. Fu, Y. Liu, and B. Guo. Computing locally injective mappings by advanced mips. ACM Transactions on Graphics, 34(4):71:1–71:12, 2015.
  • [15] X. Gao, T. Martin, S. Deng, E. Cohen, Z. Deng, and G. Chen. Structured volume decomposition via generalized sweeping. IEEE Transactions on Visualization and Computer Graphics, 22(7):1899–1911, 2016.
  • [16] K. C. Hall, P. J. Hundt, J. D. Swenson, A. P. Summers, and K. D. Crow. The evolution of underwater flight: The redistribution of pectoral fin rays, in manta rays and their relatives (myliobatidae). Journal of Morphology, 270(8):1155–1170, 2018. doi: 10 . 1002/jmor . 20837
  • [17] K. Hormann and G. Greiner. MIPS: An efficient global parametrization method. In Curve and Surface Design, pp. 153–162. 2000.
  • [18] A. Jacobson, Z. Deng, L. Kavan, and J. Lewis. Skinning: Real-time shape deformation. In ACM SIGGRAPH 2014 Courses, 2014.
  • [19] C. Johnson and C. Hansen. Visualization Handbook. 2004.
  • [20] M. A. Kolmann, J. M. Huie, K. Evans, and A. P. Summers. Specialized specialists and the narrow niche fallacy: a tale of scale-feeding fishes. Royal Society Open Science, 2018. doi: 10 . 1098/rsos . 171581
  • [21] S. Z. Kovalsky, N. Aigerman, R. Basri, and Y. Lipman. Large-scale bounded distortion mappings. ACM Trans. Graph., 34(6):191:1–191:10, Oct. 2015. doi: 10 . 1145/2816795 . 2818098
  • [22] S. Z. Kovalsky, M. Galun, and Y. Lipman. Accelerated quadratic proxy for geometric optimization. ACM Transactions on Graphics, 35(4):134:1–134:11, 2016. doi: 10 . 1145/2897824 . 2925920
  • [23] S. Kwon, Y. Kim, K. Kim, and S. Lee. Heterogeneous volume deformation and animation authoring with density-aware moving least squares. Computer Animation and Virtual Worlds, 29(1), 2018.
  • [24] F. Labelle and J. R. Shewchuk. Isosurface stuffing: fast tetrahedral meshes with good dihedral angles. In ACM Transactions on Graphics, vol. 26, p. 57, 2007.
  • [25] W. Li, L. Ritter, M. Agrawala, B. Curless, and D. Salesin. Interactive cutaway illustrations of complex 3d models. ACM Transactions on Graphics, 26(3), July 2007. doi: 10 . 1145/1276377 . 1276416
  • [26] X. Li, X. Guo, H. Wang, Y. He, X. Gu, and H. Qin. Harmonic volumetric mapping for solid modeling applications. In Proceedings of the ACM Symposium on Solid and Physical Modeling, pp. 109–120, 2007. doi: 10 . 1145/1236246 . 1236263
  • [27] Y. Lipman. Bounded distortion mapping spaces for triangular meshes. ACM Transactions on Graphics, 31(4):108:1–108:13, 2012.
  • [28] S. Liu, W. Cui, Y. Wu, and M. Liu. A survey on information visualization: recent advances and challenges. The Visual Computer, 30(12):1373–1393, 2014.
  • [29] M. Livesu, M. Attene, G. Patané, and M. Spagnuolo. Explicit cylindrical maps for general tubular shapes. Computer-Aided Design, 90:27–36, 2017. doi: 10 . 1016/j . cad . 2017 . 05 . 002
  • [30] U. Meier, O. López, C. Monserrat, M. C. Juan, and M. Alcañiz. Real-time deformable models for surgery simulation: A survey. Computer Methods and Programs in Biomedicine, 77(3):183–197, 2005. doi: 10 . 1016/j . cmpb . 2004 . 11 . 002
  • [31] J. Mensmann, T. Ropinski, and K. Hinrichs. Interactive Cutting Operations for Generating Anatomical Illustrations from Volumetric Data Sets. Journal of WSCG, 16(2):89–96, 2008.
  • [32] M. Nakao, K. W. C. Hung, S. Yano, K. Yoshimura, and K. Minato. Adaptive proxy geometry for direct volume manipulation. In 2010 IEEE Pacific Visualization Symposium, pp. 161–168, 2010. doi: 10 . 1109/PACIFICVIS . 2010 . 5429597
  • [33] M. Nakao, Y. Oda, K. Taura, and K. Minato. Direct volume manipulation for visualizing intraoperative liver resection process. Computer Methods and Programs in Biomedicine, 113(3):725–735, 2014. doi: 10 . 1016/j . cmpb . 2013 . 12 . 004
  • [34] A. Nealen, M. Mueller, R. Keiser, E. Boxerman, and M. Carlson. Physically Based Deformable Models in Computer Graphics. Computer Graphics Forum, 2006. doi: 10 . 1111/j . 1467-8659 . 2006 . 01000 . x
  • [35] R. Poranne, M. Tarini, S. Huber, D. Panozzo, and O. Sorkine-Hornung. Autocuts: Simultaneous distortion and cut optimization for uv mapping. ACM Transactions on Graphics, 36(6):215:1–215:11, 2017. doi: 10 . 1145/3130800 . 3130845
  • [36] M. Rabinovich, R. Poranne, D. Panozzo, and O. Sorkine-Hornung. Scalable locally injective mappings. ACM Transactions on Graphics, 36(2):16:1–16:16, 2017. doi: 10 . 1145/2983621
  • [37] T. Schneider and K. Hormann. Smooth bijective maps between arbitrary planar polygons. Computer Aided Geometric Design, 35–36(C):243–354, 2015. Proceedings of GMP.
  • [38] C. Schüller, L. Kavan, D. Panozzo, and O. Sorkine-Hornung. Locally injective mappings. In Proceedings of the Symposium on Geometry Processing, pp. 125–135, 2013. doi: 10 . 1111/cgf . 12179
  • [39] T. W. Sederberg and S. R. Parry. Free-form deformation of solid geometric models. SIGGRAPH Computer Graphics, 20(4):151–160, 1986. doi: 10 . 1145/15886 . 15903
  • [40] A. Shtengel, R. Poranne, O. Sorkine-Hornung, S. Z. Kovalsky, and Y. Lipman. Geometric optimization via composite majorization. ACM Transactions on Graphics, 36(4):38:1–38:11, 2017. doi: 10 . 1145/3072959 . 3073618
  • [41] E. Sifakis and J. Barbic. FEM simulation of 3D deformable solids: A practitioner’s guide to theory, discretization and model reduction. In ACM SIGGRAPH 2012 Courses, pp. 20:1–20:50, 2012.
  • [42] J. Smith and S. Schaefer. Bijective parameterization with free boundaries. ACM Transactions on Graphics, 34(4):70:1–70:9, 2015.
  • [43] M. R. Stocker, S. J. Nesbitt, B. T. Kligman, D. J. Paluh, A. D. Marsh, D. C. Blackburn, and P. W. G. The earliest equatorial record of frogs from the late triassic of arizona. Biology Letters, 2019.
  • [44] G.-D. Sun, Y.-C. Wu, R.-H. Liang, and S.-X. Liu. A survey of visual analytics techniques and applications: State-of-the-art research and future challenges. Journal of Computer Science and Technology, 28(5):852–867, 2013.
  • [45] A. Tagliasacchi, T. Delame, M. Spagnuolo, N. Amenta, and A. Telea. 3D Skeletons: A State-of-the-Art Report. Computer Graphics Forum, 35(2):573–597, 2016. doi: 10 . 1111/cgf . 12865
  • [46] S. J. Walton and M. W. Jones. Volume wires: A framework for empirical non-linear deformation of volumetric datasets. Journal of WSCG, 14(1–3):81–88, 2006.
  • [47] Y. Wang, X. Gu, and S.-T. Yau. Volumetric harmonic map. Communications in Information & Systems, 3(3):191–202, 2003.
  • [48] G. Watkins-Colwell, K. Love, Z. Randall, D. Boyer, J. Winchester, E. Stanley, and D. Blackburn. The walking dead: Status report, data workflow and best practices of the overt thematic collections network. Biodiversity Information Science and Standards, 2, 2018. doi: 10 . 3897/biss . 2 . 26078
  • [49] A. Witkin. Physically based modeling: Principles and practice particle system dynamics. ACM Siggraph Course, 1997.

Appendix A Appendix

a.1 Additional Data

The raw data for all the fish scans in this paper are available on the web at https://drive.google.com/open?id=1GEia4wJZ1Y0_GDvBS1pKIVyi-sgVL7ND.

Screencasts of the expert user straightening each example in the paper are available on the web at https://drive.google.com/open?id=13AaCmPTkbQ2s2xOXiP0eC1Z8clKeSbI4. We also include detailed timing information for user interactions here.

Code as well as binaries for Windows and Linux for Unwind can be found on Github at https://github.com/fwilliams/unwind.

The video attached to this submission can also be found on the web at https://drive.google.com/open?id=1eu0NEb_b2PWyfOOrDQ3B5udhDHMpdg_M.

a.2 Extra Results

Below are more results of fishes straightened using unwind by a marine-biologist expert user.

Input Output

Figure 14: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 15: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 16: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 17: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 18: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 19: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

Input Output

Figure 20: Input and output of our pipeline. The top image shows shows a side view and the bottom shows a top view.

a.3 Transcript of Interview With Expert User

What follows is the transcript of an interview with one of our expert users in the Tytell Lab at Tufts University.

Q: How long did it take you and your student to get familiar with the workflow?
A: We chatted about it and I showed her some stuff for maybe 5-8 minutes. It was pretty quick!

Q: What were any difficulties that you and your student faced to begin with, is that now resolved or do you still have any issues?
A: The only issue she ran into was with saving. Since it just kind of closes, she thought it didn’t save and started to re-do it. I had to explain that closing wasn’t crashing, that’s what it does when saving. For me, the only issue I was having was with missing parts of fish, but I think that’s been solved.

Q: What do you guys like about this tool?
A: I like that it’s easy to use and a lot more intuitive than the segmenting tool (which I also liked but I was one of the only people who could figure out how to use it).

Q: How does having this tool make your lives easy?
A: The tool is really useful for doing morphometrics analysis, especially when we are trying to look at things in a specific anatomical plane. Before, we had to have versions of the scan rotated at different angles to measure things in a somewhat straight line. Now we can just straighten a single scan which saves a lot of time, storage space, and confusion with different versions of files. On a less scientific note, it makes creating figures a lot easier. We don’t have to spend hours looking for the perfect scan, we can just fix any one we have.

Q: How frequently do you plan to use this tool?
A: I would use it every time I go to analyze CT scans of my fish. My species especially are really bendy and flexible, so it’s not hard to accidentally bend or twist parts while preparing a scan. So for me I’ll use it all the time.

Q: Suggestions on what to improve?
A: I think the biggest thing would be the option for different import and export formats. Lots of people end up with tiff stacks, so being able to import them would be awesome. For export, being able to save a DICOM stack or a NRRD files would be useful.

Q: Any other thoughts / feedback etc?
A: I was talking to my adviser and he was saying another thing folks might actually find useful is the ability to actually bend fish to specific angles. For example, my adviser has scans of fish with muscle fibers stained. It might be interesting to take a straight fish, and then see what the difference in muscle fiber length would look like in a bent fish. Just as an interesting ”future directions” thing.