DeepAI
Log In Sign Up

Real-time Skeletonization for Sketch-based Modeling

10/12/2021
by   Jing Ma, et al.
0

Skeleton creation is an important phase in the character animation pipeline. However, handcrafting skeleton takes extensive labor time and domain knowledge. Automatic skeletonization provides a solution. However, most of the current approaches are far from real-time and lack the flexibility to control the skeleton complexity. In this paper, we present an efficient skeletonization method, which can be seamlessly integrated into the sketch-based modeling process in real-time. The method contains three steps: local sub-skeleton extraction; sub-skeleton connection; and global skeleton refinement. Firstly, the local skeleton is extracted from the processed polygon stroke and forms a subpart along with the sub-mesh. Then, local sub-skeletons are connected according to the intersecting relationships and the modeling sequence of subparts. Lastly, a global refinement method is proposed to give users coarse-to-fine control on the connected skeleton. We demonstrate the effectiveness of our method on a variety of examples created by both novices and professionals.

READ FULL TEXT VIEW PDF

page 8

page 9

page 10

05/16/2017

Automated Body Structure Extraction from Arbitrary 3D Mesh

This paper presents an automated method for 3D character skeleton extrac...
09/06/2019

Real-time Deformation with Coupled Cages and Skeletons

Skeleton-based and cage-based deformation techniques represent the two m...
08/17/2016

Large Angle based Skeleton Extraction for 3D Animation

In this paper, we present a solution for arbitrary 3D character deformat...
12/23/2020

Skeleton-based Approaches based on Machine Vision: A Survey

Recently, skeleton-based approaches have achieved rapid progress on the ...
03/15/2018

Computing the Planar β-skeleton Depth

For β≥ 1, the β-skeleton depth (_β) of a query point q∈R^d with respect ...
06/23/2022

Hierarchical Planning with Annotated Skeleton Guidance

We present a hierarchical skeleton-guided motion planning algorithm to g...
04/11/2021

Velocity Skinning for Real-time Stylized Skeletal Animation

Secondary animation effects are essential for liveliness. We propose a s...

1 Introduction

Figure 1: An illustration of creating animatable skeleton using our system: (a) Draw a sketch, a polygon contour is approximated, and a skeleton is extracted from the simplified polygon, meanwhile, the 3D mesh is created through inflation. (b) Draw/move a part, the hierarchical relationship is deducted by fast shape intersection test and modeling sequence, and the precise connecting position from child part to parent part is calculated. (c) Refine the skeleton by multi-level control operations.

There is an increasing need for ready-to-animate models, which takes designers lots of time in shape modeling, skeleton creation, and weight painting. However, manually creating the skeleton is labor-intensive, and requires professional training. Sketch-based tools teddy_igarashi2006 liberate users from the troublesome of shape modeling, making 3D modeling accessible to novices. There are also lots of work focusing on automatic skeletonization DistanceTransform_2d3dmesh_contraction_au2008rosamean_curvature_tagliasacchi2012QMat_li2015rignet_xu2020. However, they are computation expensive and far from real-time. The skeleton is extracted only after the entire mesh model is constructed and processed by the algorithm.

In this paper, we present a real-time skeletonization method for sketch-based modeling, which enables simultaneous shape modeling and skeleton creation. A local skeleton and corresponding mesh are automatically created as soon as the polygon stroke is captured, then the subpart is connected to the existing subpart, finally a coarse-to-fine refinement strategy is provided to the user to control the skeleton complexity, see Fig. 1. Extracting an animatable skeleton in real-time is difficult with three major challenges: how to extract the local skeleton efficiently from an underlying shape; how to establish the relationship among local skeletons in real-time; and how to find the optimal skeleton structure suitable for animation.

To solve the first challenge, we construct a straight skeleton straightskel_aichholzer1996 structure directly from the input polyline and fully utilize Douglas Peucker (DP) simplification douglas1973algorithms to accelerate the process. The straight skeleton is extracted by offsetting the polygon edges inward and recording the trace of vertices. We employ straight skeleton as our base skeletonization technique because the computation cost is almost negligible compared with other methods based on binary image Zhang_thinning, voxels lee_3d_thinningDistanceTransform_2d3dAnimSkelVolNet, point clouds rosa, and meshes mesh_contraction_au2008mean_curvature_tagliasacchi2012rignet_xu2020. The simplicity of polygon data structure makes the algorithm a perfect candidate for immediate skeleton extraction. Our innovation is the full-fledged use of DP technique for speeding up animatable skeleton extraction. As shown in Fig. 1a, we simplify the dense sketch points (red) with DP, the simplified contour not only approximates the shape well but also generates the most concise input for the straight skeleton extraction algorithm. We further propose an efficient BoundedDP algorithm for extracting joints from the axis line of the straight skeleton. Both the axis curvature and silhouette shape are considered, which yields a good result of key joints.

To solve the second challenge, we reuse the skeleton axis and joints extracted from the first step and develop a general cylinder around the axis as well as an inscribed ball around the joint to approximate the 3D shape. This approximation enables the real-time intersection test in the interactive modeling process. Once two 3D shapes are found intersecting with each other, we attach one skeleton to the other according to the modeling sequence. We set the independent subparts as roots, and all follow-up subparts are attached hierarchically according to the intersection relationship and the modeling order. The attached position is calculated from Euclidean distance between parent-skeleton bones and child-skeleton joints.

To solve the third challenge, we equip users with four operations under three level-of-detail controls. As pointed by rignet_xu2020, animators need multi-level controls for skeleton complexity, and the fixed number of joints extracted solely from shape topology is not enough to capture user intention. For example, a hand may be represented by a single medial bone or a finer resolution of hierarchical finger joints. Our four operations provide users flexible control over skeleton complexity. Instead of manually inserting, deleting, and connecting bones, users only need to adjust four parameters to change the skeleton structure. The branch-level operation is based on DP algorithm, which allows users to tune the joints number for a single axis, such as the axis on a finger. For the subpart level and global level, we take inspiration from polygon mesh processing QEMremeshingPMP and skeleton pruning skeleton_pruning_2012skeleton_pruning_2020 since a skeleton can be regarded as an acyclic graph structure (tree). The three operations are joints merging, branch pruning, and edge collapsing. Each operation corresponds to a control threshold that is calculated from the current geometric state of the skeleton. Users can explore different design ideas simply by playing around with these parameters.

In summary, our contributions are four folds: a real-time skeletonization algorithm for immediately constructing an animatable skeleton from user sketch; an efficient method for fast intersection test and sub-skeletons connection; a flexible solution for multi-level of details skeleton control; an easy-to-use sketch system to create animatable models.

2 Related Work

2.1 Skeletonization

There are plenty of works for skeletonization. Based on input data types, they can be classified into: binary image and planar curve in 2d; voxels, point cloud, and mesh in 3d

report3d_skeleton2016. According to the skeletonization technique, they can be classified into propagation-based, and geometric-property-based. In the following, we shall discuss these approaches according to the latter categorization.

Propagation-based methods

The propagation-based methods mimid the grassfire or wavefront transformation moving along the boundary, and the skeleton is the quench where two or more fires/wavefronts meet. In the 2D binary domain, the most representative work is Zhang-Suen Thinning Zhang_thinning, which iteratively removes pixels along object borders until no more pixels could be removed. A large body of follow-up works extend this method to 3D voxel domain lee_3d_thinning3d_thinning_1998hybrid3d_thinning_19983d_thinning_1999_directional3d_thinning_19993d_thinning_20013d_thinning_2002. These methods are not geometric robust and have the risk of removing important features. Despite the additional processing method 3d_thinning_curve_skel_2008 is proposed to handle excessive removal, the binary propagation approach is generally computation intensive. The computation cost is greatly reduced in the 2D geometric domain, namely, the shape represented by 2D lines or curves. Aichholzer introduces a straight skeleton structure straightskel_aichholzer1996 for the polygon. The structure is progressively constructed by shrinking the polygon and recording the trace of moving vertices. Felkel presents a practical implementation for this algorithm using a doubly-linked list and a priority queue, and the time complexity only subjects to the number of polygon vertices. The 3D counterpart of this algorithm is proposed by mesh_contraction_au2008. Similar to shrinking the polygon into zero-area, Au’s algorithm guides the triangular mesh shrinking to zero volume by constrained implicit Laplacian smoothing implicit_smooth_desbrun1999 and defines an edge cost to simplify the collapsed mesh into the curve skeleton. Tagliasacchi et al. mean_curvature_tagliasacchi2012 further improve Au’s algorithm through a detailed analysis of Mean Curvature Flow (MCF) during mesh contraction, and introduce a local remeshing schema to enhance numeric stability. The mesh-based methods are time-consuming incurred by the implicit linear system solving in each contraction iteration, and the generated curve skeleton needs post-processings to be animatable. Compared with a curve skeleton, the straight skeleton structure straightskel_aichholzer1996 conforms more to an animatable skeleton. Our skeletonization algorithm benefits from such conformance and the simpleness of the planar polygon. Meanwhile, it does not need an expensive mesh contraction process to create the 3D skeleton.

Geometric property-based methods

The geometric methods strive to find the shape centers by analyzing the translation and rotation property of the inner region towards the boundary. One popular choice is Medial Axis Transformation (MAT). The MAT skeleton is defined as the locus of the centers of all maximally inscribed circles (in the 2D domain) or spheres (in the 3D domain). Plenty of MAT literature exist for different geometric entities: the 2D/3D binary ones zhou1999efficientbitter2001penalizedmat2003skeletonhassouna2005robustDistanceTransform_2d3d, the planar curve MAT_spline, the point clouds MAT_point_clouds, and the mesh QMat_li2015. These methods are computationally intensive, numerically unstable excepting the work for planar curve MAT_spline, and are sensitive to boundary perturbations. In addition to the distance transformation property, rotation traits are used in rosa. The curve skeleton is discovered by finding a rotational symmetry axis (ROSA) for an oriented point cloud. They propose an iterative planar cuts method to find the optimal ROSA plane. This method is extended by general_cylinder_zhou2015 to the mesh. They use the ROSA method to find a general cylinder decomposition and partial curve skeleton given a polygon mesh. However, these methods are also time-consuming, and the extracted curve skeleton is not ready for animation.

2.2 Automatic Rigging

Automatic rigging creates a ready-to-animate bone skeleton mesh_contraction_au2008 and binds the skeleton to the mesh. The pioneering work of automatic rigging is proposed by automatic_rigging_baran2007, which fits a predefined skeleton template to the mesh, and calculates the skinning weight through heat diffusion. However, their method is limited by the skeleton template and fails to cope with various shapes. On the contrary, our work is highly adaptable to various shapes in a creating-on-the-go fashion. It is closely related to RigMesh proposed by rigmesh_borosan2012

, in which shape modeling and skeleton creating are handled simultaneously. RigMesh uses Constrained Delaunay Triangulation to decompose the silhouette into needle-like triangles and constructs the skeleton by connecting triangle centers. It defines three strategies for skeleton connection: splitting, snapping, and connecting. When a user drags one part close to the other, a connecting suggestion from these three cases is made. The underlying meshes are merged immediately after the user accepts the recommendation. This strategy is unintuitive and inaccurate, leading to the attached mesh deviating from its original position. In contrast, our algorithm can calculate the exact attach position conforming to the user’s original intention. With the rapid development of deep learning techniques, the recent work uses a deep neural network to solve the automatic rigging problem

rignet_xu2020, however, their approach is limited to the training set and does not work well for arbitrary shapes, and the method also has a high requirement for devices. Our algorithm works for the novel shapes that a network is never trained before, and it is also practical to be deployed in low-end devices.

2.3 Sketch Modeling

Sketch-based system pioneered by Igarashi et al. teddy_igarashi2006 greatly reduces the workload of shape modeling. Users only need to draw few strokes to create a 3D shape. And the subsequent works tai2004prototypecherlin2005sketchnealen2005sketchschmidt2007shapeshopalexe2007shapenealen2007fibermeshsugihara2008sketchcordier2011sketching make sketch modeling more complete, with the power of creating complex shapes. The recent work of MonsterMash monster_mash_dvorovzvnak2020 propose a framework for casual sketch modeling and animation prototyping under single-view, the work provides lots of fun to novice users, however, the strength of simple interaction is also a weakness of the system, especially for professional users who want an accurate model. Our system allows users to create accurate shapes under different views without losing the interaction simplicity.

3 Our Method

3.1 Local Sub-Skeleton Extraction

Figure 2:

Local sub-skeleton extraction. (a) Processed sketch line. (b) Simplified polygon. (c) Straight skeleton traced by propagating polygon edges inward. (d) Animatable clean skeleton by removing redudant vertices and edges. (e) Spline interpolated by long polyline branch. (f) Slice uniformly along the spline to capture contour variation. (e) Simplified skeleton by bounded DP simplification.

Previous works extract skeleton from the merged mesh mesh_contraction_au2008mean_curvature_tagliasacchi2012, which is slow in interactive scenarios. To achieve real-time performance, we generate the local sub-skeleton directly from stroke-input. The local method contains three sub-steps: simple polygon acquisition; straight-skeleton extraction; and straight-skeleton simplification, shown in Fig. 2. In this section, we first introduce the definition of simple polygon and straight skeleton. Then, we describe the details of three sub-steps.

Figure 3: Simple polygon
Definition 1 (Simple Polygon)

A simple polygon is a polygon that does not intersect itself and has no holes. See Fig. 3 for an illustration.

Definition 2 (Straight Skeleton)

A straight skeleton is a special partitioning of a polygon into monotone regions traced by a continuous inward offsetting of the contour edges straightskel_aichholzer1996. See Fig. 4 for an illustration.

Simple polygon acquisition

To acquire valid and concise input for the local skeletonization algorithm, the raw sketch line needs to be processed. The sketch line is usually smooth but sometimes may contain noises, shown in Fig. 19. We smooth out these noises by uniformly discretizing the raw line by a small length, which yields a smooth input for our algorithm, shown in Fig.  2a. Then we apply Douglas-Peucker (DP) algorithm douglas1973algorithms to find a simplified polygon best approximating the shape, shown in Fig. 2b, details of DP are introduced in Section Straight-skeleton simplification. Lastly, we arrange the polygon vertices as counter-clockwise oriented to be ready for the next step. We denote the simple polygon as consisting a set of vertices and edges .

Straight skeleton extraction

Then a straight skeleton is extracted from the simple polygon . The basic idea is to propagate edges inward like setting fire around the border, and the straight skeleton is traced by bisectors of polygon edges. The polygons here refer to the initial silhouette polygon and intermediate offsetting polygons generated by propagation, see Fig. 4. In a formal description, all edges move at the same speed along their respective perpendicular direction. During propagation, two types of events may changes polygon’s topology: the edges and (adjacent to the edge ) collide, and vanishes; an edge collides with two consecutive edges and , splitting into two edges on collision position. We define the former as Edge Event and the latter as Split Event following the same convention from sske_impl_felkel1998 and sskel_impl_cgal_cacciola2004, see Fig. 4. The method is summarized in Algorithm 3.

Figure 4: Black: the initial silhouette polygon. Green: the intermediate offsetting polygons. Red: the traced straight skeleton. (a) Edge Event: non-consecutive edges and collide, and collapses. (b) Split Event: consecutive edges and collide simultaneously with opposite edge , and splits. The polygon at the time splits into two smaller polygons and at the next time .

Straight-skeleton simplification

Figure 5: Over simplification

The straight skeleton from the previous step contains many unnecessary vertices and edges as shown in Fig. 2c and Fig. 6. Skeleton vertices traced from high curvature borderline are clustering together. Neither the peripheral edges nor the very short skeleton edges are wanted for constructing an animatable skeleton. Thus, we remove all peripheral edges, and collapse short skeleton edges less than a given threshold, shown in Fig. 2d. In our experiment, we set the collapse threshold as , which produces a clean animatable skeleton in most cases. Still, this skeleton is not concise enough. Designers may find that nodes along the long axis are superfluous. The Douglas Peucker algorithm douglas1973algorithms is a good option to do the simplification. However, only considering the axis curve while ignoring the shape will lead to an over-simplified or even out-of-shape skeleton, see Fig. 5. To address this problem, we propose a BoundedDP algorithm considering both the axis and the shape. Details of this algorithm will be introduced in the following.

Figure 6: Straight skeleton. A straight skeleton contains two types of vertices: border vertices (red circle) and skeleton vertices (blue cirlce), and three types of edges: border edges (black segment), peripheral edges (green segment), and skeleton edges (red segment).

A spline curve is interpolated for each long polyline, shown in Fig. 2e. We observe that the region enclosing a long branch usually forms a general cylinder, which could be used as boundary restriction during simplification. To extract the general cylinder, we first generate a set of uniformly sampled points along the axis curve, and compute a segment at each by intersecting the polygon with an infinite line perpendicular to the central axis at , see Fig. 2f. Meanwhile, the intersected edges on the polygon are also recorded, forming two nearly parallel lines on the left and the right side of the central axis. By closing open holes at the end of these two lines, we get the general cylinder region , see shadow area in Fig. 2f. Then and are used as input for BoundedDP to get the simplified polyline with each polyline edge lying inside the bounded region .

Figure 7: BoundedDP simplification (red branch in Fig. 2f). The upper part of the first column shows an axis curve and its enclosing general cylinder. The lower part of the first column shows a parallel arrangement of intersected segments, which are acquired by uniformly slicing along the axis. The next three columns show the first three steps of BoundedDP. Better approximations are found for both the line and the shape as the algorithm progresses.

The original Douglas Peucker algorithm works in a recursive greedy manner: at each step, it selects a point with maximum distance to the line . If the distance is larger than a threshold , is regarded as important for fitting the original line, and is added to the output, see the first row of Fig. 7. The algorithm then recursively calls itself on the split line, and , finding the furthest point in each range. The recursion terminates until no point can be found with its distance larger than the threshold . Two essential ingredients contribute to a good approximation: (i) the criteria to select the important point; (ii) the threshold to balance between simplicity and approximation quality. For the criteria, instead of only considering the variation of axis curve, our algorithm also takes shape variation into account, which leads us to the following formula: rCl E_i & = & E_p_i + α_s E_s_i
& = & d_i^⟂ + α_s (d_i^+ + d_i^-) where measures the axis variation, namely, the perpendicular distance from current point to the straight line connecting two endpoints, see the first row of Fig. 7. To capture the shape variation, we add the second term to the formula, measuring the distances and from two endpoints of segment to the two sides of the trapezoid, which is bounded by two end segments and , see the dotted line (distances) and the shaded area (trapezoid) of the second row in Fig. 7. Notice we rearrange the segments parallelly by stretching the axis curve to a horizontal straight line, filtering the axis variation out of the second term. In each step, we choose the point maximizing the error , which is a summation of axis variation and a weighted () shape variation. The point selection procedure is summarized in Algorithm  1. For threshold, the larger the value is, the more simplified result we get, sometimes inducing undesirable skeleton, see Fig. 5. To select a proper , we initialize the threshold with a fairly large value , and try to find an approximate polyline . Then we check whether all edges of the simplified polyline are inside the shape . If there exist intersections between and the shape boundary , we tune down the current threshold by a factor of . This procedure is summarized in Algorithm 2.

Input: , , start index , end index , threshold
Output: Array of booleans marking which to be retained.
1 if  then return;
// Find point with max error
2 ;
3 ;
4 for  to  do
5       PerpendicularDistance(, Line(, ));
       ;
        // shape variation, see Fig.7
6       ;
7       if  then
8             ;
9            
10       end if
11      
12 end for
13;
// Recursive selection
14 if  then
15       BoundedDP(, , , , );
16       BoundedDP(, , , , );
17      
18 end if
ALGORITHM 1 BoundedDP: point selection.
Input: points and perpendicular segments uniformally sampled along axis curve, bounded region , a relatively large initial threshold , adaptive ratio
Output: Simplified polyline
;
  // initial threshold
1 while not IsInsideShape(, ) do
2       ;
3       BoundedDP(, , , , );
4       clear previous , add to if is ;
       ;
        // reduce threshold
5      
6 end while
ALGORITHM 2 BoundedDP: threshold tuning.

3.2 Sub-Skeleton Connection

Before starting this section, we give the following definition:

Definition 3 (Junction joint)

Joint adjacent to three or or more bones.

Definition 4 (Sleeve joint)

Joint adjacent to two bones.

Definition 5 (Terminal joint)

Joint only adjacent to one bone.

Definition 6 (Branch)

A branch begins with either a junction joint or terminal joint, and ends with a junction joint or terminal joint, probably with sleeve joints between two ends.

We connect sub-skeletons in an interactive pairwise fashion. This conforms to the real scenarios in which users interactively create 3D models part by part. The new subpart is connected to the existing subpart, meanwhile, the new sub-skeleton is connected to the corresponding sub-skeleton. When the user creates a new subpart or moves an existing subpart, see Fig. 1b, we instantly check whether the current subpart intersects with other subparts. If the current subpart intersects with another subpart, we connect the skeleton in the current subpart to the skeleton in the intersected subpart. To make intersection test real-time, we reuse the local skeleton extracted from the previous step. Specifically, we find a cylinder-ball approximation for the mesh shape, see Fig. 9. Each skeleton edge serves as the medial axis of the general cylinder, and each bone serves as the center of the inscribed ball. The ball’s radius is the average distance from the joint to the cross-section contour, which is precomputed by slicing the mesh along the bone’s perpendicular direction. Once the radius of the two end joints is computed, the radius for the general cylinder can be calculated by linear interpolation of the two end radius. We denote sub-skeleton on the current subpart as child-skeleton and sub-skeleton on the previous intersected subpart as parent-skeleton. Initially, we set the first subpart as the root. If a subpart does not intersect with any previous subpart, we also add it to the roots set. All subparts created after and intersected with a root becomes the root’s child. All subparts created after the child and intersected with the child become the child’s child (one of the root’s descendants), so on and so forth, formulating a tree hierarchy. If the terminal joints or junction joints of the child skeleton lies inside the cylinder-ball region of the parent, we regard that the two parts intersect with each other and start locating a more precise position from the child-skeleton to the parent-skeleton for attaching.

Figure 8: Distance from child joint (yellow) to parent bone (purple).

On child-skeleton, terminal joints and junction joints are selected as the candidates to be connected to the parent bones.We then tries to find an optimal position on the parent-skeleton where the child’s candidate joints could be connected to, the one minimizing the distance between the child joint and the parent-skeleton. Firstly, we search for the best bone/joint from the parent skeleton. We compute the shortest distance from the child candidate joint to each parent bone. The bone with minimum distance is selected as the best spot for connection, see the purple segment in Fig. 9. Given a parent bone with start joint and end joint , the shortest distance from parent’s bone to child’s joint is calculated as:

(1)

where defines an influence region for each bone, see shadowed area in Fig. 8 and Fig.  9. The influence region is bounded by two planes perpendicular to the bone. Only when the child joint lies inside the two planes, the perpendicular point-to-line distance is used Fig. 8a. If child joint is outside the bone’s influence region, we calculate the Euclidean Distance from the joint to the bone’s two end joints respectively, and choose the smaller one as the shortest distance Fig. 8b.

Figure 9: Sub-skeleton connection: connecting child-leg-skeleton to parent-torso-skeleton. The shadowed area shows the cylinder-ball approximation of the shape, which enables real-time intersection test. The dotted line shows the shortest distance from the candidate child joint to each parent bone. (a) Bone Split Event. (b) Joint Connect Event.

Then, we connect child-skeleton to parent-skeleton by choosing the closest joint-joint or joint-bone pair. Two types of events occur: bone split event and joint connect event, see Fig. 9. Bone split event occurs when the child‘s joint lies inside the parent bone’s influence region. We create a new joint on the parent bone, green circle in Fig.  9a, and split the bone into two. Then a new bone is created between the new joint and the child joint. Notice if the child’s joint is close to the parent bone’s end, one of the split bones will be very short. We address this problem in Sec3.3. Joint connect event occurs when the child’s joint lies outside the parent bone’s influence region. We create a new bone connecting the child joint and the nearest parent joint.

3.3 Global Skeleton Refinement

Figure 10: Refine a bird skeleton. (a) The initial skeleton under default parameters, constructed by our system automatically. Green boxes (a)-(b): simplify branches on wings by increasing threshold . Red boxes (a)-(b): acquire more complex structure on claw by reducing threshold . Yellow boxes (a)-(b): trim short JT branch by increasing threshold . Black boxes (b)-(c): collapse short internal bones by increasing threshold .

.

Figure 11: Merging. (a) Before merging. (b)After merging.

Our global refinement method aims to generate a clean skeleton suitable for animation. Specifically, it solves two problems. Firstly, sub-skeleton connection usually induces redundant short bones, especially when two or more child-skeletons connect to similar positions on the parent-skeleton. Secondly, users need more control for skeleton complexity on different parts of the model, for example, users may expect a more complex skeleton for the hand than for the torso. To equip users with interaction easiness and flexibility. We introduce four operations and a multi-level strategy. The four operations are curve simplification, joints merging, branch pruning, and edge collapsing. The multi-level complexity control contains three tiers: branch level, subpart level, and global level. Simplifying works on branch level, whereas merging, pruning and collapsing work on subpart level and global level.

Branch level operation

We control branch complexity using BoundedDP of Section 3.1. Specifically, we map the subpart-shape and the branch from 3D space into 2D space by rotating the 3D plane (Fig. 1b, Fig. 12 1st row) back to XY plane, run the BoundedDP algorithm in 2D space, and then remap the simplified branch back to its original 3D position. The distance threshold for DP simplification is , and the default value in our experiment is . Fig. 10 (green boxes) shows simplifying the wing skeleton by increasing .

Subpart and global level operation

Joints Merging: We observe that junction joints usually cluster together after the user connects the child sub-skeleton to the parent sub-skeleton. The distance between any two junction nodes inside the cluster is small, thus it is reasonable to merge these junction nodes into one, see Fig. 11. We perform a breadth-first search to find junction clusters constrained by a distance threshold, it is similar to the algorithm that finds a connected component of graph bfs_con_comp. The distance threshold for merging is , and we set its default value as . Branch Pruning: When a JT-branch (the branch ending with a junction joint and a terminal joint) is shorter than a certain length, we remove it since it is insignificant to the overall structure. The distance threshold for trimming a JT-branch is , and we set the default value as . Yellow boxes of Fig. 10 show such an example. Edge Collapsing: Junction joints and sleeve joints are internal joints. When a bone connecting two internal joints is short, it brings redundancy for an animatable skeleton, thus we collapse the two joints into one. The distance threshold for collapsing is , and we set the default value as . Black boxes of Fig. 10(b)-(c) show acquiring a more concise skeleton by increasing collapsing threshold .

Multi-level complexity control

We provide the user the option to control the skeleton structure by adjusting the above four parameters , , , and . Since a smaller threshold yields a more complex structure, the skeleton topology changes in a coarse-to-fine manner when users adjust the threshold. For example, a more complex hand skeleton can be acquired by setting a smaller value of , more knuckles appear on the finger branch, shown in red boxes of Fig. 10a-b. If the user needs the coarse representation, they can adjust the threshold to a higher value. Besides, users can select a single branch, a subpart-skeleton, or the global skeleton, and apply these operations on them. They can explore different skeleton structures simply by playing with these parameters.

4 Results

Figure 12: A gallery of models created by professionals using our system (zoom out the PDF to see the details). The first row shows the initial sketches from input image (right corner) and the silhouette polygons (drawn in 2D, then transformed to 3D). The second row shows the refined global skeleton along with the inflated 3D mesh. The third row shows the skinning weights (right corner) and the deformed models.

We implement our approach using C++, libIGL222https://libigl.github.io/, and Qt. The current implementation, available at https://github.com/jingma-git/RealSkel, is tested on a AMD Ryzen 7 3700X 8-Core Processor 3.6GHz with 62GB RAM under Ubuntu20.04 LTS system. Our program runs at an interactive rate (see the accompanying video). Fig. 12 shows a gallery of created models using our system.

4.1 Implementation Details

We use CGAL’s implementation to extract the straight skeleton sskel_impl_cgal_cacciola2004, which runs in time, where denotes the number of polygon vertices and denotes the reflex ones. For mesh creation, we first triangulate the polygon to get a 2D mesh triangle_shewchuk1996, and generate the 3D mesh by an inflation algorithm monster_mash_dvorovzvnak2020. The height field of internal vertices is calculated by Poisson equation subject to the Dirichlet boundary condition and a constant height parameter . We provide the user the option to change the model’s thickness by adjusting the height parameter . The sparse linear system is solved by LLT Cholesky factorization, which takes time, where denotes the number of vertices in the 2D mesh. We also provide the user some utility functions such as sketch symmetry line, create symmetry part, rotation, translation, and scaling, etc. After all subparts and the global skeleton are constructed, we merge all parts into a watertight mesh based on the method from mesh_arrange_zhou2016. To make our system fully automatic, we use the method from BBW to compute the skinning weights for deformation, see Fig. 12 3rd row.

4.2 Comparisons

Models Modeling Time(min) Model Accuracy(1-5 stars)
MonsterMash RigMesh Ours. MonsterMash RigMesh Ours.
Nov Pro Nov Pro Nov Pro Nov Pro Nov Pro Nov Pro
bird 10 6 40 32 18 14 2.5 3.0 3.5 3.5 4.5 4.5
cat 7 4 29 23 12 10 3.0 3.0 3.5 3.5 4.5 5.0
crab 5.5 5 33 28 17 15 2.0 2.0 3.0 3.0 4.5 4.5
deer 5 3.5 42 39 16 13 2.5 3.0 3.0 3.5 4.5 4.5
tree 8 6 28 21 18 12 3.0 3.5 3.5 3.5 4.0 4.5
panda 5 3 30 27 16 11 1.5 1.5 2.5 3.5 4.5 4.5
lorax 3 2.5 21 19 10 8 3.5 3.5 3.0 3.5 4.5 5.0
octopus 6.5 5 30 22 19 13 1.5 2.0 3.0 3.0 4.5 4.5
pokemon 4 2 32 24 14 10 2.0 2.0 3.0 3.5 4.5 4.5
Average 6 4.1 31.7 26.1 15.6 11.8 2.4 2.6 3.1 3.4 4.4 4.6
Table 1: Compare our system with MonsterMash and RigMesh. Nov: novices. Pro: professionals. Our system is easy for users to create high-quality models. The easiness is revealed by Modeling Time and the quality is revealed by Model Accuracy. The shorter the time is, the easier for the user to interact with the system. The more stars are given, the more accurate the model is. The modeling time of our system is half of the RigMesh, while the model accuracy of our system is higher than both MonsterMash and RigMesh.
Figure 13: Models created by novices using our system. Models take an average 15 minutes to create.

We compare our system with MonsterMash333https://monstermash.zone/ and RigMesh444https://cragl.cs.gmu.edu/rigmesh/ to evaluate the interactive part of our system. Twelve users participate in the user study, three with modeling experience are assigned to the professional group, six without any modeling experience are assigned to the novice group, and the rest three are assigned to the jury group to judge the model accuracy (the quality of the mesh and the skeleton). We ask the first two groups to use MonserMash, RigMesh, and our system to create 3D models, and then show the models to the third group and ask them to fill a questionnaire regarding model accuracy. During the experiment, we give the participant a user manual, and record the time they take to be familiar with the system. On average, it takes eight minutes, twenty minutes, and twelve minutes to train the participants on MonsterMash, RigMesh, and our system respectively. After the user finishes training, they are allowed to proceed to the modeling stage. The modeling time is shown in columns 2-7 of Table. 1. Models created by professionals and novices using our system are shown in Fig. 12 and Fig. 13 respectively. Models created by professionals using MonsterMash are shown in Fig. 14. Models created by professionals using RigMesh are shown in Fig. 15. The model accuracy given by the jury group is shown in columns 8-13 of Table. 1 (MonsterMash models are accessed by mesh quality, RigMesh and our system are accessed by both mesh quality and skeleton quality). Compared with the other two systems, our system takes less time for users to create accurate models.

Figure 14: Models created using MonsterMash monster_mash_dvorovzvnak2020. Models take an average 4 minutes to create. No skeleton is automatically generated. It only supports single-view modeling, which accelerates the modeling time but reduces the shape accuracy, see bird wings, cat ears, and crab body.
Figure 15: Models created using RigMesh rigmesh_borosan2012. Models take an average 26 minutes to create. The skeleton joints are redundant and inaccurate, see cat butt, panda torso, and lorax shoulder.
Models #vertices 3D Skeletonization 2D Skeletonization
MeanCurvature GeneralCylinder Zhang-Suen ChordalAxisTransform Ours.
bird 10808 582 22052 331 6 13
cat 6980 598 18088 167 4 4
crab 19079 860 26382 412 6 10
deer 10091 745 39587 217 4 10
tree 11920 681 43986 141 4 7
panda 5939 406 14057 254 6 5
lorax 8750 719 18135 212 6 9
octopus 17004 1052 21941 560 5 14
pokemon 10487 672 13195 345 6 7
Average * 701.8 24158.7 293.2 5.2 8.8
Table 2: Execution time() of our algorithm compared with other algorithms. represents the number of vertices for merged mesh of Fig. 12. Columns 3-4 show the execution time of MeanCurvature and GeneralCylinder respectively. Columns 4-6 show the execution time of Zhang-Suen’s thinning Zhang_thinning, RigMesh ChordalAxisTransform rigmesh_borosan2012, and our algorithm. In general, 2D skeletonization is faster than the 3D counterparts. Our method is significantly faster than MeanCurvature, GeneralCylinder and Zhang-Suen’s Thinning, and it is slightly slower than ChordalAxisTransform. The relationship between and the algorithm’s time complexity is revealed in Fig. 16.

We compare with other skeletonization methods to evaluate the algorithm part of our system. Our algorithm extracts skeleton from 2D polygon (sub-skeleton extraction), and then transforms the skeleton to the 3D space to construct the 3D skeleton (sub-skeleton connection and global refinement). Therefore, we compare our method with both 2D and 3D skeletonization methods. For 3D skeletonization algorithms, we choose MeanCurvature mean_curvature_tagliasacchi2012 and GeneralCylinder general_cylinder_zhou2015 to compare. MeanCurvature is based on mesh contraction, while GeneralCylinder extracts skeleton by slicing mesh to find ROSA rosa. The merged mesh of Fig. 12 is used as input to the 3D skeletonization algorithms. For 2D skeletonization algorithms, we choose Zhang-Suen’s Thinning Zhang_thinning and ChordalAxisTransform of RigMesh rigmesh_borosan2012 to compare. Zhang-Suen’s algorithm is based on binary image thinning while ChordalAxisTransform extracts chordal axis from the constrained Delaunay triangulated polygon. We use the subpart contour of Fig. 12 (1st row) as the input to the 2D skeletonization algorithms. All experiments are conducted in the same machine. Table 2 compares the execution time of our algorithm with others. Our algorithm is significantly faster than MeanCurvature, GeneralCyliner, and ZhangSuen’s Thinning. We also make qualitative comparisons. The skeleton quality of our algorithm is on par with MeanCurvature of Fig. 17, and is better than RigMesh of Fig. 15 and GeneralCylinder of Fig. 18.

Figure 16: The execution time is proportional to .
Figure 17: The curve skeleton extracted by MeanCurvature mean_curvature_tagliasacchi2012. The skeleton is smooth and medially centered.
Figure 18: The disconnected curve skeleton extracted by GeneralCylinder general_cylinder_zhou2015. A color patch represents a general cylinder. The skeleton is sensitive to the general cylinder decomposition, see bird wing, lorax head, and octopus body.

5 Conclusion and future work

In this paper, we present an efficient framework to create the animatable skeleton in real-time for sketch modeling. We divide the big problem into three smaller ones: local sub-skeleton extraction; sub-skeleton connection; global skeleton refinement; and we conquer the problem one by one. To solve the first problem, we extract a straight skeleton by propagating the silhouette polygon inward, and we accelerate the algorithm by finding a concise shape approximation through DP simplification, a novel BoundedDP algorithm is then used to transfer the straight skeleton into the animatable bone skeleton. To solve the second problem of skeleton connection, we abstract the shape with general cylinders for bone and inscribed balls for joints, which enables a real-time intersection test among parts. And the precise attaching position is located by measuring Euclidean distance between parent bones and candidate child joints. To solve the third problem of creating different resolutions of skeletons, we propose four operations under three tiers of control. The branch-level operation allows users to control the bone complexity of a single axis. The subpart level and global level operation provide users with joints merging, branch pruning, and edge collapsing operations to control the overall structure. The implemented system is demonstrated to be easy to use and enables complex shape rig creation.

In the future, we intend to provide users more options to create shapes and skeletons by reusing existing 3d models. Specifically, our pipeline will not be limited to the immediately sketched shapes, users can also import 3D models created before and connect them to the newly sketched ones, and our system will automatically create an animatable skeleton for the model consisting of old and new shapes.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China under 61972340, 61732015 and 51775492.

References

Supplementary

Figure 19: Unsteady stroke. In our system, the perimeter of the stroke is 1972.28, and the step size for uniform discretization is 10.
Input: A simple polygon with vertices and edges oriented counter-clockwise.
Output: The straight skeleton .
// Initialization
= InitDLAV() ;
  // doubly linked active vertices
1 foreach  in  do
2       = CalBisector(, );
3       update straight skeleton by ;
4      
5 end foreach
/* Compute initial events, place them in a priority queue ordered by offsetting time */
6 foreach  in  do
7       if  is convex then
8             = CalIntersectTime(, );
9             = CalIntersectPoint(, );
10             Add to ;
11            
12      else
13             = FindOppositeEdgeInDLAV();
14             = CalSplitTime(, );
15             = CalSplitPoint(, );
16             Add to ;
17            
18       end if
19      
20 end foreach
// Propagation
21 while  is not empty do
22       pop top from ;
23       if  is an  then
             // collapse edge
24             = CreateIntersectNode();
25             = CalBisector();
26            
27      else
             // split polygon
28             = CreateSplitNodes();
29             = CalBisectors();
30            
31       end if
32      update DLAV , straight skeleton , and by new vertices and bisectors, see Line33;
33      
34 end while
ALGORITHM 3 Straight skeleton extraction