Hands are used in daily lives to handle objects and to better communicate with others. Especially, hands are almost the only way to control electronic devices except limited usage of speech. It is limited since speech is hard to protect privacy, and understanding of speech is difficult in noisy environment. Recent advancements in mobile devices and wearable devices demand better communication methods rather than touch screens which limit physical space. Due to the demand of more natural and convenient interacting methods, interaction using hand gestures has received lots of attention for human-computer interactions, virtual / augmented reality, and robot controls.
1.1 Related Work
Model-based tracking methods estimate hand pose by finding optimal parameters of computer-generated hand model using both current input image and previous results . Rehg et al. and Oikonomidis et al. used multiple RGB cameras to reduce occlusions and to increase visual features [7, 8]. The method in 
tracks a full DOF hand motion by minimizing discrepancy between input RGB-D image and computer-generated model using particle swarm optimization (PSO). Generating many possible hand pose images for each frame using computer graphics is computationally expensive and requires high performance GPUs to achieve about 1520 frames per second (fps) performance. Also, this method requires a user to place a hand on pre-determined position and pose to initialize tracking.
Recently, Sridhar et al. proposed the combined method of single frame-based method and model-based tracking method to blend advantages of each method. They used multiple color cameras for model-based tracking and a depth sensor to search their database, then a voting algorithm is applied to combine the results. Although multiple camera system helps to achieve better accuracy, it requires setup and calibration processes. Moreover, it requires GPUs to process multiple inputs at each frame. Qian et al. proposed another combined method using a depth sensor . They combined an efficient initialization method and a tracking method using PSO and iterative closest point (ICP). Their hand model is designed using only spheres to simplify objective function. Sharp et al
. also proposed the combined method of two-layer re-initialization using random forest and model fitting using PSO and genetic algorithm.
Most of existing hand tracking systems are rely on expensive, high power-consuming, and high performance GPUs since hand tracking is challenging because of complex articulations, self-occlusions, deformation, and rapid motions. Consequently, these systems are inappropriate for portable and wearable devices. Hand tracking systems for those devices do not need to consider huge viewpoint changes since in general, a user’s hand is relatively close to camera. In this paper, we focus on efficient hand articulations tracking system that does not require GPUs and considers mainly the situation when a user’s hand is close to camera. Even though we mainly consider small viewpoint changes, it is very challenging because of very limited computational power. Although we design and test our system without using GPUs, it can be implemented with inexpensive and low power-consuming GPUs for better accuracy while still being able to be used for mobile devices. To the best of our knowledge, the proposed system is the first one that can automatically adjust hand shapes which we call adaptive hand model while other methods manually and experimentally decided hand model size for each user. Moreover, we focus on real-time system without using GPUs for mobile and wearable devices. We also propose hierarchical random sampling of pixels on each depth map to achieve better performance with limited computations. Lastly, we improve an efficient re-initialization method at each frame using finger detection and classification.
Fig. 1 shows the entire process of proposed method. We first process hand segmentation from depth map and choose partial pixels from hand region using hierarchical random sampling. We also extract fingertip positions and finger joint rotations using an efficient finger detection and classification algorithm for re-initialization at each frame. Then hand pose and shape are estimated by minimizing discrepancy between the selected partial pixels and computer-generated hand model using PSO. Particles are initialized by previous frame’s result and finger detection / classification result.
2.1 Hand Segmentation
We process a very simple and effective segmentation by using a black wrist band and by assuming that a user’s hand is the closest object from camera. This assumption is valid in general environments where one interacts with mobile and wearable devices. The black wrist band is to get depth voids around the wrist since depth sensor cannot capture depth from black object well. The segmentation is processed by finding the connected components from the closest point. For details, we refer the reader to .
2.2 Hierarchical Random Sampling
We propose hierarchical random sampling of pixels on each depth map for efficient comparison between depth map from sensor and computer-generated hand model. It is computationally expensive to draw computer-generated hand model on image plane and compare entire image to input depth map. To reduce required computations, this process is replaced by comparing subset of pixels on input image to computer-generated hand model with only spheres. Thus we do not need to draw computer-generated hand model on image plane since the difference can be computed without drawing the model. Although tracking accuracy is improved with more subset points, required computation for comparison is also increased. Therefore, we focus on improving the selection of pixels to process from each depth map by applying hierarchical random sampling. The sampling aims to include more pixels on the region which has large depth variations since the region can be interpreted as more informative on depth map.
First, initial samples are randomly sampled. Then, hierarchical sampling is employed to include more points on large depth variation regions. To find large depth variation regions, the gradient matrix is computed as the sum of absolute x- and y- directional gradient of depth map . The gradients of x- and y- direction are computed by Sobel operators ( and ) in hand segment region:
where indicates component-wise absolute value and represents convolution.
For initial samples with large gradient, random samples are selected around initial samples by adding random values
from discrete uniform distribution to x- and y- coordinates respectively:
The random samples in are included in hierarchical sample set if the depth difference between initial sample and random sample is greater than a threshold :
The final sample set includes both initial samples and hierarchical samples . Sampled points are converted to with in millimeter, where is the total number of samples.
2.3 Adaptive Hand Model
Tracking accuracy increases as the hand model becomes more similar to each user’s hand. Although a personalized model can be generated by scanning the user’s hand, it requires a pre-processing step for each user. Therefore, we propose the adaptive hand model to consider different hand size and shape while avoiding to scan each user’s hand.
Our adaptive hand model consists of a hand size parameter vectorfor palm size and finger lengths, and a hand pose parameter vector for hand position and joint rotations. For hand size parameters, one parameter is for palm size and five parameters are for finger lengths from thumb finger to pinky finger. Finger width parameter is not considered since finger width is relatively less important and also computation power is limited. Hand pose parameter vector is defined to estimate translations and rotations of hand joints as in Fig. 3 (a).
Since hand shape at each frame is dependent on both size and pose parameters, two parameter vectors should be optimized simultaneously. However, the combination of two parameter vectors is 32 parameters, which is really complex to optimize even without considering correlation between parameters. Therefore, size parameters are considered only when five fingers are detected and classified by the re-initialization method in Section 2.5 since in that case, the optimization of pose parameters is relatively accurate and robust.
Hand model is designed using only spheres to reduce the computational complexity of objective function . The main reason is that in comparison between input depth map and the hand model, the length from a point to the surface of a sphere is simply the length from the point to the center of sphere subtracted by the radius.
2.4.1 Objective Function
Our objective function is designed mainly to minimize Euclidean distance between sampled pixels from input depth map and computer-generated hand model. The function consists of two discrepancy terms between hand model and depth map and one validity term. Two discrepancy terms are the Euclidean distance from depth map to hand model and from the model to depth map. The validity term is to check invalid overlapping between parts of hand model. The total cost is the weighted summation of these three terms. For details of general objective function, we refer the reader to the paper 
. For our system, since hand shape at each frame is determined by both hand pose and hand size parameters, cost is also determined by both parameters. We conducted experiment with another objective function which incorporates additional cost term to minimize temporal hand size parameter changes. However, tracking accuracy is not improved since the variation of hand size parameters are already regularized by the normal distribution.
2.4.2 Particle Swarm Optimization
Modified Particle Swarm Optimization (PSO) is used to find the best hand pose parameters and hand size parameters by minimizing an objective function. Each particle represents one state of hand pose and hand size parameters in our algorithm. The optimization method first initializes particles over possible solution range, and then finds the best solution among particles using objective function. Then particles are moved from current state to the direction of the best solution. The algorithm iterates finding the best solution and moving particles to the direction of best solution until it reaches maximum generation or termination condition.
Entire particles are initialized as the sum of optimized parameters at previous frame and random values from normal distribution if corresponding finger is not detected and classified by the method in Section 2.5. Otherwise, 75% of the corresponding parameters are initialized with the same method, and 25% of them are initialized with the sum of measured parameters from Section 2.5 and random values. The distribution of random values are and . However, at the first generation, hand size parameters are not considered to focus on the optimization of pose parameters since inaccurate pose parameters lead to wrong size parameters and the size parameters from previous frame are relatively reliable.
At each generation, for pose parameters, particles are updated using global best particle and personal best particle . Personal best particle is the state when the particle has the lowest cost until current generation. Global best particle is the lowest cost state among all particles and all generations until current generation. Particles are updated to the direction of personal best particle and global best particle using the following rules:
where is the objective function, weight , weight , is the index of each particle, and denote the generation index, and is current frame index.
For size parameters, particles are not updated to avoid misleading caused by the dependency between size parameters and pose parameters. Although size parameters are not updated at each generation, the particles with better size parameters are likely to have lower cost after many generations since pose parameters will become similar.
After reaching maximum generation, both size parameters and pose parameters are updated with the global best particle.
2.5 Re-initialization at each frame
Re-initialization at each frame is important to avoid manual initialization and error accumulation. However, general re-initialization methods using random forest or CNNs require large computation load which needs high performance GPUs to achieve real-time. Therefore, we improved the efficient finger detection and classification method proposed by 
. Although this re-initialization method works in limited cases, it initializes hand pose automatically, improves tracking accuracy, and is incorporated in real-time tracking without GPUs. We improve the re-initialization by estimating palm orientation using both current measurement and prior knowledge from previous frames, which is inspired by Kalman filtering.
2.5.1 Finger Detection
A simple finger detection algorithm is employed to detect planar fingers and orthogonal fingers. We define planar fingers as the fingers which are parallel to image plane and orthogonal fingers which are orthogonal to image plane. First, palm center is measured as the maximum of distance transform of hand segment. Then a planar finger candidate is detected by finding connected component from extreme distance point from palm center until the component reaches finger length. The detected finger candidate is classified into either a finger or a non-finger based on the component size. This process is iterated until it detects five fingers or the segment does not have any extreme distance point. After detecting planar fingers, an orthogonal finger candidate is detected by finding connected component from the closest point from a camera on both depth map and hand segment within a small window. It is also classified to either a finger or a non-finger based on the size of region. This process is also iterated until it reaches same condition as planar finger case. For planar fingers, principal component analysis (PCA) is applied to each detected region in order to calculate the orientation of each detected finger. For orthogonal fingers, the orientation is assumed that it is orthogonal to image plane.
2.5.2 Finger Classification
Finger classification is to use the result from finger detection for particle initialization process in Section 2.4. The algorithm classifies each detected finger to one of five finger classes. First, palm orientation is measured by applying PCA to palm segment. However, the measured orientation is not robust and accurate enough to use directly for classification. Therefore, palm orientation is predicted by the weighted summation of previously estimated palm orientation from PSO and currently measured palm orientation . The weights in summation are decided by the measurement error and priori estimation error . This is inspired by Kalman filtering .
where is current frame index and is chosen as a constant for simplicity. In our experiments, both and are set to 0.5. After each frame, priori estimation constant is updated as .
A set of junction points of fingers and palm is computed using detected fingertips, finger orientations, and finger length. Another set of junction points is calculated using hand model and predicted palm orientation. Each junction point from an input image is matched to the closest junction point from hand model.
If more than two of detected junctions are classified into the same class, optimization is employed to find the result with minimum cost. The algorithm first finds possible combinations that have minimum changes in initial classification.
The final class is classified as follows:
where is the index of detected finger and is the number of detected fingers. Fig. 4 (d)-(f) illustrate the procedure of finger classification. In Fig. 4 (e), white circles represent detected finger junctions and colored circles from red to purple indicate hand model junctions from thumb finger to pinky finger. A yellow circle which corresponds to predicted joint junction of index finger is overlapped with a white circle. Although this re-initialization method cannot detect all the fingers at every frame, it improves tracking accuracy. It can also initialize at each frame including the very first frame, and takes only a few milliseconds using only CPU.
3 Experimental Results
The algorithm is tested using a Creative Senz3D camera and a computer with Intel Core i7-3770 3.4GHz CPU, 16GB RAM, and without GPUs. Although the machine has 16GB RAM, this algorithm only uses about 60MB memory. Although a 3.4GHz CPU is used, we believe similar computation power can be obtained using the combination of mobile CPU and mobile GPUs.
We captured 500 frames for each subject and labeled wrist and five fingertips of last 400 frames. The dataset is available on our repository111https://github.com/byeongkeun-kang/HandTracking. Initial 100 frames contain open-hand pose that camera can capture at least some fingers using the algorithm in Section 2.5 for automatic initialization. The initial frames are not used to compute accuracy. Error in accuracy is computed using 3D Euclidean distance in millimeter.
Unless specifically mentioned, we sample 256 points from each depth map and optimize with 256 particles and 6 generations. Table 1 shows the processing time at this setting. Even though the algorithm is not fully optimized, it achieves about 16 FPS using eight threads on CPU.
Comparison of adaptive hand model and fixed scale hand model for three subjects (top: subject 1, middle: subject 2, bottom: subject 3). The legend represents standard deviationsof hand size parameter randomness. Standard deviation of 0 means that the scaling factor of hand model is always the same with initial scaling factor. Larger standard deviation means more possibility of large update of scaling factor at each frame. The result shows that the performance of adaptive hand model is better than the performance of fixed scaled hand model in general.
3.1 Adaptive Hand Model
The performance comparison of hand model with adaptive scaling and fixed scaling is demonstrated in Fig. 5 for different subjects, initial scaling factors, and hand size parameter randomness. Scaling factors are chosen from 0.6 to 1.2 with 0.1 step size and standard deviation of randomness is chosen from 0 to 0.015 with 0.005 step size. Also, Table 2 clearly shows that adaptive hand model reduces error about 5. The overall results show that adaptive hand model adjusts hand scaling factor automatically to minimize discrepancy between pre-defined hand model and user’s hand, and users do not need to manually and experimentally select scaling factor of hand model. Moreover, to consider the case that the user is changed after starting tracking, we keep update hand scaling factor.
3.2 Finger Classification
Correct classification rate (CCR) is calculated for each finger with and without the proposed palm orientation prediction in Table 3. The result shows that by using the proposed prediction, the average CCR is improved from 70.6% to 84.8%.
3.3 Hierarchical Random Sampling
The performance of random sampling and hierarchical random sampling is compared in Table 4. The average error is computed using the hand model scaled by the best performance fixed scaling factor, 0.7, 0.9, and 0.7 for subject 1, 2, and 3 respectively. The average accuracy is improved from 17.01 to 16.11. The computational cost of calculating gradient is much smaller than increasing the number of generations, particles, or samples to achieve the same improvement. However, if the number of sampling pixels is too small, random sampling might be better since hierarchical random sampling prevents that sampled pixels are more globally distributed.
We present an efficient hand articulations tracking system for mobile and wearable devices which do not have high performance GPUs. We show that the proposed system achieves both automatic hand model adjustment using adaptive hand model and real-time tracking without using GPUs. We also achieve improved accuracy using hierarchical random sampling and improved efficient re-initialization at each frame while limiting required computations.
-  Erol, A., Bebis, G., Nicolescu, M., Boyle, R.D., Twombly, X.: Vision-based hand pose estimation: A review. Comput. Vis. Image Underst. 108 (2007)
-  Athitsos, V., Sclaroff, S.: Estimating 3d hand pose from a cluttered image.
-  Wang, R.Y., Popović, J.: Real-time hand-tracking with a color glove. ACM Trans. Graph. 28 (2009)
-  Tang, D., Yu, T.H., Kim, T.K.: Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In: Computer Vision (ICCV), 2013 IEEE International Conference on. (2013)
-  Tompson, J., Stein, M., Lecun, Y., Perlin, K.: Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph. 33 (2014)
-  Stenger, B., Thayananthan, A., Torr, P., Cipolla, R.: Model-based hand tracking using a hierarchical bayesian filter. Pattern Analysis and Machine Intelligence, IEEE Transactions on 28 (2006)
-  Rehg, J.M., Kanade, T.: Visual tracking of high dof articulated structures: An application to human hand tracking. In: Computer Vision — ECCV ’94. (1994)
-  Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Markerless and efficient 26-dof hand pose recovery. In: Proceedings of the 10th Asian Conference on Computer Vision. (2011)
-  Oikonomidis, I., Kyriazis, N., Argyros, A.: Efficient model-based 3d tracking of hand articulations using kinect. In: Proceedings of the British Machine Vision Conference. (2011)
-  Qian, C., Sun, X., Wei, Y., Tang, X., Sun, J.: Realtime and robust hand tracking from depth. In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. (2014)
-  Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., Freedman, D., Kohli, P., Krupka, E., Fitzgibbon, A., Izadi, S.: Accurate, robust, and flexible real-time hand tracking. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. CHI ’15, ACM (2015)
-  Kang, B., Tripathi, S., Nguyen, T.: Real-time sign language fingerspelling recognition using convolutional neural networks from depth map. In: Pattern Recognition (ACPR), 2015 3rd IAPR Asian Conference on. (2015)
-  Kalman, R.E.: A new approach to linear filtering and prediction problems. Journal of Fluids Engineering 82 (1960)