, and applied machine learning. The primary motivation of teleoperation is to allow a robot system to perform complex tasks by harnessing the cognition, creativity, and reactivity of humans through a human-machine interface (HMI). Through the years, many advancements have been made in this research field including the incorporation of haptic feedback [9, 4] and improved human skeletal and finger tracking [34, 39, 45, 7]
. Nevertheless, the cost of many high DoA teleoperation or tracking systems is prohibitive. This research bridges this gap by providing a low-cost, markerless, glove-free teleoperation solution that leverages innovations in machine vision, optimization, motion generation, and GPU compute. Despite its low-cost, the system retains the ability to capture and relay fine dexterous manipulation to drive a highly actuated robot system to solve a wide variety of grasping and manipulation tasks. Altogether, four Intel RealSense depth cameras and two NVIDIA GPUs in combination with deep learning and nonlinear optimization produced a minimal-footprint, dexterous teleoperation system. Despite the lack of tactile feedback, the system is highly capable and effective through human cognition. This result corroborates human gaze studies that indicate that humans learn to leverage vision for planning, control, and state prediction of hand actions prior to accurate hand control. Therefore, the teleoperation system exploits the human ability to plan, move, and predict the consequence of their physical actions from vision alone; a sufficient condition for solving a variety of tasks. The main contributions are summarised below:
Markerless, glove-free and entirely vision-based teleoperation system that dexterously articulates a highly-actuated robotic hand-arm system with direct imitation.
Novel cost function and projection schemes for kinematically retargeting human hand joints to Allegro hand joints that preserve hand dexterity and feasibility of precision grasps in the presence of hand joint tracking error.
Demonstration of teleoperation system on a wide variety of tasks particularly involving fine manipulations and dexterity, e.g., pulling out paper currency from wallet and grasping two cubes with four fingers as shown in Fig. LABEL:fig:title.
System assessment across two trained human demonstrators — also called pilots — revealed that high task success rates can be achieved despite the lack of tactile feedback.
Ii Related Work
Teleoperation via the human hand is traditionally done using either pure vision or sensorized glove-based solutions and relies on accurate hand tracking. Early vision-based solutions to hand tracknig include Dorner et al.  who used colour markers on the hand to track the 2D positions of joints and fit a skeleton model to obtain the hand pose and finger joints in 3D. Similarly, Theobalt et al. ,  use colour markers to track the motion of the hand in high speed scenarios in sports. Wang et al. 
propose a fully coloured glove to track the joints and pose of the hand for augmented reality (AR) and virtual reality (VR) applications. Impressive strides have also been made in the past few years in bare human hand tracking particularly with deep learning[39, 26, 45, 7, 2, 27]. Glove- and marker-free hand tracking is attractive as it produces minimal external interference with the hand and reduces the system physical footprint, but reliability and accuracy with such approaches have not yet reached desireable performance levels. Moreover, most of these works study hand tracking in isolation without evaluation of tracking performance in controlling a physical robot. Li et al.  proposed vision-based teleoperation through deep learning that calculated Shadow robotic hand joint angles from observed human hand depth images. However, this approach was not extended to a full robotic system and the mapping results had performance issues as later discussed in Section VII-A. Antotsiou et al.  show teleoperation of an anthropomorphic hand in simulation using depth based hand tracking. They focus on very simplistic grasping e.g. picking a ball on a flat table and do not show the level of dexterity as seen herein. In general, hand tracking with vision remains a very challenging problem due to self-similarity and self-occlusions observed in the hand.
do provide the user with tactile feedback in the form of physical resistance and spatial contact, but this benefit comes with heightened cost and additional bulk to the user. Sometimes these systems still require an external hand pose estimation module as in to enable a freely moving mobile hand as with DexPilot. Still, these approaches are important to investigate how relaying tactile feedback to human pilots improve teleoperation capability considering that numerous neurophysiological studies have uncovered the importance of tactile feedback in human sensorimotor control of the human hand [14, 16, 15]. Alternatively, motion capture systems provide accurate point tracking solutions, but can be expensive with multi-camera systems. Moreover, when tracking all the joints of the hands, the correspondence problem between markers on the fingers and cameras still needs to be solved. For instance, Han et al. 
track hands with an array of OptiTrack cameras while solving the correspondence problem via a neural network. They show impressive tracking of two hands in VR settings. Zhuanget al.  leverage a motion capture system and myoelectric device for teleoperative control over a hand-arm system. The system worked convincingly well as a prosthetic HMI, but experiments did not reveal how well the system works for intricate in-hand manipulation tasks (as assessed herein). Overall, these solutions are viable alternatives, but are significantly more expensive and with larger physical footprints than DexPilot. Finally, some of these approaches only consider the problem of tracking hand movements, and are not complete teleoperation solutions like DexPilot.
The teleoperation setup comprised of a robot system and an adjacent human pilot arena as shown in Fig. 1. The robot system was a KUKA LBR iiwa7 R800 series robot arm and a Wonik Robotics Allegro hand. The Allegro hand was retrofitted with four Syntouch Biotac tactile sensors at the fingertips and 3M TB641 grip tape applied to the inner surfaces of the phalanges and palm. The rubbery surfaces of both the Biotac sensors and 3M tape augmented the frictional properties of the hand while the Biotacs themselves produced 23 signals that can later be used to learn sensorimotor control from demonstrations. In total, the robotic system has 92 tactile signals, 23 joint position signals, and 23 joint torque actions. The human arena was a black-clothed table that housed four calibrated and time-synchronized Intel Realsense D415 RGB-D cameras spatially arranged to yield good coverage of the observation volume within the ideal camera sensing range. Ideally, depth observations should remain within 1 for each camera; otherwise, the depth quality degenerates. The human arena is directly adjacent to the robot to improve line-of-sight and visual proximity since teleoperation is entirely based on the human vision and spatial reasoning. The teleoperation work volume is 80 55 38 .
To produce a natural-feeling teleoperation system, an imitation-type paradigm is adopted. The bare human hand motion — pose and finger configuration — was constantly observed and measured by the visual perception module. The human hand motion is then relayed to the robot system in such a way that the copied motion is self-evident. This approach enables the human pilot to curl and arrange their fingers, form grasps, reorient and translate their palms, with the robot system following in a similar manner. DexPilot relies heavily on DART  which forms the backbone of tracking the pose and joint angles of the hand. In the following, we explain the main components of the overall system: 1) DART for hand tracking (Section V). 2) deep neural networks for human hand state estimation and robustifying DART (Section VI). 3) human hand state refinement with DART and its conversion through nonlinear optimization to Allegro hand states (Section VII-A) 4) motion generation and control through Riemannian Motion Policies (RMPs) and torque-level impedance controllers (Section VII-B). The fully system architecture is shown in Fig. 2 where these components are daisy chained. Altogether, the system produces a latency of about one second.
DART  produces continuous pose and joint angle tracking of the human hand by matching an articulated model of the hand against an input point cloud. The human hand model was obtained from  and turned into a single mesh model . With CAD software, the fingers of the mesh model were separated into their respective proximal, medial, and distal links, and re-exported as separate meshes along with an associated XML file that described their kinematic arrangement. In total, the human hand model possessed 20 revolute joints: four joints per finger with one abduction joint and three flexion joints.
Vi Estimating hand pose and joint angles with neural networks
Since DART is a model-based tracker that relies on non-linear optimisation, it needs initialisation which typically comes from estimates from previous frame. If this initialisation is not within the basin of convergence, the tracker can fail catastrophically. Often, DART can match the hand model to the raw point cloud in regions of spurious local minima leading to tracking failures every few minutes. Therefore, to allow for tracking over long periods of time — as needed for teleoperation — hand pose priors and hand segmentation can be implemented to prevent the hand model from converging to incorrect local minima. One way of getting hand pose priors would to be train a neural network on a large dataset of human hand poses. Although various datasets [39, 45] exist that provide annotations for hand pose as well as segmentation, they were not suitable for this setting primarily due to different sensor characteristics and lack of hand poses required for dexterous manipulation.
Therefore, a fabric glove (shown in Fig. 3) with coloured blobs was initially used as an effective solution for obtaining a hand pose prior with a deep neural network. The data collection proceeded in two phases. In the first phase, the user wore the glove to obtain hand pose priors for DART to track human hand robustly. This process generated hand pose and joint angle annotations for raw depth maps from the RGB-D cameras for the second phase. The second phase uses these annotations and operates on raw point cloud from corresponding depth maps and frees the user from having to wear the glove. We explain differetnt phases below.
who used it for hand tracking. It is important to clarify that hand tracking includes both the hand pose and the joint angles of the fingers. In our set-up, the user moves their hand over a table in a multi-camera studio with four Intel RealSense D415 RGB-D cameras pointing downwards to the table. The problem of hand pose estimation is formulated with the glove via keypoint localisation: ResNet-50 with spatial-softmax is used to regress from an RGB image to the 2D locations of the centers of the coloured blobs on the glove. We call this network GloveNet. The coloured blobs at finger-tips are also regressed but were found to be not helpful in full hand tracking in the end and therefore we only use the predictions of the blobs on the back of the palm for hand pose estimation. We explain that in detail in the appendix.
The hand pose can be estimated by three unique keypoints as indicated by three different coloured blobs at the back of the palm of glove. To obtain annotations for the centers of the blobs, HSV thresholding in OpenCV is used to generate segmentations and compute the centeroids of these segmented coloured blobs. To aid segmentation for high quality annotations, the user wears a black glove with coloured blobs and moves the hand over a table also covered with black cloth. The pose of the hand can be obtained via predicted 2D locations of the blobs from all four cameras: the 2D keypoints are converted to their corresponding 3D locations using the depth values resulting in each blob having four 3D predictions in total from four cameras. These 3D locations are filtered and temporally smoothed to obtain the hand pose. Hand segmentation is also obtained by removing the 3D points that fall outside the bounding volume of the hand. The dimensions of this volume were obtained heuristically from the hand pose obtained from the neural network predictions. Crucially, DART now only optimises on the segmented hand points, preventing the hand model from sliding out to points on the arm as often observed when a full point cloud is used. It is worthwhile remembering that DART does not use RGB images — the glove only provided pose priors and aided hand segmentation — and therefore the result of DART with hand pose priors and segmentation in the first phase is generating annotations for raw point cloud captured with the cameras for second phase which can operate on the bare human hand.
It is desirable to free the user from having to wear glove in future for any teleoperation. While the first phase operates on RGB image, the second phase operates directly on fused point cloud of bare human hand obtained by back-projecting four depth maps from extrinsically calibrated cameras into a global reference frame with annotations provided by first phase. Since the fused point cloud contains both the points on table as well as human body and arm it becomes imperative to first localise the hand. Points on the table are removed by fitting a plane and the remaining points containing the arm and human body are fed to a PointNet++ based  architecture that localises the hand as well as provides the hand pose. Our network is based on 
who estimate hand pose via a voting based regression to the 3D positions of specified keypoints on the hand, a technique reminiscent of spatial-softmax often used in 2D keypoint localisation. It is trained to predict 3D coordinates of 23 keypoints specified on the hand — 4 joint keypoints each on 5 fingers and 3 at the back of the palm for hand pose estimation. The loss function is standard Euclidean loss between predicted and the ground truth keypoints together with the voting loss inspired by. An auxiliary segmentation loss is also added to obtain hand segmentation. For efficiency reasons, any input point cloud of size is sub-sampled uniformly to a fixed size before feeding to our network.
While reasonable hand pose estimation and segmentation is achieved, getting high quality predictions for the 20 joint keypoints on the fingers remains difficult with this network. The uniform sub-sampling used at the input means that points on the fingers are not densely sampled and therefore a second stage refinement is needed which resamples points on the hand from the original raw point cloud given the pose and segmentation of the first stage. The overall network architecture is shown in Fig. 4. The second stage is trained on only the loss functions pertaining to the keypoints and no segmentation is needed. It uses the points sampled on the hand instead and predicts accurately the 23 keypoints. To enable robustness to any inaccuracies in the hand pose from the first stage, additional randomization is added to the hand pose for second stage. The Fig. 5 shows how the second stage refinement improves the system. Overall, both stages of our network are trained on 100K point clouds collected over a batch of 30-45 minutes each for 7-8 hours in total by running DART with priors from glove. Together they provide annotations for keypoints, joint angles and segmentation. The training takes 15 hours in total on a single NVIDIA TitanXp GPU.
While keypoints are a natural representation for Euclidean space as used in PointNet++ architectures, most articulated models use joints as a natural parameterisation. Therefore, it is desirable to have output in joint space which can serve as joint priors to DART. A third neural network is trained that maps 23 keypoint locations predicted by our PointNet++ inspired architecture to corresponding joint angles. This neural network, called JointNet, is a two-layer fully connected network that takes input of size and predicts
-dimensional vector of joint angles for fingers.
The neural networks are trained on data collected within the limits of the studio work volume across multiple human hands, ensuring accurate pose fits for this application and enabling sensible priors for DART. Qualitatively, the hand tracker worked well for hands geometrically close to the DART human hand model. Overall, average keypoint error on a validation set of seven thousand images of differing hand poses and finger configurations was 9.7 mm (comparable to results reported in , but on a different dataset) and joint error was 1.33 degrees per joint.
Vii Robot Motion Generation
Vii-a Kinematic Retargeting
Teleoperation of a robotic hand that is kinematically disparate from the human hand required a module that can map the observed human hand joints to the Allegro joints. There are many different approaches to kinematic retargeting in the literature. For instance, a BioIK solver was used to match (between the human and Shadow hand) the positions from palm to the fingertips and medial joints, and the directionality of proximal phalanges and thumb distal phalange in 
. The optimized mapping was used to label human depth images to learn end-to-end a deep network that can ingest a depth image and output joint angles for the Shadow hand. Although interesting, the result produced retargeting results that are not useful for precision grasps (e.g., pinching) where gaps between fingertips need to be small or zero. Motion retargeting is also present in the animation field. For instance, a deep recurrent neural network was unsupervised trained to retarget motion between skeletons. Although the synthesized motion look compelling, it is unclear whether these approaches work well enough for teleoperative manipulation where the important task spaces for capturing grasping and manipulation behavior are likely not equivalent to those that capture visual consistencies. The approach herein prioritized fingertip task-space metrics because distal regions are of highest priority in grasping and manipulation tasks as measured by their contact prevalence , degree of innervation , and heightened controllability for fine, in-hand manipulation skill . Moreover, the joint axes and locations between the two hands are strikingly different, and therefore, no metrics directly comparing joint angles between the two hands are used. To capture and optimize for the positioning of fingertips, both distance and direction among fingertips were considered. Specifically, the cost function for kinematic retargeting was chosen as
where are the angles of the human hand model and Allegro hand, respectively, is the vector pointing from the origin of one coordinate system to another, expressed in the origin coordinate system (see Fig. 6). Furthermore, and . The switching weight function is defined as
where and are vector sets defined in Table I, The distancing function, , is defined as
where is a scaling factor, closes the distance between a primary finger and the thumb, and forces a minimum separation distance between two primary fingers when both primary fingers are being projected with the thumb. These projections ensure that contact between primary fingers and the thumb are close without inducing primary finger collisions in a precision grasp. This was found to be particularly useful in the presence of visual finger tracking inaccuracies. Importantly, the vectors not only capture distance and direction from one task space to another, but their expression in local coordinates further contains information on how the coordinate systems, and thereby fingertips, are oriented with one another. The coordinate systems of the human hand model must therefore have equivalent coordinate systems on the Allegro model with similarity in orientation and placement. The vectors shown in Fig. 6 were a minimal set that produced the desired retargeting behavior. Finally, is a weight on regularizing the Allegro angles to zero (equivalent to fully opened the hand). This term helped greatly with reducing redundancy in solution and ensured that the hand never entered strange minima that was difficult to recover from (e.g., the fingers embedding themselves into the palm). Finally, to further reduce redundancy, the search space, and to emulate the human hand, the distal joints for the primary (index, middle, and ring) fingers of the Allegro hand were constrained to equal their medial joints. Various mappings from human hand to Allegro as produced by our kinematic retargeting are show in Fig. 7.
For implementation, the above cost function was minimized in real-time using the Sequential Least-Squares Quadratic Programming (SLSQP) algorithm [18, 19] of the NLopt library . The routine was initiated with Allegro joint angles set to zero, and every solution thereafter was initiated with the preceding solution. Moreover, the forward kinematic calculations between the various coordinate systems of both the human hand model and Allegro hand were found using the Orocos Kinematics and Dynamics library . Finally, a first-order low-pass filter was applied to the raw retargeted joint angles in order to remove high-frequency noise present in tracking the human hand and to smooth discrete events like the projection algorithm inducing step-response changes in retargeted angles.
|Vectors that originate from a primary finger (index, middle, ring) and point to the thumb.|
|Vectors between two primary fingers when both primary fingers have associated vectors , e.g., both primary fingers are being projected with the thumb.|
Vii-B Riemannian Motion Policies
Riemannian Motion Policies (RMPs) are real-time motion generation methods that calculate acceleration fields from potential function gradients and corresponding Riemannian metrics [31, 3]. RMPs combines the generation of multi-priority Cartesian trajectories and collision avoidance behaviors together in one cohesive framework. They are used to control the Cartesian pose of the Allegro palm given the observed human hand pose while avoiding arm-palm collisions with the table or operator using collision planes. Given these objectives, the RMPs generated target arm joint trajectories that were sent to the arm’s torque-level impedance controller at 200 Hz. The kinematically retargeted Allegro angles were sent to the torque-level joint controller at 30 Hz. One final calibration detail involves registering human hand pose movements with the robot system. This was accomplished by finding the transformation from the robot coordinate system to the camera coordinate system. This transformation was calculated using the initial view of the human hand and an assumed initial pose of the robot hand. To facilitate spatial reasoning of the pilot, the desired initial hand pose of the pilot is a fully open hand with the palm parallel to the table and fingers pointing forwards. The assumed initial pose of the robot mimics this pose. In this way, the robot moves in the same direction as the pilot’s hand, enabling intuitive spatial reasoning.
The DexPilot system was systematically tested across a wide range of physical tasks that test precision and power grasps, prehensile and non-prehensile manipulation, and finger gaiting (see Fig. 8 for test objects and Figs. 9, 10, 11 for teleoperative manipulation). The description of the tasks are provided in Table II.
Pick and Place
||Pick object on the table and place it in a red bowl.||grasping, releasing|
||Stacking three blocks on top of each other.||precision grasping, precision releasing|
|Pouring Beads||Pour beads from a cup into a bowl.||grasping, pouring|
|Opening Jar||Open peanut jar and place lid on table.||finger gaiting, grasping, releasing|
|Brick Gaiting||Pick up and in-hand rotate brick 180 degrees and place back down.||grasping, in-hand manipulation, releasing|
|Container||Open plastic container, extract and open cardboard box.||twisting, pulling, pushing, grasping, in-hand manipulation|
|Cup Insertion||Inserting concentric cups inside each other.||grasping, releasing|
|Tea Drawer||Pull open tea drawer, extra single bag of tea and place on table, close tea drawer.||precision grasping, pulling, pushing, releasing|
|Card Sliding||Slide a card along the box and pick it up with two fingers.||sliding, precision grasping, releasing|
|Wallet||Open the wallet and pull out paper money.||precision grasping, pulling, pushing, in-hand manipulation|
|Box Flip||Flip the box by 90 degrees and place it on the designated goal.||pushing, grasping, releasing|
Before benchmarking, the pilots went through a warm-up training phase where they tried to solve the task with 3-5 non-consecutive attempts. Later, five consecutive test trials were conducted by each pilot for each task to avoid preferential selection of results and pilots were graded based on their performance. The performance metrics for these tasks include mean completion time (CT) and success rate which capture speed and reliability of the teleoperation system. The system was tested with two pilots and the performance measures are reported in Fig. 12 (see more videos at https://sites.google.com/view/dex-pilot). Overall, the system can be reliably used to solve a variety of tasks with a range of difficulty. Differences in mean CT across tasks indicate the effects of task complexity and horizon scale. Discrepancies in mean CT across pilots per task indicate that there does exist a dependency on pilot behavior. Effects include fatigue, training, creativity, and motivation. The ability to solve these tasks reveal that the teleoperation system has the dexterity to exhibit precision and power grasps, multi-fingered prehensile and non-prehensile manipulation, in-hand finger gaiting, and compound in-hand manipulation (e.g., grasping with two fingers while simultaneously manipulating with the remaining fingers). Note, certain tasks, e.g. Container and Wallet, take a particularly long time to teleoperate largely due to the fact that these tasks are multi-stage tasks. On the other hand, the task requiring picking small cubes is particularly challenging because the behavior of releasing the grasps on these objects with the projection scheme used in kinematic retargeting can be unpredictable. Nevertheless, such a rich exhibition of dexterous skill transferred solely through the observation of the bare human hand provides empirical evidence that the approach and architecture herein works well. An important aspect that is worth highlighting is that although the full teleoperation process for a particular task may not be perfect (e.g. the pilot may lose an object in hand but fetches it again to accomplish the task), the data collected is still equally valuable in helping robot learn to recover from failures. Additionally, in the spirit of , the data can be regarded as play data which is useful to learn long range planning. Visualization of a sensorimotor solution to the Brick Gaiting task can be seen in Fig. 14. As shown, discrete events like intermittent finger-object contacts can be observed in the tactile signals. Undulations in these state-action signals reveal the rich, complex behavior evoked in the system through this embodied setting. Force estimates can also be obtained as in . This data can now be generated on-demand for a particular task with the hope that functional sensorimotor patterns may be gleaned and imparted to the system in an autonomous setting.
DexPilot enabled a highly-actuated hand-arm system to find a motor solution to a variety of manipulation tasks by translating observed human hand and finger motion to robot arm and finger motion. Importantly, several tasks like extracting money from a wallet and opening a cardboard box within a plastic container were so complex that hand-engineering a robot solution or applying learning methods directly are likely intractable. Solving these tasks and the others through the embodied robotic system revealed that a sensorimotor solution did exist and that these solutions can be generated on-demand for many demonstrations. Furthermore, creating these solutions on the system itself allows for the reading, access, and storage of the 92 tactile signals in the robot’s fingertips, 23 commanded and measured joint position and velocity signals through the hand and arm, 23 torque commands throughout the system, and any camera feeds associated with the system. This rich source of data will be critical for the application of learning methods that may hopefully learn to solve complex, multi-stage, long horizon tasks. Applying these learning methods is a future direction for this enabling work. Moreover, the current DexPilot system will be improved in a variety of ways in the future as well. For instance, human hand tracking accuracy could be improved with advancements in deep learning architectures, inclusion of RGB data, larger data sets, and changes to imaging hardware. Ideally, human hand tracking accuracy should be improved enough to greatly reduce the projection distance in the kinematic retargeting approach, enhancing fine finger control and manipulation over small objects and multi-fingered precision grasps. Grasp and manipulation control algorithms  could be implemented on the hand that automates force modulating control to reduce the control burden on the user and minimize unintentional part drops from the application of incorrect grip forces. Finally, intent recognition schemes could be implemented that enables the robot to predict human intention and deploy automated solutions, e.g., the system recognizes the human’s intent to grasp an object and the system automatically acquires the grasp. Such a co-dependent system would allow a human to direct the robot with full knowledge of a task and its solution strategy, while the robot system controls the finer details of the solution implementation.
Overall, DexPilot is a viable, low-cost solution for teleoperating a high DoA robotic system; however, there are areas that could be improved with the current implementation. For instance, the observable work volume of the pilot could be enlarged to allow for tasks that cover greater distances with better RGB-D cameras. The projection schemes in kinematic retargeting enabled successful manipulation of small objects, but can interfere with finger gaiting tasks and timely releasing grasps on small objects as shown in Fig. 16. Fortunately, this feature can be turned off when desired, but ultimately, this interference should be non-existent. This issue could be solved entirely with hand tracking that can accurately resolve situations where the human hand fingertips are making contact. Human hand tracking could also be further improved with enhanced robustness across size and shape of the pilot’s hand. The lack of tactile feedback makes precision tasks difficult to complete. To compensate, building in autonomous control features could alleviate some of the control burden on the pilot. Furthermore, the system latency could be reduced and the responsiveness of the RMP motion generator could be tuned for faster reactions. Finally, high-precision tasks like slip-fit peg-in-hole insertions pose a challenge to the current system. Peg-in-hole insertions on the NIST task board [41, 28] were attempted with DexPilot, but results were mostly unsatisfactory. Fig. 15 shows one attempt where the pilot managed to successfully guide the system to insert a 16 mm 10 mm 49.5 mm peg with a 0.1 mm hole clearance. Encouragingly, these tasks which have not been successfully solved by robotic systems that use machine vision and robotic hands are now made possible by such systems. However, success rates are typically within 10 % when under specific conditions on the initial placement of the NIST task board and parts (close to the user). The difficulty of completing such tasks could be significantly reduced with improved hand tracking performance, automated precision grip control on the assembly objects, and improved sight to the small parts and insertion locations.
We would like to thank Adithya Murali, Jonathan Tremblay, Balakumar Sundaralingam, Clemens Eppner, Chris Paxton, Yashraj Narang, Alexander Lambert, Timothy Lee, Michelle Lee, Adam Fishman, Yunzhu Li, Ajay Mandlekar, Muhammad Asif Rana, Carlos Florensa, Krishna Murthy Jatavallabhula, Jonathan Tompson, Joseph Xu, Kendall Lowrey, Lerrel Pinto, Ian Abraham, Jan Czarnowski, Raluca Scona, Tucker Hermans, Svetoslav Kolev, Umar Iqbal, Pavlo Molchanov, Emo Todorov, Artem Molchanov, Yevgen Chebotar, Viktor Makoviychuk, Visak Chadalavada and James Davidson for useful discussions and feedback during the development of this work.
Task-oriented hand motion retargeting for dexterous manipulation imitation.
Proceedings of the European Conference on Computer Vision (ECCV) Workshop, Cited by: §II.
-  (2018) OpenPose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008. Cited by: §II.
-  (2018) Rmpflow: a computational graph for automatic motion policy generation. arXiv preprint arXiv:1811.07049. Cited by: §VII-B.
-  CyberGlove Systems. Note: www.cyberglovesystems.com Cited by: §I, §II.
-  (2012) Robonaut 2—initial activities on-board the iss. In 2012 IEEE Aerospace Conference, pp. 1–12. Cited by: §I.
-  (1994) Chasing the colour glove: visual hand tracking. Ph.D. Thesis, Simon Fraser University. Cited by: §II, §VI.
-  (2018) Point-to-point regression pointnet for 3d hand pose estimation. In ECCV, Cited by: §I, §II, Fig. 4, §VI, §VI.
-  (2018) Online optical marker-based hand tracking with deep labels. ACM TOG. Cited by: §II.
-  HaptX. Note: https://haptx.com/ Cited by: §I, §II.
-  (2019) Learning joint reconstruction of hands and manipulated objects. In CVPR, Cited by: §V.
-  (2016) Deep residual learning for image recognition. In CVPR, Cited by: §XII-A.
-  (2016) Deep residual learning for image recognition. In CVPR, Cited by: §VI.
-  Imgaug. Note: https://github.com/aleju/imgaug Cited by: §XII-A.
-  (2009) Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience 10 (5), pp. 345. Cited by: §I, §II, §VII-A.
-  (1983) Tactile sensory coding in the glabrous skin of the human hand. Trends in neurosciences 6, pp. 27–32. Cited by: §II.
-  (1984) Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental brain research 56 (3), pp. 550–564. Cited by: §II.
-  The nlopt nonlinear-optimization package. Note: http://github.com/stevengj/nlopt Cited by: §VII-A.
-  (1988) A software package for sequential quadratic programming. Forschungsbericht- Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt. Cited by: §VII-A.
-  (1994) Algorithm 733: tomp–fortran modules for optimal control calculations. TOMS. Cited by: §VII-A.
-  (2015) MuJoCo HAPTIX: a virtual reality system for hand manipulation. In Humanoids, Cited by: §II.
-  (2019) Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network. In ICRA, Cited by: §II, §VII-A.
-  (2017) A glove-based system for studying hand-object manipulation via joint pose and force sensing. In IROS, Cited by: §II.
-  (2019) High-fidelity grasping in virtual reality using a glove-based system. In ICRA, Cited by: §II.
-  (2019) Learning latent plans from play. arXiv preprint arXiv:1903.01973. Cited by: §VIII.
-  (2017) Analysis of human–robot interaction at the darpa robotics challenge finals. The International Journal of Robotics Research 36 (5-7), pp. 483–513. Cited by: §I.
-  (2015) Hands deep in deep learning for hand pose estimation. arXiv preprint arXiv:1502.06807. Cited by: §II.
-  (2019) Generalized feedback loop for joint hand-object pose estimation. IEEE transactions on pattern analysis and machine intelligence. Cited by: §II.
-  Robotic grasping and manipulation for assembly. Note: https://www.nist.gov/el/intelligent-systems-division-73500/robotic-grasping-and-manipulation-assembly/assembly Cited by: §X.
-  Orocos kinematics and dynamics library. Note: https://github.com/orocos/orocos_kinematics_dynamics Cited by: §VII-A.
-  (2017) PointNet++: deep hierarchical feature learning on point sets in a metric space. CoRR abs/1706.02413. Cited by: §VI.
-  (2018) Riemannian motion policies. arXiv preprint arXiv:1801.02854. Cited by: §VII-B.
-  (2017) Embodied hands: modeling and capturing hands and bodies together. ACM SIGGRAPH Asia. Cited by: §V.
-  (2014) DART: dense articulated real-time tracking.. In RSS, Cited by: §IV, §V.
-  (2006) Model-based hand tracking using a hierarchical bayesian filter. PAMI. Cited by: §I.
-  (2008) Transcontinental telesurgical nephrectomy using the da vinci robot in a porcine model. Urology 71 (5), pp. 971–973. Cited by: §I.
-  (2019) Robust learning of tactile force estimation through robot interaction. In 2019 International Conference on Robotics and Automation (ICRA), Cited by: §VIII.
-  (2019) Learning the signatures of the human grasp using a scalable tactile glove. Nature 569 (7758), pp. 698. Cited by: §VII-A.
-  (2004) Pitching a baseball: tracking high-speed motion with multi-exposure images. In ACM TOG, Cited by: §II.
-  (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM ToG. Cited by: §I, §II, §VI.
-  (2018) Comparative peg-in-hole testing of a force-based manipulation controlled robotic hand. IEEE Transactions on Robotics. Cited by: §VII-A, §IX.
-  (2016) Robotic grasping and manipulation competition: future tasks to support the development of assembly robotics. In Robotic Grasping and Manipulation Challenge, pp. 190–200. Cited by: §X.
-  (2000) The multivariate l1-median and associated data depth. Proceedings of the National Academy of Sciences. Cited by: §XII-A.
Neural kinematic networks for unsupervised motion retargetting.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8639–8648. Cited by: §VII-A.
-  (2009) Real-time hand-tracking with a color glove. ACM ToG. Cited by: §II, §VI.
-  (2017) BigHand2.2m benchmark: hand pose dataset and state of the art analysis. In CVPR, Cited by: §I, §II, §VI.
-  (2019) Making convolutional networks shift-invariant again. ICML. Cited by: §XII-A, Table III.
Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, pp. 1–8. Cited by: §I.
-  (2019) Shared human–robot proportional control of a dexterous myoelectric prosthesis. Nature Machine Intelligence. Cited by: §I, §II.
Xii-a GloveNet: Hand Tracking with Colour Glove
Hand tracking with glove is done via keypoint detection with neural networks. The user wears a black glove with coloured blobs and moves the hand on a table covered with black cloth i.e. the scene is instrumented in a way that aids hand tracking. Since the colours are distinct and that most of the background is black, we can use OpenCV HSV colour thresholding to generate annotations for these coloured blobs. The HSV thresholds vary with the time of the day and therefore we collect data across days to build a big dataset of 50K images. We use a neural network to fit this data which makes the whole process robust to lighting changes and bad annotations and avoids the burden on the user to find the appropriate thresholds at test time. The network, called GloveNet, uses 4 layers of ResNet-50  with spatial-softmax at the end to regress to the 2D locations of finger-tips. We choose the recently proposed anti-aliased ResNet-50 from  for accurate and consistent predictions. We explain various stages of the pipeline below.
We use imgaug 
and apply various data augmentations while training. Additionally, because we want to focus on the hand moving on the table, for each training image we set the colour values of all pixels with depth beyond a threshold to zero. At training time, we either fill these zeroed out values with any of the colours on the glove or leave the image unchanged based on random number generator. We also replace these zeroed out values random noise based on some probability. All this ensures that the network learns to ignore the colours in the background that look similar to the colours on the glove.
Confidence Estimation of Predictions
We also obtain confidence for each predicted finger-tip location using test-time augmentation (TTA). We generate new images by shifting the original image by random shifts and pass them all through the network in one batch. We then subtract the applied random shifts from the predicted positions for each image to bring them into the reference frame of the original image and average them out to obtain the mean and standard deviation. We use the standard deviation as our confidence measure. We also use this to clean up the ground truth data that is noisy.
At test time, we generate four randomly shifted images per camera image and a combined total of 16 images from all four cameras. We compute the predicted finger-tip locations and their confidence measures and discard those that have low confidence. Of the confident ones we compute the euclidean distances between them and the previous finger-tip locations and turn them into probabilities via softmax:
We then push the predicted locations that have probability in a rolling buffer and compute the geometric median  to obtain the final predicted location of the finger-tip in 3D. The hyper-parameter .
|layer name||output size||parameters|
|conv1||160120||77, 64, stride 2|
3 max pool, stride 2
We found that while the predictions of the blobs at the back of the palm were stable, the predictions of finger-tips blobs tended to be quite inconsistent across time. Since the annotations were generated by computing the center of mass (CoM) of the segmented blob using the HSV colour thresholding in OpenCV, the CoM of the finger-tip were somewhat inconsistent across frames due to occlusions. Therefore, we relied only on the hand pose estimate provided by the blobs at the back of the palm.
Xii-B Architecture for Hand Pose Estimation with PointNet++
The PointNet++ implementation we used in this paper is from https://github.com/sshaoshuai/Pointnet2.PyTorch.
|layer name||mlp features||radius||num points|
|SA||[3, 64, 64, 128]||0.2||2048|
|SA||[128, 128, 128, 256]||0.4||1024|
|SA||[256, 128, 128, 256]||0.8||512|
|SA||[256, 128, 128, 256]||1.2||256|
|FP||[256+256, 256, 256]||512|
|FP||[256+256, 256, 256]||1024|
|FP||[256+128, 256, 256]||2048|
|FP||[256+3, 256, 256]||8192|
. The set abstraction layer sub-samples the points while the feature propagation layer interpolates features at a higher resolution.
A set abstraction level takes as input of points with -dim coordinates and -dim point feature. It outputs tensor of where sub-sampled points with -dim coordinates and new -dim feature vectors summarise local context.
In a feature propagation level, point features are propagated from points to points where and (with ) are point set size of input and output of set abstraction level . It is achieved by interpolating feature values of points at coordinates of the points. The interpolated features on points are then concatenated with skip linked point features from the set abstraction level.
Predicting Keypoint Locations
The backbone of the hand pose estimation is PointNet++ architecture which returns an embedding, , of size . Different MLPs are used to map this embedding to the corresponding desired outputs.
|mlp_layer1||[256, 256, 256]|
The voting layer obtains the relative positions of the 23 keypoints with respect to each point. The seg layer obtains the masks for each keypoint i.e. the neighbourhood of points that contribute to the location of a keypoint. The HandSeg layer segments hand from the background. We use Euclidean losses for both voting as well as Keypoints while a sigmoid cross-entropy is used for HandSeg.
Xii-C Architecture for JointNet
The 233 keypoint locations are unrolled to a 69-dimensional vector before feeding to the JointNet which returns a 20-dimensional vector of joint angles. Of all the hand-designed architectures we tried, we found this particular architecture to be an optimal trade-off between accuracy and efficiency.
The JointNet architecture is comprised of three layers. The layers linear1 and linear2 also use BatchNorm1d and ReLU.
Xii-D Completion Times Over 5 Consecutive Trials
We show the completion times for the 5 consecutive trails for each of the tasks. The failed trial is denoted by F. Note that the for most of the trials, the pilot only used 3-4 training trails to warm up. These 5 consecutive trails allow for testing both the ability to carry out a certain task without getting tired as well as showcasing that the tracking works without failures. Admittedly, the performace can vary depending on the pilot and how they are feeling on a given day but our experiments have revealed that the performance is in general quite similar.
|Task||Pilots||Completion Times for 5 Consecutive Trials(s)||Mean||Std.|
|Pick and Place: Brick||Pilot 2||22||22||19||16||14||19||3.57|
|Pick and Place: Spam||Pilot 2||23||23||28||29||20||25||3.78|
|Card Sliding||Pilot 2||18||12||18||15||17||16||2.54|
|Pick and Place: Pringles||Pilot 2||25||53||29||36||63||41||16.22|
|Brick Gaiting||Pilot 2||37||44||F||F||28||36||8.02|
|Box Flip||Pilot 2||174||26||90||30||67||77||60.18|
|Blocks (L)||Pilot 2||53||93||79||43||61||66||20.12|
|Peanut Jar||Pilot 2||68||105||84||87||57||80||18.45|
|Cup Insertion||Pilot 2||125||F||124||124||112||121||6.18|
|Blocks (M)||Pilot 2||99||48||82||75||152||91||38.63|
|Blocks (S)||Pilot 2||113||89||69||117||67||91||23.57|