I Introduction††footnotetext: This work was supported by the Toyota Research Institute (TRI) and the Amazon Research Awards (ARA). This article solely reflects the opinions and conclusions of its authors and not of TRI and Amazon.
This paper studies the use of tactile sensing for dexterous robotic manipulation, or tactile dexterity. Despite the evidence that humans heavily depend on the sense of touch to manipulate objects , robots still rely mostly on visual feedback. The vision-based approach has been effective for tasks such as pick-and-place  but it presents fundamental limitations to accomplish dexterous manipulation tasks that depend on more accurate and controlled contact interactions, such as object reorientation, object insertion, or almost any kind of object use.
The dynamics of manipulating an object are driven primarily by the relative motions and forces at the interfaces where the object contacts the end-effectors and the environment. With this knowledge, we believe that tactile sensors, with their ability to localize contact geometry, detect contact motion, and infer contact forces, should be at the center of our manipulation plans.
Our approach to tactile dexterity, described in Sec. III, is based on planning for robot/object interactions that render interpretable tactile information for control. At its core, this is an approach to robotic manipulation that puts tactile feedback at the center, to bypass some of its common caveats. Tactile information is by nature local and in general is not sufficient to fully describe the state of a manipulated system . Furthermore, the design of feedback policies for systems undergoing physical contact is challenging, even under the assumption of full state feedback [4, 5].
In that view, the key question becomes: How do we structure and guide manipulation planning so that tactile feedback is not only relevant, but also convenient? We propose to do it by:
Restricting contact interactions to a set of manipulation primitives that: 1) Target contacts with the object on geometric-rich features (for estimation); and 2) Define dynamic systems with simple mechanics and efficient closed-loop manipulation policies (for control). We describe four such primitives in Sec. IV.
Dividing the role of tactile control into: 1) Controlling the state of the contacts between end-effector and object; and 2) Controlling the state of the object in its environment. We describe a formulation for both in Sec. V.
We present a first study of this approach in the scenario of a dual-arm robot equipped with high-resolution tactile-sensing palms (based on ) as end-effectors, and tasked with manipulating an object on a table-top from an arbitrary initial pose to an arbitrary target pose. An offline graph-search task planner, described in Sec. VI, sequences the manipulation primitives, which are then executed in a closed-loop fashion by the robot. The focus of the experiments described in Sec. VII is evaluating the robustness of the system to external perturbations and to uncertainty in the initial pose of the object.
Ii Related Work
This section outlines previous relevant research in exploiting tactile sensing for manipulation control.
Slip Control Veiga  learns a slip detector that predicts slippage of an object within the robot hand. This model is leveraged by  to maintain a stable object grasp under externally applied perturbations on a multi fingered robot hand, where each finger acts independently to enforce sticking interactions. Dong  develops an incipient slip detection algorithm with a vision-based tactile sensor, GelSlim , and uses it to design a closed-loop tactile controller maintaining stable grasps in a bottle-cap screwing experiment. Both studies focus on controlling stick/slip interactions but do not explicitly control the trajectory of the manipulated object.
Pose Control Tian 
trains a deep convolutional neural network to predict the motion of a ball rolled on the ground with a tactile finger, directly in tactile space. This model is controlled using sample-based MPC. A drawback of this approach is its need for large quantities of real-world data, which would be challenging to collect for the rich palm/objects interactions considered in this paper.
Li  shows that localized object features can be exploited with tactile sensing to recover an accurate estimate of the its pose. This strategy has been showed effective at performing challenging manipulation tasks such as part insertion with small tolerances. Izatt  fuses tactile and visual perception by interpreting tactile imprints as local13] develops a tactile based pose estimation algorithm that exploits a high resolution tactile map of the object to localize tactile imprints.
Dexterous Manipulation Erdman  first studied the use of robotic palms as a way to achieve richer manipulation skills by exploiting the mobility of robotic manipulators. Researchers have investigated robotic palms for scooping [15, 16], tilting , grasping [18, 19], and collaborative manipulation [20, 21]. This paper draws inspiration from [22, 23, 24] by structuring complex manipulation behavior as a combination of simpler manipulation primitives.
This paper develops closed-loop tactile controllers enabling robust manipulation behavior, where the robot can react to external object perturbations. We propose an approach that divides the role of tactile control into:
Contact state control. Enforce a desired contact formation (contact/no-contact, stick/slip) between end effector and object.
Object state control. Control the motion of the object using a contact based state estimator (tactile localization).
This paper considers the scenario of manipulating an object from an initial pose to a target pose on a table top. We consider a robotic platform that is 1) dexterous, where both robotic palms can be controlled independently, and 2) tactile sensorized, where each palm is equipped with high-resolution tactile sensing (based on GelSlim technology ). We formulate the manipulation problem as a sequencing of manipulation primitives, where each primitive is designed to have a prescribed contact interaction between the robotic palms and the object. Fig. 2 shows the four manipulation primitive considered in this paper. This structuring of the manipulation task into simpler behaviors gives us the freedom to design interactions for which we understand the mechanics, are able to interpret the tactile information, and can develop effective planning algorithms.
Iv Mechanics of Manipulation Primitives
This section describes the mechanics of four manipulation primitives: grasp, pull, pivot, and push (shown in Fig. 2). For each primitive, the interactions between the palms, the object, and the environment are modeled assuming:
Known geometry of object, robot, and environment
Known coefficients of friction
Coulomb’s frictional law
Surface contact interactions with uniform pressure distribution
These simplifications help to design computationally fast trajectory planning algorithms in Section VI. The unmodeled elements of the interactions (non-uniform pressure distributions, inertial forces, deformation of contact, etc.) are addressed by designing closed-loop tactile controllers in Section V that can quickly react to undesirable slippage.
Assuming quasi-static interactions, force equilibrium dictates that contact forces on the object with the end-effector or environment are balanced by:
where is the concatenation of the object pose and the left/right palm poses, is the applied wrench on the object by the contact in the contact frame, is the external force applied by gravity in the world frame, is a grasp matrix transforming the coordinates of a contact wrench from the contact frame to the world frame , and is the number of contacts. Examples of more in detail description of the mechanics of similar manipulation primitives can be found in  for non-prehensile pushing,  for prehensile pushing, and  for pivoting.
The contact forces are constrained to remain within the friction cone in accordance to Coulomb’s frictional law. Denoting the normal and tangential components of the contact force as , we express Coulomb’s frictional law as:
In the case of point contact interactions (grasp and pivot
), the contact is unable to sustain frictional moments, implying. For contacts modeled using surface contacts (push and pull), the surface is able to resist a certain amount of frictional moment. We model surface contacts using the limit surface , which describes the set of forces and moments that can be transmitted through the contact interaction. In practice, we make use of the ellipsoidal approximation to the limit surface introduced in  that analytically represents the limit surface as an ellipsoid.
V Tactile Control
In this section, we describe tactile-based controllers that give robust behavior to the manipulation primitives. We use GelSlim , an optical-based tactile sensor that renders high resolution images of the contact surface geometry and strain field, as shown in Fig. 5.
A primary goal of the tactile controller is to enforce a desired contact state between the palms and the object. For example, during a pivot maneouver in Fig. 2, we want both palms to maintain sticking contacts with the corners of the object. By monitoring slippage, we design a controller that regulates the applied forces on the object to prevent further slip. We refer to this as Contact State Control (Sec. V-A). This is shown in Fig. 3, where the contact state controller rotates the configurations of the palms and applies additional normal force on the object in reaction to a slip event.
An important consequence of undesired slippage is that the position of the object has deviated from the the nominal trajectory. To address this, we make use of the controllers, shown in Fig. 5, that tracks in real-time local features on the object to continuously replan robot motions for the manipulation primitive. We refer to this as Object State Control (Sec. V-B)
V-a Contact State Control
Each primitive assumes a particular contact formation between the palms and the object. This assumption is likely to be broken as unmodeled perturbations are applied on the system and cause undesired slippage. We design a contact state controller that acts to enforce the planned contact modes by reacting to the binary incipient slip signal at contact , based on .
Coulomb friction states that slippage occurs when the contact force lies on the boundary of the friction cone, as shown in Fig. 4. Given an undesired slippage signal, we find local robot adjustments leading to a more stable sticking solution (i.e. contact forces nearer to the center of the friction cone). The stability margin in Fig. 4 is defined as the shortest distance from the contact force to the friction cone boundary and quantifies how close to slipping is a particular contact. The goal of this controller is to find robot adjustments that either maximize (for sticking) or minimize (for sliding) the stability margin of a contact to enforce a desired contact mode.
Given the slippage signal at each contact and the current robot pose configuration , we search for a robot palm pose adjustment solving:
with , , , defined in Section IV and where is the stability margin shown in Fig. 4. The notation is abused here for simplicity. We ensure that satisfies group constraints and does not violate the kinematics of the contact formation. In practice, this will define a different space of possible robot adjustments for each primitive. For example, in Fig. 2
, the palms can press harder and/or pivot about the contact edge. The hyperparameteris a weight chosen to enforce the desired behavior at the contact based on the slip signal . For example, following a slippage event at contact , the controller would increase .
The optimization program in (4) is non-convex due to the nonlinear nature of and the bilinear constraint associated with static equilibrium. In , the surrogate stability margin illustrated in Fig. 4 is proposed as a convex approximation to . In (5), we reformulate the optimization program in (4) using the surrogate margin and linearizing the static equilibrium Eq. (1) about the current robot configuration and contact forces , for computational efficiency. The linearized contact state controller becomes:
where the symbol is used to evaluate a term at the nominal configuration. The optimization program in (5) then takes the form of a convex quadratic program under a polyhedral approximation to the friction cone .
V-B Object State Control
The contact state controller in Section V-A regulates the applied forces on the object to enforce the desired contact mode. This local controller reacts to fight against external perturbations but does not have the ability to change the desired trajectory of the object in response to the perturbation. To address this, we design an object state controller running in parallel tasked with replanning object/palm trajectories to drive the object to its target location.
In this paper we track two types of features: points (corners of the object) and lines (edges of the object). We formulate the tactile object state estimator as an optimization program updating the pose of the object to satisfy the geometric constraints associated with the tactile features, as shown in Fig. 5. We quantify the error between two poses using the distance , where is defined as the weighted sum of the Euclidean metric in and the great circle angle metric in for the respective components . We enforce that detected lines are collinear with their associated edge on the object mesh and that the sensed points are coincident with the object corner. In addition to the detected geometric constraint, We constrain the estimated object pose to satisfy the geometric constraints consistent with the current manipulation primitive. For example, for a pull primitive, we enforces that the bottom surface of the object is in contact with the table top.
The estimated object pose is used to update the nominal robot palm pose trajectory and allows the robot to adapt to local object perturbations. This is further described in Sec. VI-B.
We formulate the manipulation planning problem as a sequencing of manipulation primitives, as described in Section IV. This section presents how to 1) sequence the manipulation primitives to achieve the task and 2) how to plan robot trajectories within a manipulation primitive to achieve a given object transformation.
Vi-a Search for Primitive Sequence
We formulate the search for the primitive sequence as a graph search problem. We adapt the regrasp graphs developed in [33, 27] to include a broader set of manipulation primitives. The nodes of the manipulation graph represent the stable placements of the object and the edges represent manipulation primitive actions transforming the object from a stable placement to another. We search for the shortest path within the constructed path achieving the desired pose to pose reconfiguration using Dijkstra’s algorithm. In the experiment shown in the title figure, the primitive sequence pullpivotpush is found to achieve the desired object transformations. We refer the readers to  for more details regarding the graph search planning framework.
Vi-B Trajectory Planning
Once the sequence of primitives is found, we compute the robot/object pose trajectories for each primitive individually. We design primitive specific planners returning a robot/object pose trajectory satisfying a desired object reconfiguration. We require that the planner can regenerate trajectories at fast frequencies to exploit the tactile object state estimator developed in Section V-B.
For grasp and pull
, we plan for object motions that kinematically stick to the end effectors. For these, we interpolate between the initial and final poses of the object to determine the palm pose trajectory. Forpivot, we compute an interpolated object trajectory between its initial and final poses about a specified center of rotation. Solving (1), we find the a palm pose relative to the object that maintains a sticking interaction at all contacts. Finally, for push, we employ a Dubins’ car planner that computes the time-optimal trajectory relating the initial and final object configurations via a single push . We consider pushes in all sides of the object and execute the trajectory with the shortest path.
We validate our approach to tactile dexterity on an ABB YuMi dual-arm robot where we evaluate the ability of the tactile controller to handle external perturbations on a table top manipulation task.
The title figure shows snapshots of an experiment where the robot manipulates an object from the initial pose to the target pose . To achieve the task, the robot follows the sequence: pull the object to the middle of the table, pivot the object to its target placement, and push it to its target location. The initial pull primitive is necessary to move the object to a location that allows the robot to perform a pivot manoeuvre with well defined inverse kinematics and that avoids collisions with the environment.
In Fig. 6, we evaluate the closed-loop performance of the tactile controller presented in Section V by quantifying its ability to handle external object perturbations during execution. We consider the regulation task of maintaining the object in a stationary pose for the pull and grasp primitives. The regulation task allows to better visualize the reactive capabilities of the controller without any loss of generality. In each experiment, we apply two successive impulsive forces on the object and evaluate the stabilizing capabilities of the tactile controller. Figure 6
plots the mean and standard deviation of the error between the object’s desired and measured pose forconsecutive experiments. In both cases, the controller quickly reacts to the disturbance by 1) detecting slippage events at the contact interfaces and 2) tracking the pose of the object using the detected object edge in the tactile signal. First, the applied normal force is increased following (5) in reaction to the detected slippage at the contact interface. Second, the robot replans its trajectory in real-time using the tactile state estimate such that the object quickly returns to its nominal pose. During this experiment, an apriltag is used to provide the ground truth pose of the object.
This paper introduces tactile dexterity, an approach to dexterous manipulation that exploits tactile sensing for reactive control. By structuring the manipulation problem as a sequence of manipulation primitives that render interpretable tactile information, we enable tactile object state estimation, tactile object control, and robust manipulation behavior. This requires restricting the types of primitive interactions so they yield simple rigid body mechanics models and efficient planners. We show that the developed tactile controllers modulate grasp forces, track the pose of the object, and handle external perturbations by replanning in real-time.
This research relies on a number of important assumptions about the structure of the problem:
Manipulation primitives. We use human intuition to develop a library of manipulation primitives that are amenable for planning and control (for controllability).
Tactile Pose Estimation. We assume that the pose of the object can be reconstructed from tactile observations (for observability).
Known environment. The manipulation primitives and planning are tailored to a flat surface environment.
These assumptions facilitate the development of state estimation, planning, and control frameworks capable of exploiting tactile sensing for dexterous manipulation. They constitute both the limitations, as well as strengths of the approach.
We believe that there are natural extensions to generalize this approach to dexterous manipulation that puts tactile feedback at the center.
General Manipulation Primitives. Developing manipulation primitives with richer contact interactions (contact switches, stick/slip interactions, etc.) will allow more expressive behavior. Due to increased complexity of the mechanics of these primitives, they will likely benefit even more from real-time tactile observations of contact. This line of research will requires controller architectures that can reason across hybrid contact switches.
Another interesting possibility currently in vogue is to learn manipulation primitives from experience, and defining a primitive more generally as a behavior for which the robot can learn a stable controller. This would potentially reduce the need for human intuition.
Object tracking. A formal approach to characterize object observability directly from tactile feedback could assist with identify discriminative features and grasps of the object. Combining touch with vision is a natural extension of this research that would allow the robot to recover from larger object disturbances.
Task-and-Motion Planning. To deal with more complex environments and tasks (e.g., pick up the box and put it on a shelf, or inside a closed drawer), we need to integrate the system with a high level task planner that can sequence manipulation primitives each with its own reachability and preconditions. Ideally, these planners would be reactive and allow the robot to switch primitives in response to unexpected deviations.
-  Johansson, R.S., Westling, G.: Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental brain research 56(3) (1984) 550–564
-  Correll, N., Bekris, K.E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J.M., Wurman, P.R.: Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering 15(1) (2016) 172–188
-  Koval, M.C., Pollard, N.S., Srinivasa, S.S.: Pose estimation for planar contact manipulation with manifold particle filters. The International Journal of Robotics Research 34(7) (2015) 922–945
-  Woodruff, J.Z., Lynch, K.M.: Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2017) 4066–4073
-  Hogan, F.R., Grau, E.R., Rodriguez, A.: Reactive planar manipulation with convex hybrid mpc. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2018) 247–253
-  Donlon, E., Dong, S., Liu, M., Li, J., Adelson, E., Rodriguez, A.: Gelslim: A high-resolution, compact, robust, and calibrated tactile-sensing finger. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2018) 1927–1934
-  Veiga, F., Van Hoof, H., Peters, J., Hermans, T.: Stabilizing novel objects by learning to predict tactile slip. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2015) 5065–5072
-  Veiga, F., Edin, B.B., Peters, J.: In-hand object stabilization by independent finger control. arXiv preprint arXiv:1806.05031 (2018)
-  Dong, S., Ma, D., Donlon, E., Rodriguez, A.: Maintaining grasps within slipping bounds by monitoring incipient slip. In: 2019 International Conference on Robotics and Automation (ICRA). (May 2019) 3818–3824
-  Tian, S., Ebert, F., Jayaraman, D., Mudigonda, M., Finn, C., Calandra, R., Levine, S.: Manipulation by feel: Touch-based control with deep predictive models. arXiv preprint arXiv:1903.04128 (2019)
-  Li, R., Platt, R., Yuan, W., ten Pas, A., Roscup, N., Srinivasan, M.A., Adelson, E.: Localization and manipulation of small parts using gelsight tactile sensing. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE (2014) 3988–3993
-  Izatt, G., Mirano, G., Adelson, E., Tedrake, R.: Tracking objects with point clouds from vision and touch. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2017) 4000–4007
-  Bauza, M., Canal, O., Rodriguez, A.: Tactile mapping and localization from high-resolution tactile imprints. arXiv preprint arXiv:1904.10944 (2019)
-  Erdmann, M.: An exploration of nonprehensile two-palm manipulation. The International Journal of Robotics Research 17(5) (1998) 485–503
-  Trinkle, J.C., Abel, J.M., Paul, R.P.: An investigation of frictionless enveloping grasping in the plane. The International journal of robotics research 7(3) (1988) 33–51
-  Trinkle, J.C.: On the stability and instantaneous velocity of grasped frictionless objects. IEEE Transactions on Robotics and Automation 8(5) (1992) 560–572
-  Erdmann, M.A., Mason, M.T.: An exploration of sensorless manipulation. IEEE Journal on Robotics and Automation 4(4) (1988) 369–379
-  Paljug, E., Yun, X.: Experimental results of two robot arms manipulating large objects. In: Robotics and Automation, 1993. Proceedings., 1993 IEEE International Conference on, IEEE (1993) 517–522
-  Yun, X.: Object handling using two arms without grasping. The International journal of robotics research 12(1) (1993) 99–106
-  Bicchi, A., Melchiorri, C., Balluchi, D.: On the mobility and manipulability of general multiple limb robots. IEEE Transactions on Robotics and Automation 11(2) (1995) 215–228
-  Bicchi, A., Sorrentino, R., et al.: Dexterous manipulation through rolling. In: ICRA. Volume 95. (1995) 452–457
Kaelbling, L.P., Lozano-Pérez, T.:
Hierarchical planning in the now.
In: Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence. (2010)
-  Woodruff, J., Lynch, K.: Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks. In: Robotics and Automation (ICRA), 2017 IEEE International Conference on. (2017)
-  Toussaint, M., Allen, K., Smith, K.A., Tenenbaum, J.B.: Differentiable physics and stable modes for tool-use and manipulation planning. In: Robotics: Science and Systems. (2018)
-  Siciliano, B., Khatib, O.: Springer handbook of robotics. Springer (2016)
-  Chavan-Dafle, N., Holladay, R., Rodriguez, A.: In-hand manipulation via motion cones. arXiv preprint arXiv:1810.00219 (2018)
-  Hou, Y., Jia, Z., Mason, M.T.: Fast planning for 3d any-pose-reorienting using pivoting. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2018) 1631–1638
-  Goyal, S., Ruina, A., Papadopoulos, J.: Planar sliding with dry friction part 1. limit surface and moment function. Wear 143(2) (1991) 307–330
-  Howe, R.D., Cutkosky, M.R.: Practical force-motion models for sliding manipulation. The International Journal of Robotics Research 15(6) (1996) 557–572
-  Aceituno-Cabezas, B., Mastalli, C., Dai, H., Focchi, M., Radulescu, A., Caldwell, D.G., Cappelletto, J., Grieco, J.C., Fernández-López, G., Semini, C.: Simultaneous contact, gait, and motion planning for robust multilegged locomotion via mixed-integer convex optimization. IEEE Robotics and Automation Letters 3(3) (2017) 2531–2538
-  Stewart, D.E., Trinkle, J.C.: An implicit time-stepping scheme for rigid body dynamics with inelastic collisions and coulomb friction. International Journal for Numerical Methods in Engineering 39(15) (1996) 2673–2691
-  Holladay, R.M., Srinivasa, S.S.: Distance metrics and algorithms for task space path optimization. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2016) 5533–5540
-  Wan, W., Mason, M.T., Fukui, R., Kuniyoshi, Y.: Improving regrasp algorithms to analyze the utility of work surfaces in a workcell. In: Robotics and Automation (ICRA), 2015 IEEE International Conference on, IEEE (2015) 4326–4333
-  Zhou, J., Hou, Y., Mason, M.T.: Pushing revisited: Differential flatness, trajectory planning, and stabilization. The International Journal of Robotics Research (2017) 0278364919872532