 # In-Hand Manipulation via Motion Cones

In this paper, we present the mechanics and algorithms to compute the set of feasible motions of an object pushed in a plane. This set is known as the motion cone and was previously described for non-prehensile manipulation tasks in the horizontal plane. We generalize its geometric construction to a broader set of planar tasks, where external forces such as gravity influence the dynamics of pushing, and prehensile tasks, where there are complex interactions between the gripper, object, and pusher. We show that the motion cone is defined by a set of low-curvature surfaces and provide a polyhedral cone approximation to it. We verify its validity with 2000 pushing experiments recorded with motion tracking system. Motion cones abstract the algebra involved in simulating frictional pushing by providing bounds on the set of feasible motions and by characterizing which pushes will stick or slip. We demonstrate their use for the dynamic propagation step in a sampling-based planning algorithm for in-hand manipulation. The planner generates trajectories that involve sequences of continuous pushes with 5-1000x speed improvements to equivalent algorithms. Video Summary -- https://youtu.be/tVDO8QMuYhc

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## I Introduction

A motion cone is the set of feasible motions that a rigid body can follow under the action of a frictional push. We can think of it as a geometric representation of the underactuation inherent to frictional contacts. Since contacts can only push, and since friction is limited, a contact can move an object only along a limited set of rays. The concept was introduced by Mason  for point contacts in the context of a planar horizontal pushing task.

Motion cones abstract the algebra involved in simulating frictional contact dynamics. A contact force on the inside (or boundary) of the friction cone produces sticking (or slipping) behavior, and leads to motion rays on the inside (or boundary) of the motion cone. Lynch and Mason  generalized the construction of motion cones to line contacts in a horizontal plane. They used them to plan stable pushing trajectories without the complexities of standard complementarity formulations of contact dynamics [29, 24, 2]. Motion cones have since been the basis of several efficient planning and control strategies for planar manipulations on a horizontal support surface [9, 5, 11, 34, 12].

This paper studies the construction of motion cones in a more general set of planar tasks. In particular, we highlight the case of prehensile manipulation in the vertical plane. In general planar tasks, external forces other than the pusher force (e.g., gravity) can alter the dynamics of contact interactions between the pusher, object, and gripper/support-plane. Motion cones efficiently capture the intricate mechanics of these tasks for simulation, planning, and control.

We present three main contributions:

• Mechanics of motion cones for planar tasks in the gravity plane. We show that the motion cone is defined by a set of low-curvature surfaces, intersecting at a point and pairwise in lines. We propose a polyhedral approximation to the motion cone for efficient computation.

• Experimental validation of the stick/slip condition of motion cones in a prehensile pushing task instrumented with a Vicon motion tracker.

• Application of motion cones in a sampling-based planning framework for in-hand manipulation using prehensile pushes (see Fig. 1). We show this yields significant speed improvements with respect to our prior work [2, 3]. Fig. 1: (top) Example friction cone and motion cone of an object moving in the vertical plane. The pusher can move the object along any direction [Vx,Vz,ωy] inside the motion cone. (bottom) A plan via motion cones. Motion cones capture local reachability. A path in the tree of motion cones generates a pushing strategy to move an object. Fig. 2: Manipulating a T-shaped object in a parallel-jaw grasp by pushing it against features in the environment. The manipulation is shown from a side view.

Fig. 2 shows an example of a pushing trajectory planned to change the grasp on a T-shaped object. The resulting trajectory is a sequence of continuous stable pushes, each where the object sticks to a particular external pusher. The proposed planning algorithm obtains these trajectories consistently in less than a second.

Section III reviews the mechanics of pushing in an arbitrary plane, while Section IV describes the process to construct the corresponding motion cones. In Section V we discuss the computational aspects in the calculation and approximation of motion cones for pushing an object gripped with a finite force. In Section VI we demonstrate a sampling-based planning approach that exploits motion cones to speed up the computation of local contact dynamics for prehensile pushing.

The generalization of motion cones to interactions with gravity opens a door for efficient and robust planning of in-hand manipulations that respect and exploit the basic principles of frictional rigid-body contact interactions: Newton’s second law, Coulomb’s friction law, the principle of Maximal Dissipation, and the rigidity of rigid-bodies.

## Ii Related Work

Planning and control through contact is a central topic in robotic manipulation research. Rigid-body contact is modeled as a series of constraints on the possible motions and forces at contact. Simplifying and exploiting these constraints is a pivotal theme in the non-prehensile manipulation literature. Goyal  introduced the concept of a limit surface, a compact mapping from the friction wrench between slider and ground and the sliding twist at contact. Mason  studied the mechanics of pushing and proposed the concept of the motion cone. These two fundamental geometric constructions provide direct force-motion mappings for contact interactions and have facilitated efficient planning and control techniques in non-prehensile manipulation [20, 21, 5, 6, 11, 34].

Recent work on trajectory optimization and manipulation planning shows that it is possible to reason about different contact modes and plan trajectories through contact in a standard rigid body dynamics framework based on complementarity constraints. These methods often have to compromise on the computational efficiency [24, 2] or the realism of contact dynamics [19, 31, 18].

In contrast, some more recent work has focused on particular types of manipulation primitives and exploits assumptions in the problem formulation to develop fast planning and robust control strategies. Shi et al.  demonstrates dynamic in-hand manipulation planning in a parallel-jaw grasp. With a pre-defined contact mode sequence at the fingers and a limit surface approximation for the force-motion interaction at the fingers, they derive a control law that can move the object to the goal pose in the grasp. Similar approaches are explored for planning and controlling in-hand manipulations by actively using gravity  or dynamic motions [13, 28, 14].

Sundaralingam and Hermans  propose a trajectory-optimization based, purely-kinematic approach for in-hand manipulation with a multi-finger gripper. They assume that the fingers on the object do not slip and impose soft constraints that in practice minimize the slip at the fingers. By assuming all finger contacts to be sticking, they bypass the need for modelling the dynamics of the manipulations and obtain kinematic plans quickly.

In our recent work , we present a fast sampling-based planning framework for in-hand manipulations with prehensile pushes, where the pusher contact is guaranteed to stick to the object using the mechanics of the manipulations. The plans are discrete sequences of continuous pushes that respect friction, contact, and rigid-body constraints.

These promising results on in-hand manipulation research, which limit to certain types of manipulation primitives for efficient planing and control, motivate us to generalize the concept of motion cones to more general pushing tasks. Motion cones, as a set of direct constraints on the object motion or control inputs, can naturally fit well in a trajectory optimization framework. As we illustrate in this paper, for sampling-based methods, the motion cone can be used to guide sampling, and for fast dynamics propagation.

## Iii Mechanics of Pushing in a Plane Fig. 3: Pushing an object (a) on a horizontal surface, (b) on an inclined surface, (c) in a grasp in the gravity plane, and (d) in a grasp in a tilted plane

Fig. 3 shows four different cases of planar pushing. In case (a), the pusher force is the only external force on the object in the plane of motion. However, in the rest of the cases, a component of gravity is also present. The concept of the motion cone as originally studied in   is limited to the case (a) in Fig. 3. Our extension of the motion cone to a general planar case is valid for all the cases in Fig. 3.

In this section we discuss the mechanics of pushing an object in a plane. First we will review the fundamental concepts, namely, the limit surface , and the generalized friction cone  that will help us model the friction interaction at the contacts involved in pushing.

### Iii-a Limit Surface

The limit surface is a common approach to model a friction interaction between an object and a support contact. In case (a) and (b) in Fig. 3, the surface on which the object rests is the support contact. In case (c) and (d), the finger contacts play the same role.

Goyal  defined the boundary of the set of all possible friction wrenches that a contact can offer as the Limit Surface. Howe and Cutkosky , Xydas and Kao  showed that an ellipsoidal approximation allows for a simpler representation of the limit surface geometry. In this paper we will assume an ellipsoidal approximation of the limit surface, which has been shown to be computationally efficient for simulating and planning pushing motions [21, 6, 25, 34].

Let be a frictional wrench on the object from the support contact in the contact frame. A mathematical representation of the ellipsoidal limit surface is given by , where . For isotropic friction, the maximum friction force is, , where is the friction coefficient between the contact and the object and is the normal force at the contact. The maximum friction torque about the contact normal is , where is the radius of the contact and an integration constant. For a uniform pressure distribution at the contact, is about 0.6 [33, 25].

When the object slides on the support contact, the friction wrench between the object and the contact () intersects the limit surface. Based on the maximal energy dissipation principle, the normal to the limit surface at the intersection point provides the direction of the twist of the object at the contact. Conversely, if the object twist () is known, we can find the friction wrench following   as,

 wc=A−1vobj\_c√vTobj\_cA−1vobj\_% c=μcN¯¯¯¯wc (1)

Here, is the velocity of the object in the contact frame and can be computed from as . The Jacobian maps the object velocity from the object frame to the support contact frame. is the unit wrench lying on a limit surface that is scaled by maximum linear friction available at contact () to produce the net frictional wrench.

Under the ellipsoidal limit surface model assumption, translational velocity of the object in the contact frame is always parallel and opposite to the linear frictional force applied by the contact in the contact frame . Moreover, the relationship between the friction wrench and the normal to the limit surface, which defines the motion direction, sets the following constraint between the angular velocity at the contact and the linear velocity:

 vx\_cωy\_c=(rc)2¯¯¯fx¯¯¯¯¯my   and  vz\_cωy\_c=(rc)2¯¯¯fz¯¯¯¯¯my (2)

Given the friction wrench on the object from the support contact, we can find the object velocity as:

 vobj=~k~JcB⋅¯¯¯¯wc , B=Diag(1,1,(rc)−2),  ~k∈IR+ (3)

Here, maps the object velocity from the support contact frame to the object frame.

### Iii-B Generalized Friction Cone

The friction between the pusher and the object can be modelled with the Coulomb friction law. Erdmann  introduced the concept of generalized friction cone () as a representation of the local friction cone at a contact in the object frame. The generalized friction cone for a pusher modelled with multiple point contacts is the convex hull of the generalized friction cones for each constituent contact .

 Wpusher={¯¯¯¯w% pusher=J⊤p⋅¯¯¯fp | ¯¯¯fp∈FCpusher} (4)

Here, is the Jacobian that maps the local contact forces () at the pusher to the object frame. is the unit wrench corresponding to unit force/s inside the friction cone/s at the pusher constituent contact/s.

Now, with the approaches for contact modelling set, we look into formulating the mechanics of pushing in a plane.

### Iii-C Mechanics of Pushing

The motion of the object in the plane of motion evolves following the net wrench acting on it. Under the quasi-static assumption, which is appropriate for slow pushing operations, the inertial forces on the object are negligible and there is force balance:

 wsupport+wpusher+mg=0 (5)

Here, (5) is written in the object frame located at the center of gravity. is the friction wrench provided by the support contact, is the wrench exerted by the pusher, is the mass of the object, is the gravitational component in the plane of motion. The Jacobian maps the support contact wrench from the contact frame (usually located at the center of pressure) of the support surface to the object frame. So, and (5) becomes:

 μcNJ⊤c⋅¯¯¯¯wc+J⊤p⋅fp+mg=0 (6)

### Iii-D Stable Pushing in a Plane

Lynch and Mason  studied pushing motions for which the pusher sticks to the object when pushing on a horizontal surface, which they referred to as Stable Pushing. For stable pushing, force at the pusher lies inside the friction cone in the local contact frame.

For a general planar case, such a condition for a stable push can be written as:

 μcNJ⊤c⋅¯¯¯¯wc+J⊤p⋅fp+mg=0 , fp∈FCpusher

For a given object motion to be possible with a stable push, the pusher needs to be able to provide a wrench that balances the net wrench produced by the friction wrench from the support contact and the gravitational force in the plane. Using (4) we can rewrite the previous equation as:

 −μcNJ⊤c⋅¯¯¯¯wc−mg=k¯¯¯¯wpusher¯¯¯¯wpusher∈Wpusher , k∈IR+ (7)

Here, is the magnitude of the pusher force. To know if an object motion can be achieved with a stable push, we simply need to check if the net required wrench falls inside the generalized friction cone of the pusher.

 −μcNJ⊤c⋅¯¯¯¯wc−mg∈Wpusher (8)

From a planning and control perspective, rather than querying if pushes are stables pushes or not, the bound on the set of object motions possible with stable pushes is more useful. This is in fact the motivation for the motion cone concept that we study in the next section.

## Iv Motion Cone for Planar Pushing

The motion cone is the set of objects motions that can be produced while keeping the pusher contact sticking. In an abstract sense, it is the motion equivalent of the generalized friction cone of the pusher.

Problem: Find the set of object motions for which the net required wrench can be balanced by a wrench inside the generalized friction cone of the pusher.

This is equivalent to finding a set of object motion for which constraint (7) holds true. Rewriting,

 −μcNJ⊤c⋅¯¯¯¯wc−mg=k¯¯¯¯wpusher, ¯¯¯¯wpusher∈Wpusher, k∈IR+
 J⊤c⋅¯¯¯¯wc=k−μcN¯¯¯¯wpusher+m−μcNg¯¯¯¯wpusher∈Wpusher , k∈IR+ (9)

Using (3) we can map the support contact wrench to the object velocity . Hence, to find a motion cone, we first find the set of support contact wrenches () that satisfy (9) and then map this wrench-set () to the set of object twists. We denote this object twist-set, i.e. the motion cone, by .

The presence of an external force, (such as the gravitational force) other than the pusher force, in the plane of motion complicates the mechanics and the structure of the motion cone. To explain this effect in detail, we will first consider the case of pushing an object on a horizontal surface where there is no such additional force.

### Iv-a Motion Cone for Pushing on a Horizontal Surface Fig. 4: To make a push inside the gravity-free motion cone stable in the gravity plane, the unit grasp wrench can be scaled such that the net pusher wrench required for the desired push falls inside/on the generalized friction cone of the pusher.

For the case of pushing on a horizontal plane, . For an object on a flat support surface with uniform pressure distribution on the support (as is Fig. 3- case (a) and (b)), the contact frame coincides with the object frame,

(identity matrix). Then we can write (

7) as:

 −μcN¯¯¯¯wc=k¯¯¯¯wpusher , ¯¯¯¯w% pusher∈Wpusher , k∈IR+
 ¯¯¯¯wc=−k¯¯¯¯wpusherμcN , ¯¯¯¯w% pusher∈Wpusher

The set of valid support contact wrenches is the negative of the generalized friction cone of the pusher, i.e, . By mapping through (3), we get the motion cone 111Mason  defined the motion cone in terms of pusher twists which is a linear Jacobian transform of the motion cone . However, we will keep it in the object twist space..

Note that for the case of pushing on a horizontal surface, and are convex polyhedral cones. Moreover, they are independent of the support normal force, i.e, the weight of the object , and friction at the support surface .

For a more general pushing tasks however, . From (9) we can see that, unlike for horizontal pushing, the system parameter () and force magnitudes ( and

) influence the direction vectors of the wrench-set

. In the next section we will focus on the case in Fig. 3-(c) – pushing an object in a parallel-jaw grasp in the plane of gravity. There the gravitational force is not zero in the plane of motion and the Jacobians and are not always identity matrices as the support (finger) contact location changes in the object frame as the object is pushed in the grasp. The case shown in Fig. 3-(b) is same as the case (c) except the Jacobians and are always identity matrices. The case shown in Fig. 3-(d) is same as the case (c) except that only a part of the gravitational force acts in the plane of motion. Fig. 5: A depiction of the process for constructing a wrench-set (~Wc). The intersection of the limit surface with the sum of the scaled generalized friction cone of the pusher and the gravitational wrench defines the wrench-set ~Wc.

### Iv-B Stable Pushing and Motion Cone in the Gravity Plane

For a case similar to Fig. 3-(c), but in a gravity-free world, we can exploit the simplification of the (9) by omitting the gravity term and compute a convex polyhedral motion cone similar to that in the horizontal pushing case, but while taking the non-identity Jacobians and into consideration. We will refer to this motion cone as a gravity-free motion cone in the later discussions.

From (8) we see that the gravitational wrench on the object pulls the net required wrench in or out of the generalized friction cone of the pusher. Some of the motions in the gravity-free motion cone may not be stable pushes in the gravity plane, while for some object motions outside the gravity-free motion cone, the gravitational force on the object can be exploited to make them stable pushes.

###### Theorem 1.

Any push inside the gravity-free motion cone can also be made a stable push in the gravity plane by increasing the grasping force above a minimum force threshold.

###### Proof.

For a motion inside a gravity-free motion cone, the support/grasp wrench direction lies inside the generalized friction cone of the pusher, i.e., . We can always find a magnitude () with which the support/grasp wrench direction needs to be scaled so that the net wrench (the vector sum of the gravitation wrench and the support/grasp wrench) is inside the generalized friction cone of the pusher. For a given , we can analytically find the bounds on needed to pull the net wrench inside the generalized friction cone of the pusher. Fig. 4 shows the graphical interpretation. ∎

Theoretically, we can make any motion in the gravity-free motion cone a stable motion in the gravity case by changing the grasping force. With increased grasping force, more motions in a gravity-free motion cone are stable in the gravity plane and become part of the motion cone. For infinite grasping force, the gravity-free motion cone coincides with the motion cone in the gravity plane. However, in practice, grippers have limited grasping force. We need to find the motion cone for a fixed grasping force. Fig. 6: A graphical illustration for the construction of the motion cone (~Vobj) from the wrench-set (~Wc). The motion cone is defined by the set of the surface normals to the limit surface where the wrench-set intersects the limit surface. We propose polyhedral approximation (¯¯¯¯Vobj) to the motion cone for computational efficiency.

## V Motion Cone in The Gravity Plane for Fixed Grasping Force

In this section we find the object motion cone in the gravity case for a given grasping force and friction parameters.

### V-a Analytical Computation

For a known , (9) is a set of three linear equalities with 4 unknowns, and . However, we know that is a unit wrench that satisfies the ellipsoidal limit surface constraint:

 ¯¯¯fx1+¯¯¯fz1+¯¯¯¯¯my(rc)2=1 (10)

Constraints (9) and (10) can be solved together analytically to find and . Specifically, after substituting and from (9) into (10), (10) becomes a quadratic equation in . Solving this quadratic equation with a constraint gives a unique solution for . Substituting this value for in (9) makes (9) a set of three linear equalities with three unknowns which can be solved for a unique solution.

For prehensile pushing in the gravity plane, the relationship between and is not linear as in the gravity-free case. To find wrench-set , we need to sweep over the boundary of and solve (9) and (10) iteratively. Fig. 5 is the depiction of the process involved in solving the constraints (9) and (10) together for computing the wrench cone . Fig. 6 shows the graphical representation of the computation in (3) that maps the wrench-set to the motion cone . Fig. 7: (top) The grasp-pusher configuration used for the experimental validation of the motion cone. (bottom) The analytically computed wrench cone (~Wc), motion cone (~Vobj), and polyhedral approximation to the motion cone (¯¯¯¯Vobj) for the above configuration and 45 N grasping force. Note that the surfaces defining the motion cone (~Vobj) are curved.

and are not polyhedral cones as in the gravity-free case, but rather can be best characterized as cones defined by low-curvature surfaces that intersect all in a point and pairwise in lines. Fig. 7 shows the wrench cone and motion cone computed analytically for one particular grasp-pusher configuration.

### V-B Polyhedral Approximation to the Motion Cone

As an object is pushed in a grasp, the position of finger contacts in the object frame change, and consequently and the motion cone also change. We need to compute the motion cone iteratively as the object moves in the grasp.

As the boundary surfaces of the motion cone have low curvatures, we propose a polyhedral approximation () of the motion cone for its efficient computation. Each edge of the polyhedral approximation of the motion cone is the object motion corresponding to each edge of the generalized friction cone of the pusher. Fig. 6 and Fig. 7 show the polyhedral approximation of the motion cone.

Procedure to compute polyhedral motion cone:

1. Solve (9) and (10) simultaneously to get for corresponding to every edge of .

2. Define the set of computed in step 1 as the generators/edges of the support/grasp wrench-cone .

3. Map to the object twist space using (3) to get the polyhedral approximation of the motion cone.

### V-C Experimental Validation of the Motion Cone

We use a manipulation platform equipped with an industrial robot arm, a parallel-jaw gripper, a feature in the environment that act as pusher, and a Vicon system for object tracking.

To evaluate the experimental validity of the motion cone in the gravity plane, we collected data for the slip observed at the pusher contact for randomly sampled pushes. We used the rectangular prism object listed in Table I and the grasping force was set to  N. Fig. 7 shows the grasp-pusher configuration used for the experiments. Due to kinematic constraints of the robot and the workspace, we limit the sampled pushes to the space of --- displacements in . Fig. 8 shows the experimental results compared to the computed one. Fig. 8: 2000 random prehensile pushes in the configuration shown in Fig. 7 are characterized by the slip observed at the pusher contact. The motion cone above is the same polyhedral motion cone (¯¯¯¯Vobj) computed in Fig. 7, but shown in a different orientation for better visualization. (a) No slipping was observed for most of the pushes inside the polyhedral approximation of the motion cone, (b) Most of the pushes for which the slipping was observed are outside the motion cone.

## Vi Planning In-Hand Manipulations via Motion Cones

In this section we demonstrate the application of motion cones for planning in-hand manipulations with stable prehensile pushes. The problem formulation is similar to the one presented in our recent work . We assume that an object is grasped in a parallel-jaw gripper and manipulated in the gravity plane by pushing against features in the environment. We assume the following physical properties of the system:

• Object geometry and mass.

• Initial and goal pose of an object in a grasp, specified by the locations of each finger contacts.

• Gripping force.

• Set of pusher contacts, specified by their geometries and locations in the object frame.

• Coefficient of friction at all contacts.

In , we present a planning framework where at the high-level, a T-RRT-based architecture samples different object poses in the grasp [4, 16]. At the low level, a rejection check is implemented using a constraint similar to (8) to evaluate if the sampled configuration can be reached using a stable prehensile push. We adopt the same high-level T-RRT-based planning approach in this paper, but without the low-level rejection check. Rather, the planner always grows the tree towards the sampled pose as best as possible using the motion cones at the nearest node to the sampled pose.

For selective exploration, the TRRT* framework relies on a transition test that filters the sampled configurations to prefer exploration in low configuration-cost regions. We define the configuration cost as the distance from the goal. The transition test softly constrains the stochastic exploration towards the goal grasp, while allowing the flexibility to explore high-cost transitions if they are necessary to get the object to the goal.

For effective connections, the T-RRT algorithm uses the underlying RRT  framework to make and rewire the connections in the tree at every step such that the cost of the nodes is reduced when possible. We define the cost of a node as the sum of the cost of the parent node and the cost of the push to reach the sampled node from the parent node. We set the cost of a push if the parent node uses the same pusher as the child and otherwise. With our node cost definition, the planner generates pushing strategies that prefer fewer pusher switch-overs to push the object to the desired pose.

Let denote a configuration of an object, i.e., the pose of the object in the gripper frame, which is fixed in the world. In this paper, we are considering planar manipulations in a parallel-jaw grasp, so the configuration space is , i.e., the object can translate in the grasp plane () and rotate about a perpendicular () to the grasp plane.

Algorithm 1 shows our in-hand manipulation planner. Let and be an initial and desired pose of the object in the gripper frame respectively. The planner initiates a tree with and generates motion cones at .

While the desired object pose is not reached within some cost threshold, a random configuration is sampled. A nearest configuration to in the tree is found and an unit-step object pose towards is computed. Using the transition test, the planner evaluates if moving in the direction of from is beneficial or not. If it is beneficial, the motionCone_push routine computes an object configuration closest to that can be reached using the motion cones at . It is further checked if moving towards is beneficial and if is an object configuration at which the grasp on the object is maintained. If both the criteria are satisfied, is added to the tree such that the local cost of and the nodes near are lowered if possible. The motion cones are generated for every new node added to the tree.

Two important routines in Algorithm 1, particularly for this paper, are generate_motionCones and motionCone_push.

generate_motionCones computes polyhedral motion cones for a given object configuration in the grasp using the procedure listed in Section V-B. At every node, we will have the same number of motion cones as that of pushers.

motionCone_push finds an object pose closest to the desired sampled pose that can be reached. This computation is done using the motion cones at the parent node . If the object twist needed from the parent node pose to the sampled pose is already inside any of the motion cones, the sampled pose can be directly reached. If the required object twist is outside all the motion cones, a twist that is inside one of the motion cones and closest to the desired twist is selected.

The use of motion cones for fast low-level unit-step propagation of the system and T-RRT-based framework for high-level planning allows us to explore the configuration space of different object poses in the grasp and generate pushing strategies for the desired in-hand manipulation.

## Vii Regrasp Examples and Experimental Results

We evaluate the performance of our planner with examples of a parallel-jaw gripper manipulating a variety of objects. The initial pose of an object in the gripper is treated as . Table II lists the goal poses (in [mm, mm, deg.]) for different examples. While there are no comparable available algorithms that can solve the type of regrasps we are interested in, we provide comparisons with our own implementations of the same high-level planner paired with different algorithms to solve the mechanics of prehensile pushing. These include sampling with rejection by a feasibility check for stable pushing , and a complementarity formulation (MNCP) that allows both sticking and slipping at the pusher contact . We compare the performance in terms of planning time and the quality of the solutions. The planning times in Table II are the median times over 10 trials. All the computations are done in MATLAB R2017a on a computer with Intel Core i7 2.8 GHz processor.

In all the examples below, we assume three line pushers on the object, one on each side faces of the object parallel to the axis and one under the object parallel to the axis. We use high friction line pushers, except in the first example. Fig. 9: Simulation and experimental run for a pushing strategy to regrasp the aluminum object with low friction pushers. In the simulation figure (top), the finger and pusher contacts are shown in green and magenta color respectively. Fig. 10: A pushing strategy for [X,Z,θY] regrasp. In simulation, the direction of gravity remains constant in the pusher frame, because in reality, the pushers are fixed features.

#### Vii-1 Regrasping an object offset to the center

In this example, the goal is to regrasp the square prism horizontally 20 mm offset from the center. We use low friction pushers first. Kolbert et al.  showed that for a similar setting, if the object is pushed horizontally in the grasp it slides down as it moves sideways in the grasp. For the low-friction pushers, our planner generates a strategy where the object is first pushed up using the bottom pusher and then the side pusher is used to virtually keep the object stationary while the fingers slide up and along the length of the object as seen in Fig. 9. This plan is similar to the one found in [2, 3].

When we replace the pushers with high-friction pushers (pushers with rubber coating), the planner detects that the desired object twist lies inside the motion cone for the side pusher at the initial grasp pose, i.e, simply pushing from the side is a valid pushing strategy.

#### Vii-2 Regrasp in [X,Z,θY]

The goal in this example is to regrasp the rectangular prism requiring twist in all three dimensions []. Similar to , our planner finds a strategy to achieve the regrasp using only one pusher. In fact, as we can see in Fig. 10, the pushing strategy our planner comes up with is more direct and seems to avoid unnecessary object motions seen in the strategy shown in -Fig.1.

#### Vii-3 Manipulating a non-convex object

In this example, the goal is to regrasp a T-shaped object. The goal pose is such that a greedy approach to push the object directly towards the goal will result in losing the grasp on the object. Fig. 11: Simulated motion of the object in the grasp for a pushing strategy to manipulate T-shaped object. Snapshots of the experimental run are is shown in Fig. 2

Our planner comes up with a pushing strategy that respects the geometric constraints of the problem as shown in Fig. 11.

## Viii Discussion

The motion cone is the set of possible twists of a pushed object. It also describes the set of motions of the pusher that yield sticking behavior. It abstracts away the complex dynamics of frictional pushing and provides direct bounds in the action or effect space for pushing tasks.

In this paper we extend the concept of motion cones to a general set of planar pushing tasks with external forces such as the gravitational force in the plane of motion. We show that the motion cone for a general planar push is defined as a cone with low-curvature faces, and propose a polyhedral approximation for efficient computation.

We demonstrate the use of motion cones as the propagation step in a sampling-based planner for in-hand manipulation. Combining a T-RRT-based high level planning framework and a motion cone-based dynamics propagation, the planner builds in-hand manipulation strategies with sequences of continuous prehensile pushes in a fraction of a second.

The motion cone provides a direct knowledge of the set of reachable configurations. Such a structure of reachable volumes could enable planning through regions/volumes of configuration space [1, 23, 26, 27]. Moreover, motion cones, as bounds on pusher actions or as bounds on the object motions, have an adequate form to be incorporated into trajectory optimization frameworks to plan pushing strategies. We believe that the extension and application of motion cones to more general settings provides new opportunities for fast and robust manipulation through contact.

## Acknowledgments

This material is based upon the work supported by Mathworks, the Chang Award and the MIT Merrill Lynch Fellowship. We would like to thank the members of the Mcube Lab, particularly Francois Hogan, Nima Fazeli, and Maria Bauza for helpful discussion and advice. Special thanks to Prof. Matt Mason for suggestions on the graphical illustrations of the motion cone in Fig. 5 and Fig. 6.

## References

• Brock and Kavraki  Oliver Brock and Lydia E Kavraki. In IEEE International Conference on Robotics and Automation (ICRA), volume 2, pages 1469–1474, 2001.
• Chavan-Dafle and Rodriguez  Nikhil Chavan-Dafle and Alberto Rodriguez. Sampling-based Planning of In-Hand Manipulation with External Pushes. In Int Symp Robot Res, 2017.
• Chavan-Dafle and Rodriguez  Nikhil Chavan-Dafle and Alberto Rodriguez. Stable Prehensile Pushing: In-Hand Manipulation with Alternating Sticking Contacts. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
• Devaurs et al.  D. Devaurs, T. Siméon, and J. Cortés. IEEE Transactions on Automation Science and Engineering, 13(2):415–424, 2016.
• Dogar and Srinivasa  M. R. Dogar and S. S. Srinivasa. In IEEE International Conference on Intelligent Robots and Systems, pages 2123–2130, 2010.
• Dogar and Srinivasa  Mehmet Dogar and Siddhartha Srinivasa. In Robotics: Science and Systems. MIT Press, July 2011.
• Erdmann  M. Erdmann. In IEEE Int Conf Robot and Sys, volume 1, pages 163–170 vol.1, 1993.
• Erdmann  Michael Erdmann. International Journal of Robotics Research, 13(3):240–271, 1994.
• Erdmann  Michael Erdmann. In International Journal of Robotics Research, volume 17, 1998.
• Goyal  S Goyal. Phd dissertation, Department of Mechanical Engineering, Cornell University, 1989.
• Hogan and Rodriguez  Francois Hogan and Alberto Rodriguez. Feedback control of the pusher-slider system: a story of hybrid and underactuated contact dynamics. In 12th Int Workshop on the Algorithmic Foundations of Robotics, 2016.
• Hogan et al.  Francois Hogan, Eudald Romo Grau, and Alberto Rodriguez. Reactive Planar Manipulation with Convex Hybrid MPC. In IEEE International Conference on Robotics and Automation, 2018.
• Holladay et al.  Anne Holladay, Robert Paolini, and Matthew T. Mason. In IEEE International Conference on Robotics and Automation, pages 3675–3681, 2015.
• Hou  Yifan Hou. Master’s thesis, Robotics Institute , Carnegie Mellon University, Pittsburgh, PA, May 2017.
• Howe and Cutkosky  Robert D. Howe and Mark R. Cutkosky. The International Journal of Robotics Research, 15(6):557–572, 1996.
• Jaillet et al.  L. Jaillet, J. Cortés, and T. Siméon. IEEE Transactions on Robotics, 26(4):635–646, 2010.
• Karaman and Frazzoli  Sertac Karaman and Emilio Frazzoli. International Journal of Robotics Research, 30(7):846–894, 2011.
• Kolbert et al.  Roman Kolbert, Nikhil Chavan-Dafle, and Alberto Rodriguez. In International Symposium on Experimental Robotics, 2016.
• Kumar et al.  V. Kumar, Y. Tassa, T. Erez, and E. Todorov. In IEEE International Conference on Robotics and Automation (ICRA), pages 6808–6815, 2014.
• Lynch et al.  K. M. Lynch, H. Maekawa, and K. Tanie. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, volume 1, pages 416–421, 1992.
• Lynch and Mason  Kevin M. Lynch and Matthew T. Mason. The International Journal of Robotics Research, 15(6):533–556, 1996.
• Mason  Matthew T. Mason. The International Journal of Robotics Research, 5(3):53–71, 1986.
• Morales et al.  Marco Morales, Roger Pearce, and Nancy M Amato. In IEEE International Conference on Robotics and Automation, pages 1029–1034, 2007.
• Posa et al.  Michael Posa, Cecilia Cantu, and Russ Tedrake. Int Journal of Robotics Res., 33(1):69–81, 2014.
• Shi et al.  J. Shi, J. Z. Woodruff, and K. M. Lynch. In 2015 IEEE International Conference on Intelligent Robots and Systems, pages 870–877, 2015.
• Shkolnik et al.  A. Shkolnik, M. Walter, and R. Tedrake. Reachability-guided sampling for planning under differential constraints. In IEEE International Conference on Robotics and Automation, pages 2859–2865, 2009.
• Shkolnik and Tedrake  Alexander Shkolnik and Russ Tedrake. Sample-Based Planning with Volumes in Configuration Space. In Available on arXiv, 2011.
• Sintov and Shapiro  A. Sintov and A. Shapiro. In IEEE International Conference on Robotics and Automation (ICRA), pages 4888–4893, 2016.
• Stewart and Trinkle  D. E. Stewart and J. C. Trinkle. Intl Journal for Numerical Methods in Engineering, 39(15):2673–2691, 1996.
• Sundaralingam and Hermans  Balakumar Sundaralingam and Tucker Hermans. In Proceedings of Robotics: Science and Systems, 2017.
• Todorov et al.  E. Todorov, T. Erez, and Y. Tassa. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012.
• Viña B et al.  F. E. Viña B, Y. Karayiannidis, C. Smith, and D. Kragic. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 399–406, 2016.
• Xydas and Kao  Nicholas Xydas and Imin Kao. The International Journal of Robotics Research, 18(9):941–950, 1999.
• Zhou and Mason  Jiaji Zhou and Matthew T. Mason. In Int Symp Robot Res, 2017.