Log In Sign Up

Asymmetric Dual-Arm Task Execution using an Extended Relative Jacobian

by   Diogo Almeida, et al.
KTH Royal Institute of Technology
Chalmers University of Technology

Dual-armed coordination is a classical problem in robotics, which gained an added relevance as robots are expected to become more autonomous and deployed outside structured environments. Classical solutions to the coordination problem include leader-follower or master-slave approaches, where the coordinated task is assigned to one arm, while the other adopts a passive role, such as the regulation of contact forces. Alternatively, cooperative task space approaches represent the task space of the dual-armed system in terms of absolute and relative motion components, where, for the most part, the relative task is split evenly between the manipulators. Relative Jacobian methods offer an alternative approach where the task space of the system contains only its relative motion. This is an attractive approach for tasks which do not require an absolute motion target, as a larger degree of redundancy becomes available for the optimization of secondary goals of the cooperative system. However, existing relative Jacobian solutions do not allow for a particular degree of collaboration to be set explicitly. In this work, we present a task-space analysis of common cooperation strategies and propose a novel, asymmetric, definition for the relative motion space. We show how this definition enables a user to prescribe a specific degree of cooperation between arms while using a relative Jacobian solution. Simulation results are provided to illustrate some properties of this novel cooperation strategy.


page 6

page 7

page 8


A Lyapunov-Based Approach to Exploit Asymmetries in Robotic Dual-Arm Task Resolution

Dual-arm manipulation tasks can be prescribed to a robotic system in ter...

Dynamic Movement Primitive based Motion Retargeting for Dual-Arm Sign Language Motions

We aim to develop an efficient programming method for equipping service ...

Dual-arm Coordinated Manipulation for Object Twisting with Human Intelligence

Robotic dual-arm twisting is a common but very challenging task in both ...

Using dual quaternions in robotics

We advocate for the use of dual quaternions to represent poses and twist...

Trimanipulation: Evaluation of human performance in a 3-handed coordination task

Many teleoperation tasks require three or more tools working together, w...

Geometric Perspectives on Fundamental Solutions in the Linearized Satellite Relative Motion Problem

Understanding natural relative motion trajectories is critical to enable...

Mixline: A Hybrid Reinforcement Learning Framework for Long-horizon Bimanual Coffee Stirring Task

Bimanual activities like coffee stirring, which require coordination of ...

1 Introduction

Consider a dual-armed robotic system, tasked with executing a coordinated manipulation task. As opposed to uncoordinated tasks, coordinated manipulation benefits from treating the two arms in the robotic system as a single kinematic chain. Concretely, the task space for coordinated manipulation can be defined in terms of absolute and relative motion components. Absolute motion is equivalent to the resultant motion from external forces applied to a jointly carried object. Relative motion, conversely, refers to the motion of one end-effector w.r.t the other [Uchiyama1987, Uchiyama1988]. This type of motion can be used to model tasks such as, e.g., assembly or tool manipulation. The formulation of the manipulation task space in terms of absolute and relative motion is called the Cooperative Task Space (CTS) [Chiacchio1996].

We are interested in analysing the resolution of the CTS variables into end-effector velocities. Recent work proposes an Extended CTS (ECTS) definition which makes use of a novel definition of absolute motion to attain an asymmetric resolution of the relative motion variable [Park2015, Park2016]. Alternatively, it is common to model solely the relative motion as a primary task quantity, by using the relative Jacobian formulation [Lewis1990] when solving the dual-armed system’s differential Inverse Kinematics (IK). In many cases, a secondary task is added which constrains the absolute motion of the system resulting in asymmetric relative motion [Ajoudani2014, Jamisola2015, Hu2015, Foresi2017].

In the following text, we discuss the CTS formulations and observe how a conflict between the definition of absolute and relative motion tasks is introduced by the ECTS method, Sec. 3

. In fact, ECTS relies in an asymmetric absolute motion definition, which we argue is at odds with the concept of absolute motion: commanding an ’asymmetric absolute motion’ will result in the appearance of relative motion components. If we consider an absolute motion task such as jointly carrying an object, this will inevitably lead to internal forces on the object. On the other hand, relative motion tasks are

inherently redundant. The resolution of the relative motion task into the robot end-effectors can be arbitrary without any fundamental conflict with the concept of relative motion [Almeida2018].

We leverage our observations to propose an asymmetric relative motion space, which enables the asymmetric resolution of the relative motion without resorting to a redefinition of the absolute motion space, Sec. 4. The new space leads to the proposal of a novel relative Jacobian formulation and a corresponding differential IK algorithm, Sec. 5. This allows a user to specify a concrete degree of asymmetry in the relative task resolution, within a relative Jacobian framework. We illustrate the key properties of our approach through numerical simulations, Sec. 6, and make our code freely available, which, in addition to the simulations, implements velocity controllers that employ all of the discussed methods111˙manipulation.

2 Related Work

A common solution to the problem of bimanual cooperative manipulation is to employ a completely asymmetrical leader-follower (or master-slave) approach [Lu1987, Zheng1989, Yan2016, Almeida2016, Almeida2016b]. Alternatively, by considering the external and internal forces on a jointly held rigid object, the task can be modelled in terms of absolute and relative motion components [Uchiyama1987, Uchiyama1988]. The CTS definition results from a modelling approach which is independent of the statics of the dual-armed system [Chiacchio1992, Chiacchio1996]. The ECTS [Park2015, Park2016] is obtained by redefining the absolute motion space of the coordinated system. CTS-based approaches have been used to describe coordinated tasks in, e.g., human-robot interaction settings [Nemec2016], the cooperative manipulation of a mechanism [Almeida2018] or the execution of a bimanual dexterous manipulation task [Cruciani2018].

Relative Jacobian methods model solely the relative motion task [Lewis1990, Lewis1996, Jamisola2015], making it a common choice when addressing tasks that require just the relative motion space, such as machining [Owen2005, Owen2008, Lee2012], assembly [Ajoudani2014] or drawing [Lee2015]. The absolute motion is part of the relative Jacobian’s redundant space. This can be exploited, e.g., to enhance self-collision and obstacle avoidance as secondary tasks [Mohri1996], or for joint-limit avoidance [Hu2015, Foresi2017, Ortenzi2018].

Hierarchical quadratic programs (HQP) [Escande2014] are an alternative approach to obtain solutions to the problem of inverse differential kinematics. While in this article we focus on pseudo-inverse solutions to the differential IK problem, all the discussed Jacobian formulations can be employed in the context of HQP, as showed in, e.g., [Ceriani2016] and [Tarbouriech2018] for the relative Jacobian.

3 Cooperative motion spaces

In this section, we perform a task-space analysis of existing cooperation strategies. From the kinematic point of view, we will consider master-slave methods as examples of purely asymmetric task execution, which, as we will show, are covered by the analysed strategies.

CTS methods employ the full dual-arm task space, by defining an absolute and a relative motion spaces. A possible approach to obtain asymmetric relative motion is to redefine the absolute motion space, as is the case with ECTS. We observe that this approach introduces conflicts between the relative and absolute motion tasks.

3.1 Notation

Consider a dual-armed system composed by two robotic manipulators. Let denote a generic coordinate frame, e.g., of the -th manipulator’s end-effector, where . Each frame is defined by a position, and orientation , expressed in a common base frame. We write the angle-axis representation of as , where is the angle one must rotate about the axis to obtain . The twist at each end-effector is defined as , where denotes the end-effectors’ angular velocity. Finally, we denote the nullspace of a matrix as .

3.2 Absolute and relative motion spaces

The absolute and relative motion components can be derived from properly defined motion frames.

(a) Symmetric execution.
(b) Asymmetric execution
Figure 1: A simple relative motion task is to open/close a pair of scissors. Note that an asymmetric execution results in a change of the average orientation (green arrow) of the two scissors’ pieces.

3.2.1 Cooperative Task Space

In CTS [Chiacchio1996], absolute and relative motion frames are defined, respectively and , such that


where and are extracted from the angle-axis representation of , and


The absolute and relative motion twists, respectively and can be obtained through differentiation,


In task space, this relation can be expressed by a linking matrix, . Let and . Then,


The CTS linking matrix is square and nonsingular, and can be recovered through matrix inversion,


It is clear from (5) that the coordinated task is divided symmetrically between the two end-effectors: each end-effector executes the prescribed absolute motion, and the relative motion task is divided evenly among them.

3.2.2 Extended Cooperative Task Space

The CTS formulation has been extended in [Park2015, Park2016]. A different definition for the absolute frame is adopted,


with . The ECTS linking matrix is given by


and , as in (4). The effect of the cooperation parameter is clear through the inversion of (7),


That is, by setting , asymmetric relative motion can be achieved, such that in the new absolute frame. The relative motion task can be solved in a serial (master-slave, or ), blended (asymmetrical, ) or parallel (symmetrical, ) mode of cooperation. Note that the introduction of asymmetries in the relative motion affects the symmetric absolute pose, Fig. 1.

Both and are composed by an absolute and a relative part,


where . These matrices depend on the definition of the frames where the absolute and relative motion are expressed. For both CTS and ECTS, the relative motion is given by . CTS adopts the symmetric resolution of the absolute motion, while ECTS makes use of an asymmetric formulation, .

The (E)CTS motion space fully specifies the cooperative motion in terms of absolute and relative variables. In many tasks, however, this overconstrains the problem. For example, a peg-in-hole assembly can be executed by having each robot arm grasp a part and executing a relative motion between its end-effectors. The absolute motion is a functional redundancy in this case, Fig. 2. Other examples include machining [Owen2005] or drawing [Lee2015].

Figure 2: The absolute motion of the cooperative system is redundant w.r.t the relative motion task.

3.2.3 Relative motion space

For tasks which can be solved exclusively through relative motion, it is possible to specify a desired only. This removes the need to adopt the dimensional linking matrices in (9). In this scenario, we can use the Moore-Penrose pseudo-inverse, , to resolve a desired relative motion into velocities of each robot end-effector,


which matches the relative part of the CTS solution (5). An homogenous solution can be added through a nullspace projection, e.g.,


In this solution, a secondary task is being mapped to the absolute motion space, with . The resulting velocity distribution is analogous to the CTS case, eq. (5). However, if the secondary objective is not prescribed in , a conflict between primary and secondary tasks is introduced which will induce asymmetries in the motion of the system’s end-effectors.

[Constructing asymmetric relative motion] Let , with initial state and , and consider the relative velocity command . Additionally, let , with . The solution (11), applied to this one-dimensional case, yields the autonomous system


The relative velocity of this system is given by , i.e., the desired relative command is attained. The degree of asymmetry in the motion of the two points can be computed through and will depend on the secondary task gain . We depict the time evolution of system (12), as well as for different values of in Fig. 3. It is clear that for a larger the more asymmetric is the motion of the two points, since the induced absolute motion will regulate to remain at the origin. For in particular, it can be seen that the injected absolute motion allows to contribute the most to the relative motion task.

Figure 3: Numerical simulation of the point-system (12). Different values of affect the degree of asymmetry in the execution of the relative motion task.

3.3 Asymmetric resolution of the CTS variables

We have seen in Sec. 3.2.1 that inverting the mapping (4) yields a symmetric distribution of the CTS variables into the two robot end-effectors. In the current literature, two predominant coordination approaches achieve an asymmetric execution of the CTS variables:

  1. In the ECTS [Park2015, Park2016], a redefined absolute motion space constrains the resolution of the relative motion to be asymmetric, Sec. 3.2.2

  2. Methods which consider only the relative motion space often add a secondary task to one of the end-effectors [Ajoudani2014, Jamisola2015, Hu2015, Foresi2017], which results in asymmetric behavior, see Example 12

Both methods combine the CTS variables to achieve asymmetric behavior. The ECTS solution for the absolute motion introduces relative motion components,


while the nullspace projection (11) maps the secondary task of the system to the absolute motion space. We will show how this projection can be modified to introduce a deliberate degree of asymmetry in the execution of the relative motion task.

4 Defining an asymmetric relative motion space

Consider that we wish to control the relative motion between the end-effectors in an asymmetric manner. Analogously to the ECTS approach, we would like to be able to specify a degree of cooperation between the arms. This can be achieved through a redefined relative motion frame. If we denote the angle-axis representations of and , then the asymmetric relative motion frame can be defined as


Asymmetric relative motion can be obtained through the differentiation of (14),


where the asymmetric relative linking matrix is


The particular solution to (15) is analogous to (10) and corresponds to the the ECTS asymmetric distribution of the relative motion(8) ,


The motion space defined by in (14) is characterized by the following properties:

  1. Commanding a relative velocity in , eq. (17), renders the asymmetric absolute motion space (6) invariant.

  2. The inverse mapping is a generalized inverse of .



and consider the solution (17). We have,


and thus, the asymmetric absolute motion frame (6) remains invariant to velocities commanded in the motion frame from (14). acts as a generalized inverse to since


The solution for the asymmetric relative motion space (17) can be obtained by adding an homogeneous component to (10). This follows straightforwardely from the fact that acts as a generalized inverse to (20),


The solution (21) generates the required absolute motion to ensure the degree of asymmetry in the execution of the prescribed relative velocity ,


We can use this result to construct a relative Jacobian method for solving the system’s differential IK which takes into account a desired degree of asymmetry.

5 Differential Inverse Kinematics

The prescribed cooperative motion can be distributed between the two end-effectors, as seen in the previous section, and each arm can solve its differential IK separately. Alternatively, we can derive differential IK algorithms which directly resolve a desired cooperative motion to the joint state of the complete dual-arm chain.

5.1 Notation

Let represent the joint variables of the -th manipulator, with , and its Jacobian, such that . We will omit the Jacobian’s dependency on the joint variables for the remaining of this text, and assume that the manipulators’ Jacobians are full rank, i.e., the manipulators are not operating in a singular configuration. Finally, let

be the skew-symmetric matrix such that, for

, .

5.2 Joint Jacobians

Let . We can solve the cooperative task by using any of the methods in section 3 and computing the differential IK for each individual manipulator. As an example, a relative motion task can be solved into joint space by computing


where . The nullspace of has dimension . Alternatively, all the previously discussed cooperative motion definitions can be extended to a mapping from task to joint spaces through the construction of joint Jacobians. This is equivalent to treating the two manipulators as a single kinematic chain. We can obtain the joint Jacobians through a pre-multiplication with the appropriate linking matrix, e.g., . It is often convenient, however, to express the cooperative task w.r.t object frames rigidly connected to each end-effector, . These can represent, e.g., the tip of the peg and the center of the hole in a peg-in-hole type of assembly task. In this case, a transformation between the twists at and the corresponding is needed.

We define two virtual sticks, which connect to the end-effector’s positions , such that . The necessary screw transformation is defined as such that . Let . The joint Jacobians are


for the CTS formulations, where and


in case only a relative motion task is specified, with . Note that, in general, . In fact, treating the dual-armed system as a single kinematic chain results, in general, in a larger dimension of the Jacobian nullspace, as and thus


In practice, the larger nullspace includes absolute motion components which are not present when solving (23). The Jacobian in (25) is often called relative Jacobian.

The forward differential kinematics for a relative motion task are expressed as


In the following, we will focus on finding feasible joint space solutions for (27). It is well known that such solutions have the general form


with being part of the homogeneous solution to (27

). The joint space vector

composes a secondary task, added to in a task-priority manner. By setting we get the minimum norm solution for . The symmetric absolute motion space is orthogonal to the relative motion space used in the relative Jacobian definition (25),


This absolute motion is then a functional redundancy to (27) and thus, in general, its minimum norm solution can contain absolute motion components, depending on the manipulators’ design. The desirability of this property depends on the task requirements. If the absolute motion must be prescribed, using the CTS is ideal. In light of the discussion in Sec. 3.3, we argue that the ECTS can be used to achieve asymmetric relative motion when . However, if is to be commanded, ECTS requires 222In fact, this is the case in [Park2016].. Otherwise, the resulting asymmetric absolute motion command will contain relative motion components, which is undesirable. Alternatively, using (23) prevents any non-specified absolute motion from occurring. When it is acceptable for absolute motion to be exploited as a functional redundancy, exploitation strategies include, e.g., workspace obstacle and self-collisions avoidance [Mohri1996].

5.3 Asymmetric relative Jacobian

Figure 4: Initial robot configurations for the case studies. Fig. 3(a) and 3(b): linear displacement between frames seen from a front and a top view. Fig. 3(c) and 3(d): rotational displacement seen from a front and a top view.

A common choice for is to specify a motion for one of the system’s end-effectors [Ajoudani2014, Jamisola2015, Hu2015, Foresi2017], e.g.,


As seen in Example 12, this secondary task will be mapped into the system’s absolute motion space and induce asymmetries in the execution of . We propose instead to use the asymmetric relative motion space defined in section 4 to extend the relative Jacobian methods and allow for setting the degree of cooperation through a parameter. We use (16) to define an asymmetric Jacobian,


We can now solve the differential IK by defining


Imposing (32) as a secondary task will filter out undesired asymmetric absolute motion, which can occur as part of the minimum norm solution represented by the pseudoinverse, since it belongs to an orthogonal motion space, eq. (19). The desired task space relative motion will be preserved, however, as shown in Corollary 4, and as we will illustrate in Sec. 6. Note that asymmetric absolute motion necessarily contains relative motion components, which is in general undesirable: for many tasks, non-prescribed relative motion can lead to internal stresses on the robot manipulator.

(a) Translational relative motion task.
(b) Rotational relative motion task.
Figure 5: Evolution of the relative errors in the two types of motion task, when . The dashed lines represent the solution when using from (32) as a primary task. The solid lines use (32) in a task priority manner (28).

5.4 Relative task

The choice of in (28) defines the relative motion task. This can be set as the output of a task-specific controller [Almeida2018], a feedforward command from a teleoperator, or as an error signal from a pose regulation task [Caccavale2000, Park2016]. In our examples, we will assume without loss of generality that the task is to align the coordinate frames . The alignment error is given by for the displacement between frames and for the orientation. If we denote the error quaternion as , where is the vector and the scalar part, we set as the feedback control law [Caccavale2000],


where is positive definite.

6 Case Studies

We illustrate different properties of the novel relative Jacobian (31) with two case studies, in which the primary task is a relative motion task given by (33) with . The case studies are based on a simulated Rethink Robotics’ Baxter dual-armed robot. Two relative motion tasks are considered, one where the alignment error is purely translational, Fig. 3(a)-3(b), the other where the error is purely rotational, Fig. 3(c)-3(d). The initial poses of the object frames, expressed in the reference frame (Baxter’s torso frame), are given by and , with . The subscript index corresponds to the left and to the right manipulator.

6.1 Nullspace projection of the extended relative Jacobian solution

(a) ECTS
(b) Proposed
Figure 6: Results for the execution of a pure translational task.
(a) ECTS
(b) Proposed
Figure 7: Results for the execution of a pure rotational task.

In this case study, we show how the joint space solution (32) introduces undesirable relative motion components, as a consequence of the asymmetric orthogonality from eq. (19). This property implies that assymetric absolute motion lies in the nullspace of (31), which contains relative motion components, eq. (13). As such, the nullspace projection (28) is required to use our proposed extended relative Jacobian. In the numerical simulations, we set to denote a blended mode where the right manipulator () executes most of the relative task. Fig. 4(a) depicts the evolution of the relative error for a purely translational task. Conversely, Fig. 4(b) shows the error progression in a purely rotational task. The solution (32) induces undesired relative motion in both tasks. This is clearly visible in the rotational error for the translational task and in the translational error for the rotational task. The nullspace projection of (32) in (28) filters out these undesired components.

6.2 Asymmetric relative motion

(a) ECTS
(b) Proposed
Figure 8: Translational relative task.

The nullspace projection of the asymmetric solution (32) removes some motion components which are redundant to the asymmetric relative task. In this case study, we show that the symmetric absolute motion still remains as an exploitable functional redundancy, and that asymmetric relative motion is achieved, by comparing our method against the ECTS solution with , for different values of the coefficient . Note that when , we obtain the default CTS and relative Jacobian solutions, respectively.

We depict the joint solutions when in Fig. 6, for the translational relative motion task and in Fig. 7 for the rotational task, where it can be observed that the asymmetric task execution results in distinct final configurations for the system. The norm of the joint space trajectory was for the ECTS solution and for ours, showing that the availability of the larger redundant space allows for a smaller norm of joint velocities to be computed by the differential IK method. In addition, we show the induced symmetrical absolute motion in both cases in Fig. 8 and 9. We provide the source code required to test other values of 333˙manipulation.

The smaller joint trajectory norm of our method is a result of the exploitation of the absolute motion as a functional redundancy. In the translational motion task our method changes the absolute orientation of the system, Fig. 7(b), and conversely a linear component is introduced in the rotational motion task, Fig. 8(b). In comparison, ECTS will induce symmetrical absolute motion on the system when , Fig. 7(a) and 8(a), however, this is only along the dimensions commanded by , as noted in eq. (22).

(a) ECTS
(b) Proposed
Figure 9: Rotational relative task.

7 Conclusions

We study how existing methods to model the differential kinematics of dual-armed robotic systems distribute the relative motion between end-effectors, in the context of cooperative manipulation tasks. The asymmetric execution of a relative motion task can be achieved by redefining the absolute motion space, as in the ECTS formulation, or through the addition of a secondary task in a relative Jacobian setting. The ECTS approach requires absolute motion to be part of the system’s primary task, and relies on an unintuitive definition of an asymmetric absolute motion space, which is useful solely for the purpose of the asymmetric execution of the relative task. In a relative Jacobian setting, adding a secondary task induces time-varying asymmetries which are a function of the secondary task’s definition, as we illustrated in Example 12.

In contrast, we show how a deliberate degree of asymmetry can be imposed on the execution of a relative motion task within a relative Jacobian formulation. This approach allows for, e.g., a master-slave execution of the relative motion task, as opposed to the time-varying asymmetric execution of other relative Jacobian methods. We illustrate the properties of this approach with two case studies. In the first study, we show that the asymmetric absolute motion which lies in the nullspace of our proposed Jacobian includes undesirable relative motion components, and we illustrate how our proposed differential IK method filters out these components. Then, in our second case study, we show how the absolute motion space is retained as a functional redundancy even after applying our differential IK scheme, through a comparison with the ECTS approach: for the same relative motion task, our method obtains smaller joint velocity norms, thanks to the induced absolute motion.