Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations

by   Ziyuan Jiao, et al.
Drexel University

We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated, and solved by trajectory optimization. This new VKC perspective of mobile manipulation allows a service robot to (i) produce well-coordinated motions, suitable for complex household environments, and (ii) perform intricate multi-step tasks while interacting with multiple objects without an explicit definition of intermediate goals. In simulated experiments, we validate these advantages by comparing the VKC-based approach with baselines that solely optimize individual components. The results manifest that VKC-based joint modeling and planning promote task success rates and produce more efficient trajectories.



page 1

page 2

page 5

page 6


Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learning

Mobile manipulation tasks remain one of the critical challenges for the ...

Design-Informed Kinematic Control for Improved Dexterous Teleoperation of a Bilateral Manipulator System

This paper explores the possibility of improving bilateral robot manipul...

N^2M^2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments

Despite its importance in both industrial and service robotics, mobile m...

Motion Planning for Multi-Mobile-Manipulator Payload Transport Systems

In this paper, a kinematic motion planning algorithm for cooperative spa...

An optimization framework for simulation and kinematic control of Constrained Collaborative Mobile Agents (CCMA) system

We present a concept of constrained collaborative mobile agents (CCMA) s...

Learning Extended Body Schemas from Visual Keypoints for Object Manipulation

Humans have impressive generalization capabilities when it comes to mani...

Motion Planning for a Humanoid Mobile Manipulator System

A high redundant non-holonomic humanoid mobile dual-arm manipulator syst...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Mobile manipulation is a core capability for a service robot to function properly in household environments and excel in various tasks. However, since service robots usually come with a bulky mobile base along with a large base footprint, they often struggle in household environments due to three unique challenges: (i) An indoor space is confined and clustered, constraining the robot’s locomotion and posing additional challenges for mobile manipulation. (ii) The majority of tasks involve manipulating objects with diverse structures (e.g., articulated objects like doors and drawers), challenging to generate motion plans for mobile manipulators with a unified approach. (iii) The base and arm’s movements have to be coordinated to ensure efficient and safe operations during mobile manipulation.

There exists research efforts focusing on each of the above challenges, i.e., motion planning in confined space [5, 7, 12], control or learning frameworks for opening doors and drawers [17, 21, 20, 41], and whole-body planning for foot-arm coordination [36, 4, 8]. However, a unified approach that tackles all three challenges altogether in household settings is still mostly missing. Consequently, a service robot’s mobile manipulation skills are far from ideal in terms of efficacy or fluency.

In stark contrast, humans possess fluid manipulation skills and interact with an environment efficiently. Cognitive psychologies and philosophers have proposed a theory of body schema [9]: Humans maintain a body’s representation during their motions and interactions with the environment; this representation is malleable and can be extended to incorporate external objects. By treating the manipulated object as part of the extended limb, the theory of body schema provides a plausible account for why humans excel in complex manipulation tasks, from picking and placing an object, to opening doors and drawers, to tool-use [15]. Although the idea of the body schema has been introduced to the robotics community to represent robot structures and guide robot’s behaviors [14], it has left untouched whether the theory of body schema would promote a service robot’s (mobile manipulation in particular) planning and execution skills in complex manipulation tasks. And if it does, what would be a proper representation at a computational level?

Fig. 1: Diverse interactions a service robot needs to perform in a household environment. By abstracting the objects’ kinematic structures and forming a VKC, a service robot can plan and act more efficiently with improved foot-arm coordination.

To answer this question, we propose to abstract the object being manipulated—rigid objects and constrained mechanisms (doors, drawers, etc.)—by its kinematic structure. We integrate the kinematics of the robot and the manipulated object by constructing a single kinematic chain using the idea of Virtual Kinematic Chain (VKC[26]. Specifically, the kinematic structure of the manipulated object is augmented to the manipulator by connecting the end-effector to an attachable location of the object. A virtual transformation between the mobile base and the virtual base link is also augmented by incorporating its navigation information.

From this new VKC-based perspective, a mobile manipulation task in a household environment is represented by altering the state or the structure of the VKC, which leads to a motion planning problem on VKC, formulated and solved by trajectory optimization. This new perspective enables a service robot to plan and act efficiently by allowing it to directly incorporate external objects and plan the motion as a whole to achieve better foot-arm coordination; see examples in Fig. 1. In simulations, we validate the proposed VKC perspective in various mobile manipulation tasks. Our experiments show that the consolidated kinematic models are particularly suitable for service robots by alleviating intermediate goal definitions for motion planners; they promote coordinated motions among base, arm, and object.

I-a Related Work

The idea of Virtual Kinematic Chain (VKC) could be traced back to 1997 by Pratt et al[32] for bipedal robot locomotion [31]. This idea was later adopted to chain serial manipulators to form one kinematic chain [26] and to dual-arm manipulation tasks; for instance, connecting parallel structures via rigid-body objects [42], modeling whole-body control of mobile manipulators [43]. Recently, VKC is also adopted for wheeled-legged robot control [23]. In this paper, we further push the idea of VKC to a mobile manipulator and demonstrate its advantages in modeling and planning complex manipulation tasks in household environments.

Motion planning is among the largest and most fundamental fields in robotics. In essence, methods can be roughly categorized into three major doctrines: search-based (e.g., A [13], D [38]), sampling-based (e.g., RRT [24] and its variants [22, 19]), and trajectory optimization (e.g., CHOMP [33], TrajOpt [35]). We formulate the motion planning problem on VKC following the conventions in TrajOpt, as it incorporates kinematic constraints better than sampling-based methods while avoiding searching in large spaces.

Efficiently performing mobile manipulation tasks are challenging. Notable efforts have recently been dedicated to algorithms or system implementations, focusing on interactive manipulation tasks. For instance, equilibrium point control [17] and impedance control structure [39] are introduced to open doors and drawers. To improve efficiency, Gochev et al

. used a heuristic-based method to reduce the search space 

[10]. Taking advantage of solving the inverse kinematics, Burget et al. proposed a whole-body motion planning approach for humanoid’s constrained motion [6], and Bodily et al. proposed an algorithm for jointly optimizing a robot’s base position and joint motions [4]. More recently, Toussaint et al. proposed a multi-bounded tree search algorithm to solve multi-step manipulation tasks involving tool-use [40]. Despite their promising results, prior arts primarily focus on a specific problem setup (e.g., opening door and drawer, using tools). In comparison, the proposed approach rethinks mobile manipulation from a more general viewpoint using VKCs and tackles a broader range of tasks.

I-B Overview

The remainder of this paper is organized as follows. Section II outlines notations and formally presents the problem definition. Section III details the VKC modeling. In a series of mobile manipulation tasks, we demonstrate the efficacy of VKCs with a high success rate in Section IV. We conclude the paper with discussions in Section V.


Fig. 2: Overview of the mobile manipulation planning schematics using the proposed VKC-based approach. (a) After abstracting out the underlying kinematics of the manipulated object and the mobile manipulator, (b) a VKC is constructed. The yellow boxes denote where the virtual connections are established: (i) One between and , the virtual base frame in the world coordinate and the robot’s actual base frame, to reflect the navigational information, and (ii) another between and , the robot’s end-effector frame and the attachable frame of the object, to transfer effects of the manipulator to the manipulated object.

Ii Notations and Problem Definition

This section introduces the notations throughout the paper and the problem setup describing a mobile manipulation task.

The physical properties and kinematics of links and joints are defined following the Unified Robot Description Format (URDF) in Robot Operating System (ROS) and organized in a tree representation . Table I lists all the related notations:

Group Notation       Description


A tree represents the robot kinematic model
Robot base link’s frame; the root of
Robot end-effector link’s frame
, a kinematic chain from to
Frame of link in the kinematic chain


A tree represents the object kinematic model
Object base link’s frame; the root of
Object attachable link’s frame
, a kinematic chain from to
Frame of link in the kinematic chain


A serial VKC with Degree of Freedom (DoF)
, the state of VKC in joint space
 (), the joint goal state
A homogeneous transformation from to
The goal pose of in the world frame
TABLE I: Notations used for constructing VKCs.

Below, we further summarize the above notations:

  • [leftmargin=*,noitemsep,nolistsep,topsep=0pt]

  • The group Robot refers to notations related to the mobile manipulator, which consists of three components: mobile base, manipulator, and end-effector.

  • The group Object refers to notations related to the manipulated objects, which could be as simple as a rigid link or be an articulated object with two or more links connected by either a prismatic, revolute, or fixed joint. We introduce a virtual joint defined as an attachment, a local transformation from the object’s attachable frame (i.e., the link a mobile manipulator can grasp on) to the robot’s end-effector frame .

  • The group Others refers to constructed VKC, its state space, and other related notations in a manipulation task.

Constructing a VKC requires the inputs of robot kinematic tree , object kinematic tree , and transformation from an object attachable frame to the robot end-effector frame . The chain’s forward kinematics (FK), inverse kinematics (IK), and Jacobians can be effectively solved by existing kinematic solvers (e.g., KDL [37]).

Assuming a rigid connection between the end-effector and the attachable link during manipulation, performing a mobile manipulation task can be regarded as reaching desired VKC poses. As a result, we treat a mobile manipulation task as a motion planning problem on the VKC and solve it by trajectory optimization. Formally, it is equivalent to finding a collision-free path from the initial pose to goals in joint space and/or goal poses in Euclidean space.

The objective function of the trajectory optimization can be formally expressed as:


wherein we penalize the overall weighted squared traveled distance of every joint with the finite forward difference and overall smoothness of the trajectory with the second-order finite central difference . and are diagonal weight matrices for each joint, respectively. represents the trajectory sequence , where denotes the VKC state at the time step.

Iii Vkc Modeling

The proposed VKC modeling constructs a serial kinematic chain by (i) incorporating both robot and object kinematics via a virtual joint and (ii) augmenting a virtual base to the robot base; see Fig. 2b for a graphic illustration.

Iii-a Vkc Construction

Below we formally describe the 4-step procedure of constructing the VKC, , by consolidating the robot and the object kinematics models.

Original Structure

The kinematic models of the mobile manipulator and the manipulated object are assumed given by the perception module or by the simulator.

Kinematic Inversion

Let us take the task of opening a door as an example. In conventional kinematic notation, the door is the child link, and the door frame is its parent link in the original . To construct a VKC, this parent-child relationship needs to be inverted before it can be attached to the robot’s end-effector, i.e., the door becomes the parent link that “transforms” the door frame. Of note, such an inversion also requires updating the joint connecting the two links, since a joint (i.e., revolute/prismatic) typically constrains the child link’s motion w.r.t. the child link’s frame.

Vkc Construction

After inverting the original , a virtual joint between and is inserted, whose transformation is denoted as . In our application, the transformation of the virtual joint is updated by the actual grasping pose right before the VKC construction to minimize kinematic discrepancies introduced by the execution error. Next, the motion planner will be invoked to plan following motions for the actual VKC. The joint type could also be determined by the grasping type between the gripper and the object (e.g., revolute joint for grasping a cylindrical handle, fixed joint for grasping a rigid ball) to alleviate the inaccuracies during the execution.

Virtual Base Frame

A virtual base frame is further added and connected to the mobile base through two perpendicular prismatic joints and a revolute joint, enabling the mobile base’s omnidirectional motions on the ground plane.

After the above procedure, the constructed VKC remains in serial and forms an equality constraint to Eq. 1:


It specifies the kinematics of the VKC, which includes its forward kinematics and other physical constraints of the manipulated object; e.g., the manipulated object is fixed to the ground, which leads to a closed chain: . Failing to account for this constraint may damage the manipulated object or the mobile manipulator.

Iii-B Goals

The goal of the mobile manipulation can be formulated as an inequality constraint, in addition to the equality constraint introduced by the VKC construction in Eq. 2:


which bounds the squared norm between the final state in the goal space and the goal state with a tolerance . The function is a task-dependent function that maps the joint space of a VKC to the goal space that differs from task to task.

Again, let us take the example of opening a door. In the first phase when the robot is reaching the door handle, maps the joint space of a VKC to the robot’s end-effector pose. In this case, the goal is the robot’s end-effector pose , and Eq. 3 can be rewritten in a simplified form . In the second phase when the robot is opening the door, maps VKC’s joint space to the joint of the door’s revolute axis. Hence, is merely the angle of the revolute joint, and the trajectory of the other joints in the VKC are implicitly generated by the optimization process, together with obstacle avoidance and trajectory smoothing. Of note, Eqs. 3 and 2 are not the only forms of constraints that a VKC-based approach can incorporate; in fact, it is straightforward to add additional task constraints to the same optimization problem in Eq. 1, depending on various task-specific requirements.

Iii-C Additional Constraints

During the trajectory optimization, we further impose several safety constraints. Without loss of generality, we assume an omnidirectional base and purely kinematic constraints in this paper. However, extra constraints, such as nonholonomic constraints for non-omnidirectional mobile bases or dynamic constraints for arms, could be formulated into the optimization problem by incorporating additional time, first-order, or second-order terms [34].


Eq. 4 is an inequality constraint that defines joint limits, in which and specify the lower and upper bound of every joint, respectively. Eq. 5 is an inequality constraint that bounds the joint velocity by and the joint acceleration by to obtain a feasible trajectory that can be executed without saturation. denotes the infinity norm.

Eqs. 7 and 6 are inequality constraints that check link-object collisions and link-link collisions, respectively, where and are the number of links and the number of objects, respectively. is a pre-define safety distance, and is a function that calculates the signed distance [35] between -th link and -th object ; the function is defined as .

The inequality constraints introduced by Eqs. 7 and 6 make the preceding optimization problem highly non-convex and unsolvable by a generic convex solver. In this paper, we approximate it by a sequence of convex problems [35], solved by a sequential convex optimization method.

Iii-D Advantages

As formally derived in the above sections, solving mobile manipulation as trajectory optimization using the proposed VKC-based approach introduces two advantages:

  1. [leftmargin=*,noitemsep,nolistsep,topsep=0pt]

  2. Eliminating unnecessary intermediate goals. Let us use the example of opening a door: Only one goal—the door’s angle to be opened to—is required. The final poses of the mobile base and the manipulator are directly produced during the trajectory optimization process without manually specifying unnecessary intermediate goals. Hence, the VKC-based approach provides versatility and simplicity for modeling mobile manipulation tasks.

  3. Coordinating locomotion and manipulation. Using VKCs, the trajectory optimization jointly generates trajectories of the mobile base and the manipulator, producing coordinated locomotion and manipulation, which is oftentimes challenging for conventional methods.

These two advantages are crucial for a service robot operating in a household environment. Below, we demonstrate these advantages in a series of mobile manipulation tasks.


Iv Motion Planning on Vkcs

In this section, we start with simulation setup, followed by the evaluation of the trajectory-optimization-based motion planning on VKC from three perspectives: (i) the necessary trajectory initialization required by the motion planner, (ii) the improvement on base-arm coordination using the VKC approach compared with traditional setups that have to plan their motions separately, and (iii) the capability of operating in a household environment and performing various tasks.

Iv-a Platform and Simulation Setup

The service robot platform we adopted for testing is a Universal Robot UR5e manipulator mounted on a Clearpath Husky A200 UGA; see Fig. 2b for a graphic illustration. The simulation environment is an arena with in size, discretized into grids. All the experiments are conducted using a desktop with an Intel i7-9700K CPU, running with ROS-Industrial Tesseract [2].

Iv-B Trajectory Initialization

Although a gradient-descent-based algorithm can effectively solve the trajectory optimization problem on VKCs, it may also be easily stuck at local minima near the given initial trajectory [33, 35]. As a result, a proper trajectory initialization is favored to improve the optimization result. Two primary trajectory initialization methods [27] are:

  1. [leftmargin=*,noitemsep,nolistsep,topsep=0pt]

  2. Stationary: The trajectory is initialized by way-points that are the same as the initial pose .

  3. Interpolated: The trajectory

    is initialized by way-points that are linearly interpolated between the initial pose

    and the goal pose (not or ; see below).

In this paper, we further devise an A-based trajectory initialization method, which adopts A to search for a feasible path given the initial and the goal pose of the mobile base. Next, we investigate how different trajectory initialization methods affect the planning results on VKC in three 3D virtual scenarios; see Figs. 2(c), 2(b) and 2(a). The robot’s task is to pick up the rigid stick and use it to reach a target indicated by the red cube. This task consists of three steps: navigate to the stick, manipulate the stick and pick it up, and navigate to and reach the target with the stick. The three scenarios designed for evaluation are in increasing complexity: no obstacle (Fig. 2(a)), two small obstacles (Fig. 2(b)), or a much larger one (Fig. 2(c)). Experimental results reported below are the average of 50 different initial poses, each with 10 times.

(a) Scenario 1
(b) Scenario 2
(c) Scenario 3
(d) success rate
Fig. 3: Comparisons of motion planning on VKCs by different trajectory initialization methods. (a)-(c) The experimental scenarios in increasing complexity. The robot’s initial pose is uniformly sampled within the blue region; it is tasked to pick up the stick and use it to reach the red cube. (d) The proposed A-based trajectory initialization has the highest success rates (almost always) in generating a feasible plan. In comparison, the Stationary method fails to generate feasible plans in scenario 2 and 3. Similarly, the Interpolated method struggles in scenarios 3 ( success rate).

A successfully optimized trajectory is a converged result without violating any constraints (e.g., collisions). Fig. 2(d) compares success rates. When the environment is clean (Scenario 1), even the simplest Stationary trajectory initialization method performs well. When there is additional complexity introduced by the obstacles (Scenario 2), the Stationary method deteriorates, whereas the Interpolated method still maintains a high success rate. When the navigable space is significantly reduced (Scenario 3), only the proposed A-based initialization method can consistently perform well to generate feasible plans. Taken together, experimental results indicate that combining the proposed A-based initialization with the optimization-based motion planner can well handle the challenging motions that require combining navigation and manipulation in cluttered space with obstacle avoidance.


Iv-C Comparisons with Baselines

(a) succ. rate
(b) base effort
(c) mani. effort
(d) time
Task 1: Push to open a door.
(e) succ. case
(f) B1 fail. case
(g) B2 fail. case
Task 2: Pull to open a drawer.
(i) succ. rate
(j) base effort
(k) mani. effort
(l) time
Task 2: Pull to open a drawer.
(h) VKC fail. case
(h) VKC fail. case
(m) succ. case
(n) B1 fail. case
(o) B2 fail. case
(p) VKC fail. case
Fig. 4: Quantitative comparisons between the VKC-based approach and two baselines on two mobile manipulation tasks: push to open a door, and pull to open a drawer. Modeling a mobile manipulation using VKC leads to (a)(i) higher success rate and more effective trajectories in terms of smaller (b)(j) base and (c)(k) manipulator’s efforts, but requires (d)(l) longer planning time. Of note, baselines require additional definition of intermediate goals, which are often empirically found, (e)(m) shown in the green and pink shaded areas. (f)(n) Typical failure cases of two tasks in B1 are due to collision. Typical failure cases of two tasks in (g)(o) B2 or (h)(p) VKC-based approach are due to constraint violations.

We design two mobile manipulation tasks to validate the advantages of the proposed VKC modeling: (i) rotating the doorknob and pushing to open a door (with two revolute joints), and (ii) pulling to open a drawer (with one prismatic joint); see Figs. 3(m) and 3(e). The robot starts from a uniformly sampled initial pose in the blue-shaded regions.

To benchmark the proposed VKC-based approach, we design two baselines that have to individually optimize the mobile base and mobile manipulator’s trajectories. Baseline 1 (B1): Use the proposed A-based trajectory initialization to first plan the mobile base’s motion, and the arm pose is then found by solving the inverse kinematics from the door handle to the mobile base at each way-point. Baseline 2 (B2): On top of B1, the manipulated object and the manipulator’s poses are further refined at each time step to avoid collisions between the mobile manipulator and the environment.

Of note, our VKC-based approach only needs to specify one task goal—the desired door angle or the proper drawer length. In contrast, baselines require additionally specifying the mobile base’s pose when reaching the doorknob and after opening the door, as the base and the manipulator are planned and optimized individually. We compute these intermediate mobile base’s poses by sampling from feasible regions that are empirically found; see the pink areas in Figs. 3(m) and 3(e) for reaching and the green areas for final poses.

We evaluate the planning results by four criteria: (i) the percentage of task completion without violating the constraints as the success rate, (ii) the total base traveling distance as the base’s effort, (iii) the sum of each joint’s accumulated angular displacement during the entire task execution as the manipulator’s effort, and (iv) planning time.

We summarize results in Fig. 4. Planning individually (B1) yields a 48% and a 74% success rate for opening door and drawer, respectively. The primary reason for failure is the collisions between the mobile manipulator and the door/drawer; see Figs. 3(n) and 3(f). Although introducing a collision check to refine motions (B2) improves the success rate, the proposed VKC-based approach significantly outperforms the baselines. The failure cases of B2 are mainly caused by violating velocity and acceleration constraints; see Figs. 3(o) and 3(g).

Planning on VKCs yields more efficient motions with less base and manipulator efforts; see an instance in Fig. 6. Overall, the proposed VKC-based approach produces smoother trajectories with less fluctuation in the speed profile.

Fig. 5: The VKC-based approach is advantageous in modeling and planning household mobile manipulation tasks for a service robot.

(a) Our simulated environments and the corresponding task performances. (b) The success rate, and base and arm efforts measured by their travel distance in 6 different mobile manipulation tasks. The black line segments denote the standard deviations. The proposed

VKC-based approach generally leads to a higher success rate with less base and arm efforts.
Fig. 6: The joint position and velocity profiles of the door-opening task in Fig. 3(e). VKC-based approach generates more efficient trajectory with less fluctuation compared to the baselines.

Taken together, our experimental results demonstrate that by using the proposed VKC-based approach, the planning of mobile manipulator avoids contrived definitions of intermediate poses and achieves a higher success rate with more efficient motions, at the cost of a slightly longer planning time due to solving the trajectory optimization problem with higher DoFs compared to those in the baselines.

Iv-D Complex Tasks

We further test the VKC modeling in a confined household environment with six daily manipulation tasks; see Fig. 5a.

  1. [leftmargin=*,noitemsep,nolistsep,topsep=0pt]

  2. Pick up an object randomly sampled on the coffee table (reachable by the robot) and place it on the dining table. B2 performs the same as the B1 since the object is fixed and does not collide with the robot.

  3. Push to open a narrow door. The robot needs to coordinate its foot and arm movements to open a narrow door (1.1 times wider than the robot) and pass through.

  4. Open a drawer in the corner. Although space is confined, the robot only needs a backward motion, which should be relatively straightforward.

  5. Pull out a dishwasher rack, similar to the above task and also in a confined space. However, it is more challenging as the robot has to manipulate to its side.

  6. Pull to open a refrigerator.

  7. Pull to open a cabinet, similar to the above task. However, since the cabinet has a larger door size, the chair in the environment will block the robot’s movement.

In a successful trial, the robot should produce a collision-free trajectory without violating any kinematic or safety constraints. We compare the results in Fig. 5b with two baselines (B1 and B2). Overall, the proposed VKC-based method demonstrates a higher success rate with less base and arm costs measured by the total distance traveled.

Specifically, in Task

, B1 can hardly find a path to go through the door because of the highly constrained free-space, which requires good foot-arm coordination; task

reveals a similar result. In Task

, the low drawer height increases the difficulty for the mobile base to reach a feasible region; hence, both baselines underperform in terms of their success rate. An interesting observation is in Task


: The baseline methods perform well in the pulling action, likely due to the empty space for the furniture doors to swing.

These results provide ample evidence of excellent foot-arm coordination produced by the proposed VKC-based approach, well-suited for confined household environments.

V Conclusion and Discussion

We presented a modeling method that incorporates the kinematics of a robot’s mobile base, arm, and the manipulated object in VKCs. From this new perspective, a mobile manipulation task is regarded as a motion planning problem on VKCs, solved by trajectory optimization. This approach alleviates the definition of intermediate goals and well coordinates base and arm movements, resulting in a higher success rate with more efficient trajectories in various mobile manipulation tasks. Our simulated experiments validate the advantages introduced by the proposed VKC-based modeling approach, showing its potential in scaling up to complex operations in a household environment [18].

we discuss two issues related to the presented work in greater depth. To deploy the VKC-based method on a physical service robot, objects’ kinematic structures have to be identified. Although some existing methods can achieve this goal by visual sensors [3, 30, 28, 1, 11], results could be error-prone, and a force/torque control framework should be implemented to handle kinematic discrepancies or uncertainties/errors in object positions. Additionally, an effective and stable grasp is a prerequisite of constructing a VKC

connecting the robot and objects. While grasping remains an unsolved problem, more powerful machine learning-based methods are emerging 

[25, 29, 16], which could shed light on this direction.



  • [1] B. Abbatematteo, S. Tellex, and G. Konidaris (2019) Learning to generalize kinematic models to novel objects. In Conference on Robot Learning (CoRL), Cited by: §V.
  • [2] L. Armstrong Optimization motion planning with tesseract and trajopt for industrial applications - ros-industrial. ROS-Industrial. External Links: Link Cited by: §IV-A.
  • [3] P. R. Barragän, L. P. Kaelbling, and T. Lozano-Pérez (2014) Interactive bayesian identification of kinematic mechanisms. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §V.
  • [4] D. M. Bodily, T. F. Allen, and M. D. Killpack (2017) Motion planning for mobile robots using inverse kinematics branching. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A, §I.
  • [5] R. Buchanan, T. Bandyopadhyay, M. Bjelonic, L. Wellhausen, M. Hutter, and N. Kottege (2019) Walking posture adaptation for legged robot navigation in confined spaces. Robotics and Automation Letters (RA-L) 4 (2), pp. 2148–2155. Cited by: §I.
  • [6] F. Burget, A. Hornung, and M. Bennewitz (2013) Whole-body motion planning for manipulation of articulated objects. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A.
  • [7] S. H. Cheong, B. Y. Cho, J. Lee, C. Kim, and C. Nam (2020) Where to relocate?: object rearrangement inside cluttered and confined environments for robotic manipulation. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I.
  • [8] S. Chitta, B. Cohen, and M. Likhachev (2010) Planning for autonomous door opening with a mobile manipulator. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I.
  • [9] S. Gallagher (2006) How the body shapes the mind. Clarendon Press. Cited by: §I.
  • [10] K. Gochev, A. Safonova, and M. Likhachev (2012) Planning with adaptive dimensionality for mobile manipulation. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A.
  • [11] M. Han, Z. Zhang, Z. Jiao, X. Xie, Y. Zhu, S. Zhu, and H. Liu (2021) Reconstructing interactive 3d scenes by panoptic mapping and cad model alignments. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §V.
  • [12] Z. Han, J. Allspaw, G. LeMasurier, J. Parrillo, D. Giger, S. R. Ahmadzadeh, and H. A. Yanco (2020) Towards mobile multi-task manipulation in a confined and integrated environment with irregular objects. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I.
  • [13] P. E. Hart, N. J. Nilsson, and B. Raphael (1968) A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics 4 (2), pp. 100–107. Cited by: §I-A.
  • [14] M. Hoffmann, H. Marques, A. Arieta, H. Sumioka, M. Lungarella, and R. Pfeifer (2010) Body schema in robotics: a review. IEEE Transactions on Autonomous Mental Development 2 (4), pp. 304–324. Cited by: §I.
  • [15] N. P. Holmes and C. Spence (2006) Beyond the body schema: visual, prosthetic, and technological contributions to bodily perception and awareness. Human body perception from the inside out: Advances in visual cognition, pp. 15–64. Cited by: §I.
  • [16] J. Ichnowski, Y. Avigal, V. Satish, and K. Goldberg (2020) Deep learning can accelerate grasp-optimized motion planning. Science Robotics 5 (48). Cited by: §V.
  • [17] A. Jain and C. C. Kemp (2010) Pulling open doors and drawers: coordinating an omni-directional base and a compliant arm with equilibrium point control. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A, §I.
  • [18] Z. Jiao, Z. Zhang, W. Wang, D. Han, S. Zhu, Y. Zhu, and H. Liu (2021) Efficient task planning for mobile manipulation: a virtual kinematic chain perspective. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), Cited by: §V.
  • [19] S. Karaman and E. Frazzoli (2010) Incremental sampling-based algorithms for optimal motion planning. Robotics Science and Systems VI 104 (2). Cited by: §I-A.
  • [20] Y. Karayiannidis, C. Smith, F. E. V. Barrientos, P. Ögren, and D. Kragic (2016) An adaptive control approach for opening doors and drawers under uncertainties. Transactions on Robotics (T-RO) 32 (1), pp. 161–175. Cited by: §I.
  • [21] Y. Karayiannidis, C. Smith, F. E. Vina, P. Ogren, and D. Kragic (2012) “Open sesame!” adaptive force/velocity control for opening unknown doors. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), Cited by: §I.
  • [22] J. J. Kuffner and S. M. LaValle (2000) RRT-connect: an efficient approach to single-query path planning. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A.
  • [23] A. Laurenzi, E. M. Hoffman, M. P. Polverini, and N. Tsagarakis (2019) An augmented kinematic model for the cartesian control of the hybrid wheeled-legged quadrupedal robot centauro. Robotics and Automation Letters (RA-L). Cited by: §I-A.
  • [24] S. M. Lavalle and J. J. Kuffner Jr (2000) Rapidly-exploring random trees: progress and prospects. In Intenational Workshop on Algorithmic and Computational Robotics, Cited by: §I-A.
  • [25] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang, F. Sun, and J. Zhang (2019) Pointnetgpd: detecting grasp configurations from point sets. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §V.
  • [26] N. Likar, B. Nemec, and L. Žlajpah (2014) Virtual mechanism approach for dual-arm manipulation. Robotica 32 (6). Cited by: §I-A, §I.
  • [27] B. Magyar, N. Tsiogkas, J. Deray, S. Pfeiffer, and D. Lane (2019) Timed-elastic bands for manipulation motion planning. Robotics and Automation Letters (RA-L) 4 (4), pp. 3513–3520. Cited by: §IV-B.
  • [28] R. Martín-Martín and O. Brock (2019)

    Coupled recursive estimation for online interactive perception of articulated objects

    International Journal of Robotics Research (IJRR), pp. 1–37. Cited by: §V.
  • [29] A. Mousavian, C. Eppner, and D. Fox (2019) 6-dof graspnet: variational grasp generation for object manipulation. In

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    Cited by: §V.
  • [30] A. Paolillo, K. Chappellet, A. Bolotnikova, and A. Kheddar (2018) Interlinked visual tracking and robotic manipulation of articulated objects. Robotics and Automation Letters (RA-L) 3 (4), pp. 2746–2753. Cited by: §V.
  • [31] J. Pratt, C. Chew, A. Torres, P. Dilworth, and G. Pratt (2001) Virtual model control: an intuitive approach for bipedal locomotion. International Journal of Robotics Research (IJRR) 20 (2), pp. 129–143. Cited by: §I-A.
  • [32] J. Pratt, P. Dilworth, and G. Pratt (1997) Virtual model control of a bipedal walking robot. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A.
  • [33] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa (2009) CHOMP: gradient optimization techniques for efficient motion planning. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A, §IV-B.
  • [34] C. Rösmann, F. Hoffmann, and T. Bertram (2017) Kinodynamic trajectory optimization and control for car-like robots. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), Cited by: §III-C.
  • [35] J. Schulman, Y. Duan, J. Ho, A. Lee, I. Awwal, H. Bradlow, J. Pan, S. Patil, K. Goldberg, and P. Abbeel (2014) Motion planning with sequential convex optimization and convex collision checking. International Journal of Robotics Research (IJRR) 33 (9), pp. 1251–1270. Cited by: §I-A, §III-C, §III-C, §IV-B.
  • [36] K. Shankar (2016) Kinematics and local motion planning for quasi-static whole-body mobile manipulation. Ph.D. Thesis, California Institute of Technology. Cited by: §I.
  • [37] R. Smits, H. Bruyninckx, and E. Aertbeliën (2011) KDL: kinematics and dynamics library. Note: Cited by: §II.
  • [38] A. Stentz (1997) Optimal and efficient path planning for partially known environments. In Intelligent unmanned ground vehicles, pp. 203–220. Cited by: §I-A.
  • [39] M. Stuede, K. Nuelle, S. Tappe, and T. Ortmaier (2019) Door opening and traversal with an industrial cartesian impedance controlled mobile robot. In Proceedings of International Conference on Robotics and Automation (ICRA), Cited by: §I-A.
  • [40] M. Toussaint, K. Allen, K. A. Smith, and J. B. Tenenbaum (2018) Differentiable physics and stable modes for tool-use and manipulation planning. In Proceedings of Robotics: Science and Systems (RSS), Cited by: §I-A.
  • [41] J. Wang, S. Lin, C. Hu, Y. Zhu, and L. Zhu (2020) Learning semantic keypoint representations for door opening manipulation. Robotics and Automation Letters (RA-L) 5 (4), pp. 6980–6987. Cited by: §I.
  • [42] Y. Wang, C. Smith, Y. Karayiannidis, and P. Ögren (2015) Cooperative control of a serial-to-parallel structure using a virtual kinematic chain in a mobile dual-arm manipulation application. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), Cited by: §I-A.
  • [43] Y. Wang, C. Smith, Y. Karayiannidis, and P. Ögren (2016) Whole body control of a dual-arm mobile robot using a virtual kinematic chain. International Journal of Humanoid Robotics 13 (01). Cited by: §I-A.