Log In Sign Up

Differentiable Robotic Manipulation of Deformable Rope-like Objects Using Compliant Position-based Dynamics

Robot manipulation of rope-like objects is an interesting problem that has some critical applications, such as autonomous robotic suturing. Solving for and controlling rope is difficult due to the complexity of rope physics and the challenge of building fast and accurate models of deformable materials. While more data-driven approaches have become more popular for finding controllers that learn to do a single task, there is still a strong motivation for a model-based method that could be used to solve a large variety of optimization problems. Towards this end, we introduced compliant, position-based dynamics (XPBD) to model rope-like objects. Using geometric constraints, the model can represent the coupling of shear/stretch and bend/twist effects. Of crucial importance is that our formulation is differentiable, which can solve parameter estimation problems and improve the matching of rope physics to real-life scenarios (i.e., the real-to-sim problem). For the generality of rope-like objects, two different solvers are proposed to handle the inextensible and extensible effects of varied material stiffness for the rope. We demonstrate our framework's robustness and accuracy on real-to-sim experimental setups using the Baxter robot and the da Vinci research kit (DVRK). Our work leads to a new path for robotic manipulation of the deformable rope-like object taking advantage of the ready-to-use gradients.


page 1

page 7

page 9

page 10


Static Stability of Robotic Fabric Strip Folding

Planning accurate manipulation for deformable objects requires predictio...

MORPH-DSLAM: Model Order Reduction for PHysics-based Deformable SLAM

We propose a new methodology to estimate the 3D displacement field of de...

Symplectic Integration for Multivariate Dynamic Spline-Based Model of Deformable Linear Objects

Deformable Linear Objects (DLOs) such as ropes, cables, and surgical sut...

Self-propulsion on spandex: toward a robotic analog gravity system

Numerous laboratory systems have been proposed as analogs to study pheno...

Active Perception and Modeling of Deformable Surfaces using Gaussian Processes and Position-based Dynamics

Exploring and modeling heterogeneous elastic surfaces requires multiple ...

An Inextensible Model for Robotic Simulations of Textiles

We introduce a new isometric strain model for the study of the dynamics ...

I Introduction

For problems in grasping soft objects, modeling and simulation of deformable objects have recently been considerable interest in many robotic applications [12, 8], to manipulating soft tissue [23, 9], cloth [10, 42], and even fluids [13, 38]. Among soft structures, deformable linear objects (DLOs), including rope-like objects, strings, cables, beams, etc., are studied (cable routing [30], wire insertion [19], flexible rope [47], and knotting of surgical thread [24]). Recently, techniques involving visual servoing [50, 49], latent dynamics learning [46], and adaptive estimation [45], have all be explored for controlling DLOs. Further reviews in this space can be found in review papers on the topic [44, 37, 48].

Classical methods for modeling rope reside from Cosserat rod theory [7]

which involves an analytical, partial differential equation representing the rope as a continuous 3D-dimensional curve exhibiting both bending and torsion. However, the analytical dynamics are limited by the computational efficiency and stability of solving this PDE with two-point boundary conditions (given by the start and end of the rope).

Alternative to exact analytical models, a model-free method may try to identify representations of rope using purely visual descriptors from a camera without involving classical mechanics. These model-free representations include curvature and Fourier-based shapes [31, 32]

, as well as neural network features

[39, 15, 40]. Instead of directly modeling physical mechanics, the state space representation is embedded in these latent space features. However, these approaches need a large quantity and variety of data for training, which is not easy to get in field robotic applications (either in simulation or real life). Meanwhile, the collected datasets are limited to the distribution of objects manipulated. Thus models that learn from simulation or limited real-world data encounter a real-to-sim transfer challenge or an over-fitting problem, respectively.

Fig. 1: The physical experiments and simulation results for the shape control task of inextensible rope on Baxter | The top row shows the simulation result for our shape control task of inextensible rope on the Baxter, rendered in the OpenGL. And the bottom rope is our physical result for this control task in the real scene.

An increasing number of simulators are made available for robot learning involving rope-like objects [18, 20]. These simulators have been focused on their use for data generation and lack a method for integration in a model-based control context.In general, many existing robotics simulators (including the above, as well as Bullet [3]) and Gym [2], unable to simulate soft bodies) only support gradient calculations based on finite differences. However, differentiable physics and simulation have become critical features for the robot learning community [11]. The works in [5, 33, 41, 22] have applied differentiable framework into rigid articulated body simulation, while others have done the same for deformable objects [16, 6, 14]. These methods provide gradients within the differentiable framework, where optimization tasks for modeling and control are natively supported. Meanwhile, the differentiability can be easily deployed into neural layers for learning-based methods, such as in [34, 26].

One simulation approach that has gained significant interest due to its speed, stability, and capability to model the spectrum from rigid to soft to liquid is Position-based dynamics (PBD) [27, 28]. Unlike the traditional Lagrangian force-based method, the geometric position constraints are solved iteratively. Simulating soft objects involves utilizing the Compliant Position-based Dynamics (XPBD) method [29] which has been shown for a variety of deformable solid bodies, cloth, and chains. Some researchers have directly integrated the Cosserat model into PBD simulators to define the twist and bend, stretch and shear the a the constraints, by introducing particles with orientation information to describe angular updates [17]. Next, the authors added scale parameters for rope-like objects to preserve volume consistency [1]. Although several works have been applied for real-to-sim tasks of soft deformable tissue [23, 9], rigid articulated robots [21], and fluids [13, 35], with constraint-based formulation of PBD or XPBD. However, there is a missing combination of modeling and control approaches applied to rope-like objects in literature, exploiting the capability of XPBD.

In the paper, we propose a solution to formulate a real-to-sim modeling and control framework. We extended the original position-based dynamics (PBD) for rope-like objects proposed in [17] with a compliance parameter by following the XPBD work in [29]

. Meanwhile, it is deployed differently using the automatic differentiation functionality available in PyTorch for gradient updates. The new compliant position-based dynamics (XPBD) for rope-like objects can guarantee stable forward simulation while the back-propagation of losses can be applied for modeling and control tasks and parameter identification. The contributions of this paper are as follows:

  • We describe rope-like objects as a compliant position-based dynamics (XPBD) model by constructing geometrical constraints defining their behaviors.

  • We introduced a differentiable framework for modeling and simulation that works well with auto-differentiation and algorithm configurations for constraint solving.

  • We defined the problem of parameter identification and manipulation control of rope-like objects in real-to-sim context (Fig. 1).

  • We validate our methods in real-world experiments using robot manipulators (Baxter) and surgical robots (dVRK).

Ii Methods

Ii-a Compliant Position-Based Dynamics (XPBD) Modeling of Rope-like Object

Fig. 2: Proposed geometric model of rope-like objects | The discrete particles include orientation representation of the rope-like object for modeling with compliant position-based dynamics (XPBD).

The XPBD method will be a foundation for building the differentiable model for DLOs. We firstly discretize the DLO into a sequence of particles (Fig. 2) with Cartesian coordinates . Meanwhile, quaternions are used to describe orientations in-between adjacent particles , and will be used to solve the bending and twist deformation of the DLO. Unlike force-based methods such as Euler-Bernoulli beam or Cosserat rod theory, full strain and torsion deformations can be updated with a position-based dynamics solver. As with all simulations using PBDs, the methods starts with a list of constraints, that describe the dynamics of particles. Solving the constraints involves updating and directly using a non-linear projected Jacobi method used for constrained optimization problems. The method to solve for the gradient updates is derived below.

Constraints can be linearized by Taylor series expansion,




where the Lagrange multiplier change can be found by introducing the compliance parameter [29],


where and are the weighted terms to guarantee the conservation of linear and angular momentum. Generally, it refers to the mass/inertia matrix as and . For simplification of DLOs, we use uniformed scalar weights instead of matrices for each dimensionality, which has not much impact on simulation and dynamical performance but reduces computational load. That is,


Next, we will introduce several geometrical constraints to simulate the behaviors of the DLOs.

Ii-A1 Shear and Stretch Constraint

According to Cosserat theory, the shear and stretch measures the deformation regarding the tangent direction of the rope-like object. Therefore, the stretch/compressed length should be constrained relative its rest pose, which indicates in-extensible elasticity. Simultaneously, the normal direction (i.e., the rotated from world frame noted by ) for each cross-section should be parallel to the tangent direction of object’s centerline, see Fig. 2. It measures the shear strain with respect to non-deformed states. Thus, for each pair of neighboring particles, the shear-stretch deformation can be integrated into a generalized constraint as, , which is,


where is the rotation matrix from the local frame of line segment to world frame represented using quaternion, represent the states in rest pose. According to gradient calculation in [17], we can obtain,



is the skew-symmetrical matrix representation of a vector.

Ii-A2 Bend and Twist Constraint

In differential geometry, the Darboux vector is used to parameterize strain deformation with respect to frame rotation. According to Cosserat theory, the Darboux vector can be expressed as a quaternion by measuring the rod’s twist in the tangent direction. Thus, the difference between the current and resting configuration should be evaluated, i.e. . According to [17], the bend and twist constraint can be computed for each pair of two adjacent quaternions (shown in Fig. 2) by, , which is,


where is the imaginary part of the quaternion and is the conjugate quaternion. The constraint gradients can be calculated as,


Ii-A3 Distance Constraint

One property of modeling ropes is that they can be considered either extensible or inextensible based on how stiff they are, and ultimately this can be modeled as a constraint. Although the above strain deformation have considered the inextensible property implicitly, the iterative solver in Eq. 2 cannot guarantee that all the constraints will be satisfied. Therefore, we can explicitly consider the distance constraint for an enhanced inextensible chain structure using only discretized particle positions. We define by,


The gradients can be easily obtained by,


Ii-B Real-to-Sim Parameter Identification

Given the above constraints, the dynamics of a rope-like object can be represented by a set of discretized particles (shown in Fig. 2) with position and orientation evolution. However, it is just an approximation of the real model for the rope objects.

For the above shear/shear, bend/twist and distance constraints, we can introduce the additional stiffness parameter to weight the updates during iteration steps, namely by,


Which stands for shear/stretch, bend/twist, and distance constraints regarding position or orientation. Moreover, stands for position changes due to external gravity, and stands for the successive over-relaxation parameter and is applied to accelerate the convergence speed further. Thus, we can introduce the above parameters in a real-to-sim setup by inserting them into the following differentiable framework.

Ii-C Differential Framework

To perform an optimal control task, the ready-to-use gradients will be needed to compute. We introduce the compliant position-based dynamics (XPBD) inside a differentiable framework, as shown in Algorithm 1 111In this paper, we only consider the quasi-static dynamical states. Only the gravity will be regarded as without any external torques. Thus, the Euler prediction and integration of velocity and angular speed will not be involved.

. We rely on the automatic differentiation function provided by PyTorch to obtain the gradients. It can be natively integrated into optimization or learning methods, since the coding framework can be viewed as a differentiable layer which support both forward and backpropagation operations. The computing memory might be limited by the number of iterations for substep simulation. Denoting the gradient variable at time

by , we can formulate the following optimization problem as,


where , and are the position-based constraints, and

is the loss function derived from the system states.

represents the selected gradient variable, which can be control states, system parameters or system states.

// Initialize the gradients variable
// Position states Euler prediction
// Constraints solving loop
3 while iter iterations do
       // Apply shear/stretch constraints using Eq. 5
4       ,
       // Apply bend/twist constraints using Eq. 7
       // Apply distance constraints using Eq. 9
       // Update constraints changes
// Obtain the loss function
// Calculate the gradients
12 return
Algorithm 1 Differentiable Framework for Rope-like Objects

Iii Real-to-Sim and Shape Control Problem Setup

In this part, we used the proposed differentiable framework to conduct three different experiments both on Rethink Baxter and the da Vinci Research Kit (dVRK) robotic platforms. The Baxter represents a situation of more significant rope manipulation, whereas the dVRK represents a situation of surgical automation involving blood vessel manipulation. We considered the Baxter-rope experiment to involve an inextensible DLO, while the dVRK blood-vessel experiment to be an extensible DLO. For both setups, we looked at both a real-to-sim problem (i.e., parameter estimation based on observations from the real world), and a control problem (i.e., using the XPBD model we developed to have rope configuration reach a target configuration iteratively). The implementation of each experiment will be provided later.

Iii-a Solver Setups

In classical PBD solver [27], the Jacobi approach averages each constraint step changes, and keeps updating iteratively. The convergence compromises between the number of iterations and the satisfaction of other constraints. However, it was not suitable for simulation of the inextensible effects for ropes with significant axial stiffness.

Precisely, we needed to guarantee the fulfillment of the distance constraint in Eq. 9 completely. A direct linear solver was proposed [43] based on the tridiagonal matrix algorithm, i.e., Thomas algorithm, to preserve the inextensible characteristics of rope-like objects. We will refer to methods individually as the Jacobi XPBD and Thomas XPBD, respectively. All other constraints, such as shear/stretch and bend/twist constraints, used the Jacobi method.222Thus, in this paper, when Jacobi XPBD and Thomas XPBD are indicated, it is distinguishing how the distance constraint is solved.

Iii-B Data Preprocessing

In our experiments, we obtained both the point cloud and RGB image data using the Microsoft Azure Kinect. For the 2D images obtained, we extracted the centerline of the rope using MATLAB skeletonization function333

Since the raw point cloud data was noisy, we projected them to a hyperplane defined by the gravity vector and two endpoints of the rope, as shown in Fig.

3. The projected point cloud was constrained within a plane, which was easier for identification of bend/twist effects. We used the de-noised 2D centerline from the image and projected 3D point clouds as ground truth data for loss computation in the following section.

Fig. 3: Projecting captured rope point cloud to 2D plane | Front view (left) and top view (right) for the original point cloud (blue) and projection point cloud (orange). The red arrow is the gravity vector. We projected the point cloud onto the plane defined by the gravity vector and two rope endpoints.

Iii-C Loss Functions

Fig. 4: Visualization of different position-based losses for shape matching | The rope was simulated with the particle-based method and represented by connected lines. Curve dotted line was the simulated rope, discretized into particles in black points. is line for neighbouring particles, such as from particle to particle The preprocessed real data (representing 3D point cloud projected along gravity or 2D centerline) were split into different segments (i.e. , , etc). Points of real data are in blue, and the start/endpoints (i.e., , ) of each segment group set are in blue and circled in red. The solid blue lines indicate the minimum distance from the point cloud to the simulated rope particles. The dashed-point red lines indicate correspondence of each segment’s start/endpoints to the simulated rope particles.

We defined four types of primary losses to investigate for shape matching. We looked at the combinations of these losses for ablation, resulting in 9 different aggregate loss functions for optimization. The loss functions are shown in Table I.

  • Point-to-Line : As shown in Fig. 4, we intended to find the minimum distance from each preprocessed real data point , to the line segments between each pair of adjacent simulated particle . Thus, the primary loss regarding point-to-line loss was


    where we discretized the rope to particle nodes, along preprocessed real data points, and was the projected locations on each line segment with minimum distance.

  • Segment-to-Particle : We splitted the real data points (preprocessed for noises and re-ordered) into different segments along the vector formulated by two rope endpoints, i.e. , etc shown in Fig. 4. The start/endpoints (i.e., , ) of each segment were determined according to the proportional length to the whole data set. We evaluated the distance between the start/endpoints of each segment and the corresponding particle in the simulation. Then, the segment-to-particle loss was,

  • Segment-to-Line : As above, we already divided the real data into segments. The projection distance between all points in each segment set () and the corresponding line segment () could be used to evaluate the loss. Thus, the segment-to-line loss was,


    where was the number of real data points for each segment group set, and was the projected locations on the line segment with minimum distance.

  • Lowest 3D Point Along Gravity Vector : The lowest point along the gravity direction of the denoised real data was compared with the one in simulation. This could compensate for lacking depth information while using the 2D centerline. The loss was defined as,


In our experiments, the point-to-line, segment-to-particle, and segment-to-line losses could be applied to either the 3D point cloud or 2D centerline. We performed 9 different combinations of types of losses as shown in Table I. The loss function , , contained 3D information only, while , , were considering 2D cases only. , , were applied to both 2D cases and the included 3D lowest point with depth information.

Meaning Point-to-Line for 3D and 2D Segment-to-Particle for 3D and 2D Segment-to-Line for 3D and 2D
TABLE I: Loss type | We used nine types of loss functions for our differential framework and made a comparison over these loss functions.

Iv Experiments And Result Analysis

Iv-a Inextensible Rope Parameter Identification

Parameter identification and position estimation of the control point were carried out on an inextensible rope manipulated by the Baxter robot. We fixed one endpoint of the rope to the environment, and the other endpoint of the rope to the manipulator end-effector (referred as the control point). We continuously moved the control point and collected 36 frames of point cloud for the rope deformation. The Microsoft Azure Kinect was used to obtain the 2D RGB image frames for centerline extraction at the same time. Since only translational movement (fixed rotation angle) of end-effector is controlled, the point cloud had occlusions and was not fully visible at the local region near the control point. We applied the keypoint-based kinematics reconstruction from [25] to identify the 3D position of the control point. Therefore, the ground truth data for 3D point cloud and 2D centerline, as well as control point were ready for real-to-sim transfer.

In XPBD simulation, considering the computational cost and stiffness of the rope, the number of particle were set by . was the control point and corresponding to the endpoint was fixed to the environment. According to Section II-B, there were six constraint stiffness parameters needed to estimate, as shown in Table II. Because 3D point cloud could be more informative than 2D centerline which lacked depth data, the parameters were optimized with using the 3D point-to-line loss. We then estimated the position of the control point (i.e. , see Algorithm 1) using all loss functions in Tab. I to evaluate the accuracy of parameter identification.

For parameter identification, a single gravity parameter was inferred based on Thomas solver, the comparison of simulated results before and after parameter optimization was shown in Fig. 5. After parameter optimization, the simulation states was approximately approaching to the ground truth point cloud. As for the Jacobi solver, it needed two sets of gravity stiffness to deal with different convergency of distance constraint for varied rope states (bending and tensioned), as shown in Table. III.

Parameter Initial Optimized Meaning
0.04 0.024 position stiffness (gravity)
1.0 0.48 position stiffness (distance)
1.0 1.16 position stiffness (shear/stretch)
1.0 0.52 quaternion stiffness (shear/stretch)
1.0 1.49 quaternion stiffness (bending/twist)
1.0 0.61 successive over-relaxation weight
TABLE II: Parameter estimation result for inextensible rope | The parameter estimation result of the differentiable Thomas XPBD solver for the inextensible rope.
Fig. 5: Parameter optimization results for inextensible rope (i.e., real-to-sim result) | The bending (left) and the tensioned (right) status of the simulated rope before and after parameter optimization. PCL, in red, is the real point cloud that the simulation is trying to match up to.

For position estimation of control point, we made a comparison over the nine kinds of loss functions shown in Table I firstly. The 3D point cloud included the rope information in 3D space, while the 2D centerline did not contain depth information. As a result, the simulation result containing the 3D information showed the best performance, i.e., , , and as shown in the Fig. 7. For , , and , whose loss function consisted of 2D information and partial 3D information (the lowest point along gravity direction), the simulation result was improved compared to , , and , which only considered 2D information. As shown in Fig. 7 and Fig. 6 , obtained the best performance among all these losses. It simply proved the accuracy of parameter identification, and showed that the 3D data contributed more than 2D cases for control point estimation.

Fig. 6: Losses on inextensible rope as a function of objective |   By using different loss functionsI, we obtained nine kinds of simulation results for all frames. We used OBJ1 to evaluate the deviation caused by different loss functions. The simulation result got from has the smallest deviation compared with the ground truth one.
Fig. 7: Deformation estimation for inextensible rope using different loss functions | From Right to Left and from Top to Bottom: The simulation result for 9 different loss functions based on Thomas XPBD solver. Blue lines represented ground truth provided by the point clouds. Red lines represented the optimized result regarding different loss functions.
Fig. 8: Deformation estimation for inextensible rope on Baxter | Each Column: (1) Raw RGB image. (2) The simulation result based on Thomas XPBD, with good convergency for both bending and tensioned status. The simulation result with gravity stiffness of (3) (better convergency for bending status and the length of tensioned rope in the middle exceeds the normal length) and (4) (better convergency for tensioned status and the elasticity of the bending ropes in the top and bottom fail to reach the expected.) from Jacobi XPBD solver. Blue ropes represented ground truth provided by the point clouds. Orange ropes represented the optimized deformation of the rope. For the light occlusion, we used keypoint detection[25] and point cloud instead of Aruco Marker for the location identification of end-effector and fixed point respectively.
Fig. 9: Comparison of Thomas and Jacobi XPBD solver on inextensible rope | Control point deviation and rope length regarding different frames of simulation using Thomas XPBD and Jacobi XPBD.
Fig. 10: Parameter identification for extensible rope on dVRK | This experiment shows the simulation result with different . From left to right: (1) Original RGB image from Azure Kinect. (2) Both sides were . (3) Both sides were . (4) The relax side was and the extensible (tensioned) side was . The simulation result got from (4) was the closest to the ground truth.
Fig. 11: Deformation identification and key points estimation for extensible rope on dVRK | Top: Original RGB image from Azure Kinect. bottom: Simulation result for the extensible rope by using Jacobi XPBD solver. Orange ropes represented the optimized deformation of the rope. The manipulation policy from the differential framework made the simulated result approaches the ground truth one. The right red marker is the control point and the left one is the reference marker.

Fig. 12: Trajectory and Deviation of control/reference points for extensible rope | Left: The pixel deviation of the control point and a reference point in 2D image regarding frames. Right: Control point and reference points trajectory for the real case and the simulation one.

We also compared the performance of Thomas XPBD and Jacobi XPBD when estimating the position of the control point. The middle plot in Fig. 13 showed the rope length change of the simulated result from 36 frames. The average length deviation over the ground truth length for Thomas XPBD solver was for all structures, but for Jacobi solver, it was . Especially when the rope became tense, the error for Jacobi solver became more significant. The same phenomenon occurred on the error of the control point’s position in Fig. 9. The left and right plots in Fig. 13 indicated that the distance constraint in the Jacobi XPBD was hard to be satisfied when the rope was approaching the tensioned status. The Jacobi XPBD intuitively tried to comprise the satisfaction of each constraint solving. It iteratively struggled to maintain the stretch, twist, and distance against gravity. Thus, it resulted in a no-complete convergency of distance constraint . For Thomas XPBD, the distance constraint would be satisfied between each neighboring node simultaneously within one step solving, which was similar as the position update mechanism of the gravity. In this case, Thomas XPBD solver ensured that the length of rope was unchanged, and it was more suitable for the simulation of inextensible rope. The simulation result shown in Fig. 8 proved our conclusion.

Iv-B Extensible Rope Parameter Identification and Key Points Estimation

Parameter identification and position estimation of control point were carried out on an extensible, flexible silicone rod using the dVRK surgical robot. The silcone rod was chosen to resemble a compliant and dissected vessel, a technique that is often used in surgery to avoid damaging the vessel. The two endpoints of the rope were fixed, and the control point was in the middle of the rope. We moved the control point to collect 30 frames of point cloud data. In simulation, the number of particle nodes were . and corresponding to both the endpoints were fixed and was the control point. We did not need to preserve the inextensible behavior completely since the silicon rope had a lower axial stiffness. Thus, we only implemented the Jacobi XPBD solver for the distance constraint in this experiment. The loss function used was in Table I, and we estimated the same parameters as the ones indicated in Section II-B. After parameter estimation, we estimated the position of the control point (i.e. ) using and , i.e., point-to-line for both 3D and 2D cases. For parameter identification, we used grid-search to identify the optimized six parameters in Table III for extensible rope. Since the control point was set at the middle of the rope, manipulation could result in one side being tight and the other being loose. Thus, the effect of the gravity over two sides was different, and we needed to define different gravity weights for the two sides, as shown in Fig. 10.

Parameter Inextensible rope extensible rope
0.26 (bending) / 0.001 (tensioned) 25 (relax) / 3 (tensioned)
1.19 0.87
0.80 0.873
1.19 1.0
0.80 1.30
0.79 0.61
TABLE III: Parameter estimation result for Jacobi solver on inextensible and extensible rope | The parameter estimation result of the differentiable Jacobi XPBD solver for the inextensible and extensible rope.
Fig. 13: Simulation result analysis for distance constraint of inextensible rope on Baxter | (1) The evolution of maximal distance constraint value of all nodes (i.e., ) regarding iterations. (2) The evolution of rope length regarding iterations. (3) The value of each pair of neighbouring nodes of the last iteration. Both the tensioned and bending status are shown based on differentiable Thomas solver and Jacobi solver of XPBD simulation.( for Jacobi XPBD solver.)

With the optimized parameters, we estimated the position of the control point and a reference point, namely by key points. Apart from the control point, there was a reference point with a red marker, shown in Fig. 10. The marker was used only for comparison between ground truth trajectory and simulation result in Fig. 12 Right and not for parameter estimation. Fig. 12 Left showed the error curve of the location of reference and control point. Even though the raw data of the extensible rope was noisier, the average error of the control point and the reference point was acceptable, which proved the robustness of our solver. In Fig. 11, we showed the original image and simulation result for different frames, and no matter how the rope deformed, our solver could get a desirable outcome. Even though Thomas solver outperformed the Jacobi solver in the previous experiment, Thomas XPBD was too tough to simulate the silicon rope. Thomas XPBD required the length of the rope to remain unchanged, which contradicted the silicon rope’s extensible property. Thus, the Thomas solver diverged for the simulation, and the Jacobi solver was more suitable.

Fig. 14: Shape control for inextensible rope on Baxter | Top Row: The ideal shape control task by identified parameters. Bottom Row: The not-ideal cases. The target control shape for ropes are in low opacity, while our control results are in solid opacity.

Iv-C Inextensible Rope Shape Control

Based on the Thomas XPBD solver, the shape control task was carried out on the same Baxter setup as the Section IV-A. The ground truth shape of the rope was obtained by locating the left control point (end-effector), right fixed endpoint and the middle three Aruco Markers shown [36] in the Fig. 1 and Fig. 14. We fixed the simulated rope to the same right endpoint. By setting the middle three points as the shape target, we optimized the position of the control point from the differentiable Thomas XPBD simulation.

As shown in Fig. 14, the transparent rope with low opacity was our shape control target status, and the solid ones with high opacity were the result controlled by the identified parameters from Section IV-A. Our controlled shape could almost overlap with the target rope, as shown in the Top Row of Fig. 14. However, some had deviations as shown in the Bottom Row of Fig. 14. One of the reason is we only considered three shape points as the target for loss computation. The accuracy will be improved by considering more discretized segments of the ground truth data. Another reason was because we did not consider the rotational control of the end-effector when solving the inverse kinematics of the Baxter, which caused the rope not to be able to move to the desired position perfectly.

V Conclusion and Future Works

This paper used a compliant position-based framework to conduct the differential real-to-sim tasks for parameter identification and shape control tasks. Several geometrical constraints were introduced to model the rope-like objects’ coupling stretch/shear and bending/twisting effects. To inspect the inextensible and extensible impact, the Thomas solver and the Jacobi solver are proposed for the distance constraint. The experiment results on the Baxter robot and DVRK platform proved the validity and robustness of our solvers. The shape control tasks showed a novel path for real-to-sim robotic manipulation operations.

The future works will consider differentiable control with collision handling and rigid-deformable coupling. Meanwhile, more advanced tasks will be considered, such as surgical thread manipulation in field environments using the proposed inextensible solver.


Many thanks to Yutong Zhang for the rendering of the cover photo. This work was supported by NSF CAREER award 2045803 and the US Army Telemedicine and Advanced Technologies Research Center.


  • [1] B. Angles, D. Rebain, M. Macklin, B. Wyvill, L. Barthe, J. P. Lewis, J. von der Pahlen, S. Izadi, J. P. C. Valentin, S. Bouaziz, and A. Tagliasacchi (2019) VIPER: volume invariant position-based elastic rods. CoRR abs/1906.05260. External Links: Link, 1906.05260 Cited by: §I.
  • [2] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba (2016) Openai gym. arXiv preprint arXiv:1606.01540. Cited by: §I.
  • [3] E. Coumans and Y. Bai (2016–2021)

    PyBullet, a python module for physics simulation for games, robotics and machine learning

    Note: Cited by: §I.
  • [4] C. D’Ettorre, A. Mariani, A. Stilli, F. Rodriguez y Baena, P. Valdastri, A. Deguet, P. Kazanzides, R. H. Taylor, G. S. Fischer, S. P. DiMaio, et al. (2021) Accelerating surgical robotics research: a review of 10 years with the da vinci research kit. IEEE Robotics and Automation Magazine. Cited by: Differentiable Robotic Manipulation of Deformable Rope-like Objects Using Compliant Position-based Dynamics.
  • [5] F. de Avila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z. Kolter (2018) End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31, pp. . External Links: Link Cited by: §I.
  • [6] T. Du, K. Wu, P. Ma, S. Wah, A. Spielberg, D. Rus, and W. Matusik (2021) DiffPD: differentiable projective dynamics. External Links: 2101.05917 Cited by: §I.
  • [7] M. Gazzola, L. Dudte, A. McCormick, and L. Mahadevan (2018) Forward and inverse problems in the mechanics of soft filaments. Royal Society open science 5 (6), pp. 171628. External Links: Document, Link Cited by: §I.
  • [8] Y. Han, R. Batra, N. Boyd, T. Zhao, Y. She, S. Hutchinson, and Y. Zhao (2021) Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer. CoRR abs/2112.06374. External Links: Link, 2112.06374 Cited by: §I.
  • [9] Y. Han, F. Liu, and M. C. Yip (2021) A 2d surgical simulation framework for tool-tissue interaction. External Links: 2010.13936 Cited by: §I, §I.
  • [10] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg (2020) VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation. In Robotics: Science and Systems Current file Overview 8 (RSS), Cited by: §I.
  • [11] Y. Hu, L. Anderson, T. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand (2020) DiffTaichi: differentiable programming for physical simulation. External Links: 1910.00935 Cited by: §I.
  • [12] I. Huang, Y. Narang, C. Eppner, B. Sundaralingam, M. Macklin, T. Hermans, and D. Fox (2021) DefGraspSim: simulation-based grasping of 3d deformable objects. External Links: 2107.05778 Cited by: §I.
  • [13] J. Huang, F. Liu, F. Richter, and M. C. Yip (2021) Model-predictive control of blood suction for surgical hemostasis using differentiable fluid simulations. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 12380–12386. External Links: Document Cited by: §I, §I.
  • [14] Z. Huang, Y. Hu, T. Du, S. Zhou, H. Su, J. B. Tenenbaum, and C. Gan (2021) PlasticineLab: a soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, External Links: Link Cited by: §I.
  • [15] S. Huo, A. Duan, C. Li, P. Zhou, W. Ma, and D. Navarro-Alarcon (2021) Keypoint-based bimanual shaping of deformable linear objects under environmental constraints using hierarchical action planning. CoRR abs/2110.08962. External Links: Link, 2110.08962 Cited by: §I.
  • [16] K. M. Jatavallabhula, M. Macklin, F. Golemo, V. Voleti, L. Petrini, M. Weiss, B. Considine, J. Parent-Levesque, K. Xie, K. Erleben, L. Paull, F. Shkurti, D. Nowrouzezahrai, and S. Fidler (2021) GradSim: differentiable simulation for system identification and visuomotor control. External Links: 2104.02646 Cited by: §I.
  • [17] T. Kugelstadt and E. Schömer (2016-07) Position and orientation based cosserat rods. pp. . External Links: Document Cited by: §I, §I, §II-A1, §II-A2.
  • [18] R. Laezza, R. Gieselmann, F. T. Pokorny, and Y. Karayiannidis (2021) ReForm: a robot learning sandbox for deformable linear object manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 4717–4723. External Links: Document Cited by: §I.
  • [19] R. Lagneau, A. Krupa, and M. Marchal (2020) Automatic shape control of deformable wires based on model-free visual servoing. IEEE Robotics and Automation Letters 5 (4), pp. 5252–5259. External Links: Document Cited by: §I.
  • [20] X. Lin, Y. Wang, J. Olkin, and D. Held (2020)

    SoftGym: benchmarking deep reinforcement learning for deformable object manipulation

    In Conference on Robot Learning, Cited by: §I.
  • [21] F. Liu, M. Li, J. Lu, E. Su, and M. C. Yip (2022) Parameter identification and motion control for articulated rigid body robots using differentiable position-based dynamics. External Links: 2201.05753 Cited by: §I.
  • [22] F. Liu, M. Li, J. Lu, E. Su, and M. C. Yip (2022) Parameter identification and motion control for articulated rigid body robots using differentiable position-based dynamics. External Links: 2201.05753 Cited by: §I.
  • [23] F. Liu, Z. Li, Y. Han, J. Lu, F. Richter, and M. C. Yip (2021) Real-to-sim registration of deformable soft tissue with position-based dynamics for surgical robot autonomy. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 12328–12334. External Links: Document Cited by: §I, §I.
  • [24] B. Lu, W. Chen, Y. Jin, D. Zhang, Q. Dou, H. K. H. Chu, P. Heng, and Y. Liu (2020) A learning-driven framework with spatial optimization for surgical suture thread reconstruction and autonomous grasping under multiple topologies and environmental noises. CoRR abs/2007.00920. External Links: Link, 2007.00920 Cited by: §I.
  • [25] J. Lu, F. Richter, and M. Yip (2021) Pose estimation for robot manipulators via keypoint optimization and sim-to-real transfer. External Links: 2010.08054 Cited by: Fig. 8, §IV-A.
  • [26] P. Ma, T. Du, J. Z. Zhang, K. Wu, A. Spielberg, R. K. Katzschmann, and W. Matusik (2021)

    DiffAqua: a differentiable computational design pipeline for soft underwater swimmers with shape interpolation

    ACM Transactions on Graphics (TOG) 40 (4), pp. 132. Cited by: §I.
  • [27] M. Macklin, M. Müller, and J. Bender (2017) Position-based simulation methods in computer graphics. Eurographics Tutorial. Cited by: §I, §III-A.
  • [28] M. Macklin, M. Müller, N. Chentanez, and T. Kim (2014-07) Unified particle physics for real-time applications. ACM Trans. Graph. 33 (4). External Links: ISSN 0730-0301, Link, Document Cited by: §I.
  • [29] M. Macklin, M. Müller, and N. Chentanez (2016) XPBD: position-based simulation of compliant constrained dynamics. MIG ’16, New York, NY, USA, pp. 49–54. External Links: ISBN 9781450345927, Link, Document Cited by: §I, §I, §II-A.
  • [30] D. McConachie, T. Power, P. Mitrano, and D. Berenson (2020) Learning when to trust a dynamics model for planning in reduced state spaces. IEEE Robotics and Automation Letters 5 (2), pp. 3540–3547. External Links: Document Cited by: §I.
  • [31] D. Navarro-Alarcon, Y. Liu, J. G. Romero, and P. Li (2014) On the visual deformation servoing of compliant objects: uncalibrated control methods and experiments. The International Journal of Robotics Research 33 (11), pp. 1462–1480. External Links: Document, Link, Cited by: §I.
  • [32] D. Navarro-Alarcon and Y. Liu (2018) Fourier-based shape servoing: a new feedback method to actively deform soft objects into desired 2-d image contours. IEEE Transactions on Robotics 34 (1), pp. 272–279. External Links: Document Cited by: §I.
  • [33] Y. Qiao, J. Liang, V. Koltun, and M. C. Lin (2021-18–24 Jul) Efficient differentiable simulation of articulated bodies. In Proceedings of the 38th International Conference on Machine Learning, M. Meila and T. Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, pp. 8661–8671. Cited by: §I.
  • [34] Y. Qiao, J. Liang, V. Koltun, and M. Lin (2021) Differentiable simulation of soft multi-body systems. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), External Links: Link Cited by: §I.
  • [35] F. Richter, R. K. Orosco, and M. C. Yip (2021) Image based reconstruction of liquids from 2d surface detections. External Links: 2111.11491 Cited by: §I.
  • [36] F. J. Romero-Ramirez, R. Muñoz-Salinas, and R. Medina-Carnicer (2018) Speeded up detection of squared fiducial markers. Image and Vision Computing 76, pp. 38–47. External Links: ISSN 0262-8856, Document, Link Cited by: §IV-C.
  • [37] J. Sanchez, J. Corrales, B. Bouzgarrou, and Y. Mezouar (2018) Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey. The International Journal of Robotics Research 37 (7), pp. 688–716. External Links: Document Cited by: §I.
  • [38] C. Schenck and D. Fox (2018) SPNets: differentiable fluid dynamics for deep neural networks. In Proceedings of the Second Conference on Robot Learning (CoRL), Zurich, Switzerland. Cited by: §I.
  • [39] P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, J. Ichnowski, E. Novoseller, M. Hwang, M. Laskey, J. Gonzalez, and K. Goldberg (2021-07) Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies. In Proceedings of Robotics: Science and Systems, Virtual. External Links: Document Cited by: §I.
  • [40] A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar (2019-06) Learning robotic manipulation through visual planning and acting. In Proceedings of Robotics: Science and Systems, FreiburgimBreisgau, Germany. External Links: Document Cited by: §I.
  • [41] K. Werling, D. Omens, J. Lee, I. Exarchos, and C. K. Liu (2021-07) Fast and Feature-Complete Differentiable Physics Engine for Articulated Rigid Bodies with Contact Constraints. In Proceedings of Robotics: Science and Systems, Virtual. External Links: Document Cited by: §I.
  • [42] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel (2020-07) Learning to Manipulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA. External Links: Document Cited by: §I.
  • [43] L. Xu and Q. Liu (2018-07-01) Real-time inextensible surgical thread simulation. International Journal of Computer Assisted Radiology and Surgery 13 (7), pp. 1019–1035. External Links: ISSN 1861-6429, Document, Link Cited by: §III-A.
  • [44] H. Yin, A. Varava, and D. Kragic (2021) Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics 6 (54), pp. eabd8803. External Links: Document, Link, Cited by: §I.
  • [45] M. Yu, H. Zhong, and X. Li (2021) Shape control of deformable linear objects with offline and online learning of local linear deformation models. External Links: 2109.11091 Cited by: §I.
  • [46] W. Zhang, K. Schmeckpeper, P. Chaudhari, and K. Daniilidis (2021-05) Deformable linear object prediction using locally linear latent dynamics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), External Links: Document, Link Cited by: §I.
  • [47] F. Zhong, Y. Wang, Z. Wang, and Y. Liu (2019) Dual-arm robotic needle insertion with active tissue deformation for autonomous suturing. IEEE Robotics and Automation Letters 4 (3), pp. 2669–2676. External Links: Document Cited by: §I.
  • [48] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, J. Pan, W. Yuan, and M. Gienger (2021) Challenges and Outlook in Robotic Manipulation of Deformable Objects. IEEE Robotics and Automation Magazine. External Links: Link Cited by: §I.
  • [49] J. Zhu, D. Navarro-Alarcon, R. Passama, and A. Cherubini (2021-05) Vision-based Manipulation of Deformable and Rigid Objects Using Subspace Projections of 2D Contours. Robotics and Autonomous Systems. External Links: Link, Document Cited by: §I.
  • [50] J. Zhu (2020-03) Vision-based robotic manipulation of deformable linear objects. Theses, Université Montpellier. External Links: Link Cited by: §I.