DeepAI AI Chat
Log In Sign Up

Learning Latent Graph Dynamics for Deformable Object Manipulation

by   Xiao Ma, et al.

Manipulating deformable objects, such as cloth and ropes, is a long-standing challenge in robotics: their large number of degrees of freedom (DoFs) and complex non-linear dynamics make motion planning extremely difficult. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). To tackle the challenge of many DoFs and complex dynamics, G-DOOM approximates a deformable object as a sparse set of interacting keypoints and learns a graph neural network that captures abstractly the geometry and interaction dynamics of the keypoints. Further, to tackle the perceptual challenge, specifically, object self-occlusion, G-DOOM adds a recurrent neural network to track the keypoints over time and condition their interactions on the history. We then train the resulting recurrent graph dynamics model through contrastive learning in a high-fidelity simulator. For manipulation planning, G-DOOM explicitly reasons about the learned dynamics model through model-predictive control applied at each of the keypoints. We evaluate G-DOOM on a set of challenging cloth and rope manipulation tasks and show that G-DOOM outperforms a state-of-the-art method. Further, although trained entirely on simulation data, G-DOOM transfers directly to a real robot for both cloth and rope manipulation in our experiments.


page 1

page 5

page 7

page 8

page 12

page 14


Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Using visual model-based learning for deformable object manipulation is ...

Graph-Transporter: A Graph-based Learning Method for Goal-Conditioned Deformable Object Rearranging Task

Rearranging deformable objects is a long-standing challenge in robotic m...

RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks

Modeling and manipulating elasto-plastic objects are essential capabilit...

Learning When to Trust a Dynamics Model for Planning in Reduced State Spaces

When the dynamics of a system are difficult to model and/or time-consumi...

DiffSRL: Learning Dynamic-aware State Representation for Deformable Object Control with Differentiable Simulator

Dynamic state representation learning is an important task in robot lear...

Realtime Simulation of Thin-Shell Deformable Materials using CNN-Based Mesh Embedding

We address the problem of accelerating thin-shell deformable object simu...

Manipulating Deformable Objects by Interleaving Prediction, Planning, and Control

We present a framework for deformable object manipulation that interleav...