Learning Latent Graph Dynamics for Deformable Object Manipulation

by   Xiao Ma, et al.

Manipulating deformable objects, such as cloth and ropes, is a long-standing challenge in robotics: their large number of degrees of freedom (DoFs) and complex non-linear dynamics make motion planning extremely difficult. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). To tackle the challenge of many DoFs and complex dynamics, G-DOOM approximates a deformable object as a sparse set of interacting keypoints and learns a graph neural network that captures abstractly the geometry and interaction dynamics of the keypoints. Further, to tackle the perceptual challenge, specifically, object self-occlusion, G-DOOM adds a recurrent neural network to track the keypoints over time and condition their interactions on the history. We then train the resulting recurrent graph dynamics model through contrastive learning in a high-fidelity simulator. For manipulation planning, G-DOOM explicitly reasons about the learned dynamics model through model-predictive control applied at each of the keypoints. We evaluate G-DOOM on a set of challenging cloth and rope manipulation tasks and show that G-DOOM outperforms a state-of-the-art method. Further, although trained entirely on simulation data, G-DOOM transfers directly to a real robot for both cloth and rope manipulation in our experiments.



There are no comments yet.


page 1

page 5

page 7

page 8

page 12

page 14


Learning Predictive Representations for Deformable Objects Using Contrastive Estimation

Using visual model-based learning for deformable object manipulation is ...

Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation

We present a framework for visual action planning of complex manipulatio...

Learning When to Trust a Dynamics Model for Planning in Reduced State Spaces

When the dynamics of a system are difficult to model and/or time-consumi...

DiffSRL: Learning Dynamic-aware State Representation for Deformable Object Control with Differentiable Simulator

Dynamic state representation learning is an important task in robot lear...

Realtime Simulation of Thin-Shell Deformable Materials using CNN-Based Mesh Embedding

We address the problem of accelerating thin-shell deformable object simu...

Graph-based Task-specific Prediction Models for Interactions between Deformable and Rigid Objects

Capturing scene dynamics and predicting the future scene state is challe...

Keep it Simple: Data-efficient Learning for Controlling Complex Systems with Simple Models

When manipulating a novel object with complex dynamics, a state represen...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.