Enabling Visual Action Planning for Object Manipulation through Latent Space Roadmap

03/03/2021
by   Martina Lippi, et al.
0

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a Latent Space Roadmap (LSR) for task planning, a graph-based structure capturing globally the system dynamics in a low-dimensional latent space. Our framework consists of three parts: (1) a Mapping Module (MM) that maps observations, given in the form of images, into a structured latent space extracting the respective states, that generates observations from the latent states, (2) the LSR which builds and connects clusters containing similar states in order to find the latent plans between start and goal states extracted by MM, and (3) the Action Proposal Module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on two simulated box stacking tasks and a folding task executed on a real robot.

READ FULL TEXT

page 1

page 5

page 18

research
03/19/2020

Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation

We present a framework for visual action planning of complex manipulatio...
research
03/27/2023

Ensemble Latent Space Roadmap for Improved Robustness in Visual Action Planning

Planning in learned latent spaces helps to decrease the dimensionality o...
research
03/01/2023

DeFNet: Deconstructed Fabric Folding Strategy Based on Latent Space Roadmap and Flow-Based Policy

Fabric folding through robots is complex and challenging due to the defo...
research
03/24/2022

Augment-Connect-Explore: a Paradigm for Visual Action Planning with Data Scarcity

Visual action planning particularly excels in applications where the sta...
research
03/02/2023

PlaNet-Pick: Effective Cloth Flattening Based on Latent Dynamic Planning

Why do Recurrent State Space Models such as PlaNet fail at cloth manipul...
research
10/29/2019

Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation

The fundamental challenge of planning for multi-step manipulation is to ...
research
07/24/2018

Learning Plannable Representations with Causal InfoGAN

In recent years, deep generative models have been shown to 'imagine' con...

Please sign up or login with your details

Forgot password? Click here to reset