Visual Robot Task Planning

by   Chris Paxton, et al.
Johns Hopkins University

Prospection, the act of predicting the consequences of many possible futures, is intrinsic to human planning and action, and may even be at the root of consciousness. Surprisingly, this idea has been explored comparatively little in robotics. In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment. Models are trained via imitation learning on a variety of domains, including navigation, pick-and-place, and a surgical robotics task. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information.


page 1

page 4

page 5

page 6

page 8


Learning to Imagine Manipulation Goals for Robot Task Planning

Prospection is an important part of how humans come up with new task pla...

Planning in the imagination: High-level planning on learned abstract search spaces

We propose a new method, called PiZero, that gives an agent the ability ...

FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environment

Painting is an artistic process of rendering visual content that achieve...

Synthesizing Manipulation Sequences for Under-Specified Tasks using Unrolled Markov Random Fields

Many tasks in human environments require performing a sequence of naviga...

Action sequencing using visual permutations

Humans can easily reason about the sequence of high level actions needed...

Learning and Planning for Temporally Extended Tasks in Unknown Environments

We propose a novel planning technique for satisfying tasks specified in ...

Learning recurrent representations for hierarchical behavior modeling

We propose a framework for detecting action patterns from motion sequenc...

Please sign up or login with your details

Forgot password? Click here to reset