Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

by   Danny Driess, et al.

In this paper, we propose a deep convolutional recurrent neural network that predicts action sequences for task and motion planning (TAMP) from an initial scene image. Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g. first-order logic) with continuous motion planning such as nonlinear trajectory optimization. Due to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we develop a neural network which, based on an initial image of the scene, directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. A key aspect is that our method generalizes to scenes with many and varying number of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene in images as input to the neural network, instead of a fixed feature vector. Results show runtime improvements of several magnitudes. Video:


Spatial Reasoning via Deep Vision Models for Robotic Sequential Manipulation

In this paper, we propose using deep neural architectures (i.e., vision ...

Learning When to Quit: Meta-Reasoning for Motion Planning

Anytime motion planners are widely used in robotics. However, the relati...

A Mollification Scheme for Task and Motion Planning

Task and motion planning is one of the key problems in robotics today. I...

Learning to Search in Task and Motion Planning with Streams

Task and motion planning problems in robotics typically combine symbolic...

Policy-Guided Lazy Search with Feedback for Task and Motion Planning

PDDLStream solvers have recently emerged as viable solutions for Task an...

Receding Horizon Task and Motion Planning in Dynamic Environments

Complex manipulation tasks in crowded environments require careful integ...

Customizing First Person Image Through Desired Actions

This paper studies a problem of inverse visual path planning: creating a...

Please sign up or login with your details

Forgot password? Click here to reset