DeepAI AI Chat
Log In Sign Up

Learning Manipulation under Physics Constraints with Visual Perception

by   Wenbin Li, et al.

Understanding physical phenomena is a key competence that enables humans and animals to act and interact under uncertain perception in previously unseen environments containing novel objects and their configurations. In this work, we consider the problem of autonomous block stacking and explore solutions to learning manipulation under physics constraints with visual perception inherent to the task. Inspired by the intuitive physics in humans, we first present an end-to-end learning-based approach to predict stability directly from appearance, contrasting a more traditional model-based approach with explicit 3D representations and physical simulation. We study the model's behavior together with an accompanied human subject test. It is then integrated into a real-world robotic system to guide the placement of a single wood block into the scene without collapsing existing tower structure. To further automate the process of consecutive blocks stacking, we present an alternative approach where the model learns the physics constraint through the interaction with the environment, bypassing the dedicated physics learning as in the former part of this work. In particular, we are interested in the type of tasks that require the agent to reach a given goal state that may be different for every new trial. Thereby we propose a deep reinforcement learning framework that learns policies for stacking tasks which are parametrized by a target structure.


page 2

page 9

page 11

page 12


Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning

Understanding physical phenomena is a key component of human intelligenc...

Unsupervised Intuitive Physics from Past Experiences

We are interested in learning models of intuitive physics similar to the...

Visual Stability Prediction and Its Application to Manipulation

Understanding physical phenomena is a key competence that enables humans...

To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction

Understanding physical phenomena is a key competence that enables humans...

Towards Human-Level Learning of Complex Physical Puzzles

Humans quickly solve tasks in novel systems with complex dynamics, witho...

Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

We introduce an approach to model surface properties governing bounces i...

Solving Physics Puzzles by Reasoning about Paths

We propose a new deep learning model for goal-driven tasks that require ...