DeepAI AI Chat
Log In Sign Up

Robotic Visuomotor Control with Unsupervised Forward Model Learned from Videos

by   Haoqi Yuan, et al.

Learning an accurate model of the environment is essential for model-based control tasks. Existing methods in robotic visuomotor control usually learn from data with heavily labelled actions, object entities or locations, which can be demanding in many cases. To cope with this limitation, we propose a method that trains a forward model from video data only, via disentangling the motion of controllable agent to model the transition dynamics. An object extractor and an interaction learner are trained in an end-to-end manner without supervision. The agent's motions are explicitly represented using spatial transformation matrices containing physical meanings. In the experiments, our method achieves superior performance on learning an accurate forward model in a Grid World environment, as well as a more realistic robot control environment in simulation. With the accurate learned forward models, we further demonstrate their usage in model predictive control as an effective approach for robotic manipulations.


page 1

page 5

page 6

page 7


Unsupervised Learning for Physical Interaction through Video Prediction

A core challenge for an agent learning to interact with the world is to ...

Object-centric Forward Modeling for Model Predictive Control

We present an approach to learn an object-centric forward model, and sho...

Learning Predictive Models From Observation and Interaction

Learning predictive models from interaction with the world allows an age...

Hindsight for Foresight: Unsupervised Structured Dynamics Models from Physical Interaction

A key challenge for an agent learning to interact with the world is to r...

Inverting Learned Dynamics Models for Aggressive Multirotor Control

We present a control strategy that applies inverse dynamics to a learned...

Learning Partially Structured Environmental Dynamics for Marine Robotic Navigation

We investigate the scenario that a robot needs to reach a designated goa...

CLOUD: Contrastive Learning of Unsupervised Dynamics

Developing agents that can perform complex control tasks from high dimen...