Visual Interaction Networks

06/05/2017
by   Nicholas Watters, et al.
0

From just a glance, humans can make rich predictions about the future state of a wide range of physical systems. On the other hand, modern approaches from engineering, robotics, and graphics are often restricted to narrow domains and require direct measurements of the underlying states. We introduce the Visual Interaction Network, a general-purpose model for learning the dynamics of a physical system from raw visual observations. Our model consists of a perceptual front-end based on convolutional neural networks and a dynamics predictor based on interaction networks. Through joint training, the perceptual front-end learns to parse a dynamic visual scene into a set of factored latent object representations. The dynamics predictor learns to roll these states forward in time by computing their interactions and dynamics, producing a predicted physical trajectory of arbitrary length. We found that from just six input video frames the Visual Interaction Network can generate accurate future trajectories of hundreds of time steps on a wide range of physical systems. Our model can also be applied to scenes with invisible objects, inferring their future states from their effects on the visible objects, and can implicitly infer the unknown mass of objects. Our results demonstrate that the perceptual module and the object-based dynamics predictor module can induce factored latent representations that support accurate dynamical predictions. This work opens new opportunities for model-based decision-making and planning from raw sensory observations in complex physical environments.

READ FULL TEXT

page 6

page 8

research
04/28/2020

Visual Grounding of Learned Physical Models

Humans intuitively recognize objects' physical properties and predict th...
research
09/09/2019

Belief Regulated Dual Propagation Nets for Learning Action Effects on Articulated Multi-Part Objects

In recent years, graph neural networks have been successfully applied fo...
research
05/19/2023

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

Humans and animals have a rich and flexible understanding of the physica...
research
01/21/2020

Learning to Control PDEs with Differentiable Physics

Predicting outcomes and planning interactions with the physical world ar...
research
04/15/2019

Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

We introduce an approach to model surface properties governing bounces i...
research
06/04/2018

Graph networks as learnable physics engines for inference and control

Understanding and interacting with everyday physical scenes requires ric...
research
11/09/2021

Which priors matter? Benchmarking models for learning latent dynamics

Learning dynamics is at the heart of many important applications of mach...

Please sign up or login with your details

Forgot password? Click here to reset