DeepAI
Log In Sign Up

RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces

07/02/2020
by   Sebastien Ehrhardt, et al.
21

We present RELATE, a model that learns to generate physically plausible scenes and videos of multiple interacting objects. Similar to other generative approaches, RELATE is trained end-to-end on raw, unlabeled data. RELATE combines an object-centric GAN formulation with a model that explicitly accounts for correlations between individual objects. This allows the model to generate realistic scenes and videos from a physically-interpretable parameterization. Furthermore, we show that modeling the object correlation is necessary to learn to disentangle object positions and identity. We find that RELATE is also amenable to physically realistic scene editing and that it significantly outperforms prior art in object-centric scene generation in both synthetic (CLEVR, ShapeStacks) and real-world data (street traffic scenes). In addition, in contrast to state-of-the-art methods in object-centric generative modeling, RELATE also extends naturally to dynamic scenes and generates videos of high visual fidelity

READ FULL TEXT

page 7

page 18

page 19

page 20

page 21

page 22

page 23

page 24

10/13/2021

Unsupervised Object Learning via Common Fate

Learning generative object models from unlabelled videos is a long stand...
08/17/2019

Neural Re-Simulation for Generating Bounces in Single Images

We introduce a method to generate videos of dynamic virtual objects plau...
07/30/2019

GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Generative models are emerging as promising tools in robotics and reinfo...
04/27/2020

Towards causal generative scene models via competition of experts

Learning how to model complex scenes in a modular way with recombinable ...
07/01/2020

Future Urban Scenes Generation Through Vehicles Synthesis

In this work we propose a deep learning pipeline to predict the visual f...
06/27/2012

Learning Object Arrangements in 3D Scenes using Human Context

We consider the problem of learning object arrangements in a 3D scene. T...
04/09/2021

GATSBI: Generative Agent-centric Spatio-temporal Object Interaction

We present GATSBI, a generative model that can transform a sequence of r...

Code Repositories

relate

Official PyTorch implementation of 'RELATE: Physically Plausible Multi-Object SceneSynthesis Using Structured Latent Spaces'.


view repo