Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects

by   Adam R. Kosiorek, et al.

We present Sequential Attend, Infer, Repeat (SQAIR), an interpretable deep generative model for videos of moving objects. It can reliably discover and track objects throughout the sequence of frames, and can also generate future frames conditioning on the current frame, thereby simulating expected motion of objects. This is achieved by explicitly encoding object presence, locations and appearances in the latent variables of the model. SQAIR retains all strengths of its predecessor, Attend, Infer, Repeat (AIR, Eslami et. al., 2016), including learning in an unsupervised manner, and addresses its shortcomings. We use a moving multi-MNIST dataset to show limitations of AIR in detecting overlapping or partially occluded objects, and show how SQAIR overcomes them by leveraging temporal consistency of objects. Finally, we also apply SQAIR to real-world pedestrian CCTV data, where it learns to reliably detect, track and generate walking pedestrians with no supervision.



There are no comments yet.


page 18

page 19

page 20

page 21

page 22

page 23

page 24

page 25


Unsupervised and interpretable scene discovery with Discrete-Attend-Infer-Repeat

In this work we present Discrete Attend Infer Repeat (Discrete-AIR), a R...

Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure

Our goal in this work is to generate realistic videos given just one ini...

Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction

We propose a deep video prediction model conditioned on a single image a...

Object-Centric Image Generation with Factored Depths, Locations, and Appearances

We present a generative model of images that explicitly reasons over the...

GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture

Recent studies on unsupervised object detection based on spatial attenti...

Unsupervised Discovery of 3D Physical Objects from Video

We study the problem of unsupervised physical object discovery. Unlike e...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.