Reusable Slotwise Mechanisms

02/21/2023
by   Trang Nguyen, et al.
0

Agents that can understand and reason over the dynamics of objects can have a better capability to act robustly and generalize to novel scenarios. Such an ability, however, requires a suitable representation of the scene as well as an understanding of the mechanisms that govern the interactions of different subsets of objects. To address this problem, we propose RSM, or Reusable Slotwise Mechanisms, that jointly learns a slotwise representation of the scene and a modular architecture that dynamically chooses one mechanism among a set of reusable mechanisms to predict the next state of each slot. RSM crucially takes advantage of a Central Contextual Information (CCI), which lets each selected reusable mechanism access the rest of the slots through a bottleneck, effectively allowing for modeling higher order and complex interactions that might require a sparse subset of objects. We show how this model outperforms state-of-the-art methods in a variety of next-step prediction tasks ranging from grid-world environments to Atari 2600 games. Particularly, we challenge methods that model the dynamics with Graph Neural Networks (GNNs) on top of slotwise representations, and modular architectures that restrict the interactions to be only pairwise. Finally, we show that RSM is able to generalize to scenes with objects varying in number and shape, highlighting its out-of-distribution generalization capabilities. Our implementation is available online[https://github.com/trangnnp/RSMgithub.com/trangnnp/RSM].

READ FULL TEXT

page 7

page 8

page 14

page 16

page 17

page 18

page 19

page 20

research
10/16/2022

Stochastic Occupancy Grid Map Prediction in Dynamic Scenes

This paper presents two variations of a novel stochastic prediction algo...
research
04/20/2022

Simplicial Attention Networks

Graph representation learning methods have mostly been limited to the mo...
research
03/04/2021

Graph-based Task-specific Prediction Models for Interactions between Deformable and Rigid Objects

Capturing scene dynamics and predicting the future scene state is challe...
research
05/17/2022

GraphMapper: Efficient Visual Navigation by Scene Graph Generation

Understanding the geometric relationships between objects in a scene is ...
research
04/17/2023

RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding

Human drivers naturally reason about interactions between road users to ...
research
10/20/2022

Solving Reasoning Tasks with a Slot Transformer

The ability to carve the world into useful abstractions in order to reas...
research
09/24/2019

Recurrent Independent Mechanisms

Learning modular structures which reflect the dynamics of the environmen...

Please sign up or login with your details

Forgot password? Click here to reset