S2RMs: Spatially Structured Recurrent Modules

07/13/2020
by   Nasim Rahaman, et al.
5

Capturing the structure of a data-generating process by means of appropriate inductive biases can help in learning models that generalize well and are robust to changes in the input distribution. While methods that harness spatial and temporal structures find broad application, recent work has demonstrated the potential of models that leverage sparse and modular structure using an ensemble of sparingly interacting modules. In this work, we take a step towards dynamic models that are capable of simultaneously exploiting both modular and spatiotemporal structures. We accomplish this by abstracting the modeled dynamical system as a collection of autonomous but sparsely interacting sub-systems. The sub-systems interact according to a topology that is learned, but also informed by the spatial structure of the underlying real-world system. This results in a class of models that are well suited for modeling the dynamics of systems that only offer local views into their state, along with corresponding spatial locations of those views. On the tasks of video prediction from cropped frames and multi-agent world modeling from partial observations in the challenging Starcraft2 domain, we find our models to be more robust to the number of available views and better capable of generalization to novel tasks without additional training, even when compared against strong baselines that perform equally well or better on the training distribution.

READ FULL TEXT

page 2

page 7

page 12

page 13

page 17

page 18

page 19

page 20

06/06/2022

Is a Modular Architecture Enough?

Inspired from human cognition, machine learning systems are gradually re...
06/15/2020

Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning

Modeling the dynamics of real-world physical systems is critical for spa...
08/07/2022

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...
06/09/2022

On the Generalization and Adaption Performance of Causal Models

Learning models that offer robust out-of-distribution generalization and...
11/29/2019

Learning Modular Representations for Long-Term Multi-Agent Motion Predictions

Context plays a significant role in the generation of motion for dynamic...
11/29/2019

Learning with Modular Representations for Long-Term Multi-Agent Motion Predictions

Context plays a significant role in the generation of motion for dynamic...
12/15/2021

Feature-Attending Recurrent Modules for Generalization in Reinforcement Learning

Deep reinforcement learning (Deep RL) has recently seen significant prog...