S2RMs: Spatially Structured Recurrent Modules

by   Nasim Rahaman, et al.

Capturing the structure of a data-generating process by means of appropriate inductive biases can help in learning models that generalize well and are robust to changes in the input distribution. While methods that harness spatial and temporal structures find broad application, recent work has demonstrated the potential of models that leverage sparse and modular structure using an ensemble of sparingly interacting modules. In this work, we take a step towards dynamic models that are capable of simultaneously exploiting both modular and spatiotemporal structures. We accomplish this by abstracting the modeled dynamical system as a collection of autonomous but sparsely interacting sub-systems. The sub-systems interact according to a topology that is learned, but also informed by the spatial structure of the underlying real-world system. This results in a class of models that are well suited for modeling the dynamics of systems that only offer local views into their state, along with corresponding spatial locations of those views. On the tasks of video prediction from cropped frames and multi-agent world modeling from partial observations in the challenging Starcraft2 domain, we find our models to be more robust to the number of available views and better capable of generalization to novel tasks without additional training, even when compared against strong baselines that perform equally well or better on the training distribution.


page 2

page 7

page 12

page 13

page 17

page 18

page 19

page 20


Is a Modular Architecture Enough?

Inspired from human cognition, machine learning systems are gradually re...

Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning

Modeling the dynamics of real-world physical systems is critical for spa...

Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...

On the Generalization and Adaption Performance of Causal Models

Learning models that offer robust out-of-distribution generalization and...

Learning Modular Representations for Long-Term Multi-Agent Motion Predictions

Context plays a significant role in the generation of motion for dynamic...

Enhancing Spatiotemporal Prediction Model using Modular Design and Beyond

Predictive learning uses a known state to generate a future state over a...

TaylorPDENet: Learning PDEs from non-grid Data

Modeling data obtained from dynamical systems has gained attention in re...

Please sign up or login with your details

Forgot password? Click here to reset