Log In Sign Up

Is a Modular Architecture Enough?

by   Sarthak Mittal, et al.

Inspired from human cognition, machine learning systems are gradually revealing advantages of sparser and more modular architectures. Recent work demonstrates that not only do some modular architectures generalize well, but they also lead to better out-of-distribution generalization, scaling properties, learning speed, and interpretability. A key intuition behind the success of such systems is that the data generating system for most real-world settings is considered to consist of sparsely interacting parts, and endowing models with similar inductive biases will be helpful. However, the field has been lacking in a rigorous quantitative assessment of such systems because these real-world data distributions are complex and unknown. In this work, we provide a thorough assessment of common modular architectures, through the lens of simple and known modular data distributions. We highlight the benefits of modularity and sparsity and reveal insights on the challenges faced while optimizing modular systems. In doing so, we propose evaluation metrics that highlight the benefits of modularity, the regimes in which these benefits are substantial, as well as the sub-optimality of current end-to-end learned modular systems as opposed to their claimed potential.


page 16

page 19

page 23

page 24

page 27

page 28

page 30

page 32


Learning Modular Structures That Generalize Out-of-Distribution

Out-of-distribution (O.O.D.) generalization remains to be a key challeng...

S2RMs: Spatially Structured Recurrent Modules

Capturing the structure of a data-generating process by means of appropr...

Towards Modular Machine Learning Solution Development: Benefits and Trade-offs

Machine learning technologies have demonstrated immense capabilities in ...

A Perspective on Objects and Systematic Generalization in Model-Based RL

In order to meet the diverse challenges in solving many real-world probl...

Deploying Deep Ranking Models for Search Verticals

In this paper, we present an architecture executing a complex machine le...

Learning Label Modular Prompts for Text Classification in the Wild

Machine learning models usually assume i.i.d data during training and te...

On the Difficulty of Warm-Starting Neural Network Training

In many real-world deployments of machine learning systems, data arrive ...