A Study of Compositional Generalization in Neural Models

06/16/2020
by   Tim Klinger, et al.
18

Compositional and relational learning is a hallmark of human intelligence, but one which presents challenges for neural models. One difficulty in the development of such models is the lack of benchmarks with clear compositional and relational task structure on which to systematically evaluate them. In this paper, we introduce an environment called ConceptWorld, which enables the generation of images from compositional and relational concepts, defined using a logical domain specific language. We use it to generate images for a variety of compositional structures: 2x2 squares, pentominoes, sequences, scenes involving these objects, and other more complex concepts. We perform experiments to test the ability of standard neural architectures to generalize on relations with compositional arguments as the compositional depth of those arguments increases and under substitution. We compare standard neural networks such as MLP, CNN and ResNet, as well as state-of-the-art relational networks including WReN and PrediNet in a multi-class image classification setting. For simple problems, all models generalize well to close concepts but struggle with longer compositional chains. For more complex tests involving substitutivity, all models struggle, even with short chains. In highlighting these difficulties and providing an environment for further experimentation, we hope to encourage the development of models which are able to generalize effectively in compositional, relational domains.

READ FULL TEXT

page 6

page 7

page 8

page 18

page 20

page 22

page 24

page 26

research
06/19/2021

Improving Compositional Generalization in Classification Tasks via Structure Annotations

Compositional generalization is the ability to generalize systematically...
research
12/20/2022

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Large-scale models combining text and images have made incredible progre...
research
02/08/2021

Concepts, Properties and an Approach for Compositional Generalization

Compositional generalization is the capacity to recognize and imagine a ...
research
06/15/2023

Modularity Trumps Invariance for Compositional Robustness

By default neural networks are not robust to changes in data distributio...
research
08/17/2022

Learning Transductions to Test Systematic Compositionality

Recombining known primitive concepts into larger novel combinations is a...
research
09/22/2021

COVR: A test-bed for Visually Grounded Compositional Generalization with real images

While interest in models that generalize at test time to new composition...
research
08/22/2019

The compositionality of neural networks: integrating symbolism and connectionism

Despite a multitude of empirical studies, little consensus exists on whe...

Please sign up or login with your details

Forgot password? Click here to reset