DeepAI AI Chat
Log In Sign Up

Visually Grounded Continual Learning of Compositional Semantics

by   Xisen Jin, et al.
University of Southern California

Children's language acquisition from the visual world is a real-world example of continual learning from dynamic and evolving environments; yet we lack a realistic setup to study neural networks' capability in human-like language acquisition. In this paper, we propose a realistic setup by simulating children's language acquisition process. We formulate language acquisition as a masked language modeling task where the model visits a stream of data with continuously shifting distribution. Our training and evaluation encode two important challenges in human's language learning, namely the continual learning and the compositionality. We show the performance of existing continual learning algorithms is far from satisfactory. We also study the interactions between memory based continual learning algorithms and compositional generalization and conclude that overcoming overfitting and compositional overfitting may be crucial for a good performance in our problem setup. Our code and data can be found at


Renate: A Library for Real-World Continual Learning

Continual learning enables the incremental training of machine learning ...

Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition

Human language acquisition is an efficient, supervised, and continual pr...

Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments

Continual learning refers to the ability of humans and animals to increm...

ADAM: A Sandbox for Implementing Language Learning

We present ADAM, a software system for designing and running child langu...

Label-Efficient Online Continual Object Detection in Streaming Video

To thrive in evolving environments, humans are capable of continual acqu...

CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation

Vision-Language Pretraining (VLP) has shown impressive results on divers...