Meta-Learning to Compositionally Generalize

06/08/2021
by   Henry Conklin, et al.
12

Natural language is compositional; the meaning of a sentence is a function of the meaning of its parts. This property allows humans to create and interpret novel sentences, generalizing robustly outside their prior experience. Neural networks have been shown to struggle with this kind of generalization, in particular performing poorly on tasks designed to assess compositional generalization (i.e. where training and testing distributions differ in ways that would be trivial for a compositional strategy to resolve). Their poor performance on these tasks may in part be due to the nature of supervised learning which assumes training and testing data to be drawn from the same distribution. We implement a meta-learning augmented version of supervised learning whose objective directly optimizes for out-of-distribution generalization. We construct pairs of tasks for meta-learning by sub-sampling existing training data. Each pair of tasks is constructed to contain relevant examples, as determined by a similarity metric, in an effort to inhibit models from memorizing their input. Experimental results on the COGS and SCAN datasets show that our similarity-driven meta-learning can improve generalization performance.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

10/05/2020

Meta-Learning of Compositional Task Distributions in Humans and Machines

Modern machine learning systems struggle with sample efficiency and are ...
08/03/2021

Generalization in Multimodal Language Learning from Simulation

Neural networks can be powerful function approximators, which are able t...
01/27/2021

Similarity of Classification Tasks

Recent advances in meta-learning has led to remarkable performances on s...
02/03/2020

Revisiting Meta-Learning as Supervised Learning

Recent years have witnessed an abundance of new publications and approac...
03/01/2021

A Brief Summary of Interactions Between Meta-Learning and Self-Supervised Learning

This paper briefly reviews the connections between meta-learning and sel...
02/10/2020

Compositional ADAM: An Adaptive Compositional Solver

In this paper, we present C-ADAM, the first adaptive solver for composit...
07/19/2018

Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

Systematic compositionality is the ability to recombine meaningful units...

Code Repositories

tensor2struct-public

Semantic parsers based on encoder-decoder framework


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.