DeepAI AI Chat
Log In Sign Up

Real-World Compositional Generalization with Disentangled Sequence-to-Sequence Learning

by   Hao Zheng, et al.

Compositional generalization is a basic mechanism in human language learning, which current neural networks struggle with. A recently proposed Disentangled sequence-to-sequence model (Dangle) shows promising generalization capability by learning specialized encodings for each decoding step. We introduce two key modifications to this model which encourage more disentangled representations and improve its compute and memory efficiency, allowing us to tackle compositional generalization in a more realistic setting. Specifically, instead of adaptively re-encoding source keys and values at each time step, we disentangle their representations and only re-encode keys periodically, at some interval. Our new architecture leads to better generalization performance across existing tasks and datasets, and a new machine translation benchmark which we create by detecting naturally occurring compositional patterns in relation to a training set. We show this methodology better emulates real-world requirements than artificial challenges.


page 1

page 2

page 3

page 4


Disentangled Sequence to Sequence Learning for Compositional Generalization

There is mounting evidence that existing neural network models, in parti...

Transcoding compositionally: using attention to find more generalizable solutions

While sequence-to-sequence models have shown remarkable generalization p...

Compositional generalization through meta sequence-to-sequence learning

People can learn a new concept and use it compositionally, understanding...

Compositional Generalization for Primitive Substitutions

Compositional generalization is a basic mechanism in human language lear...

Sequence-Level Mixed Sample Data Augmentation

Despite their empirical success, neural networks still have difficulty c...

Vector-based Representation is the Key: A Study on Disentanglement and Compositional Generalization

Recognizing elementary underlying concepts from observations (disentangl...

Learn to Compose Syntactic and Semantic Representations Appropriately for Compositional Generalization

Recent studies have shown that sequence-to-sequence (Seq2Seq) models are...