Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Tasks

09/30/2021
by   Yichen Jiang, et al.
5

Systematic compositionality is an essential mechanism in human language, allowing the recombination of known parts to create novel expressions. However, existing neural models have been shown to lack this basic ability in learning symbolic structures. Motivated by the failure of a Transformer model on the SCAN compositionality challenge (Lake and Baroni, 2018), which requires parsing a command into actions, we propose two auxiliary sequence prediction tasks that track the progress of function and argument semantics, as additional training supervision. These automatically-generated sequences are more representative of the underlying compositional symbolic structures of the input data. During inference, the model jointly predicts the next action and the next tokens in the auxiliary sequences at each step. Experiments on the SCAN dataset show that our method encourages the Transformer to understand compositional structures of the command, improving its accuracy on multiple challenging splits from <= 10 to 100 97.8 be induced in Transformers given minimal but proper guidance. We also show that a better result is achieved using less contextualized vectors as the attention's query, providing insights into architecture choices in achieving systematic compositionality. Finally, we show positive generalization results on the groundedSCAN task (Ruis et al., 2020). Our code is publicly available at: https://github.com/jiangycTarheel/compositional-auxseq

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/19/2021

Improving Compositional Generalization in Classification Tasks via Structure Annotations

Compositional generalization is the ability to generalize systematically...
research
07/14/2021

Learning Algebraic Recombination for Compositional Generalization

Neural sequence models exhibit limited compositional generalization abil...
research
06/18/2020

Compositional Generalization by Learning Analytical Expressions

Compositional generalization is a basic but essential intellective capab...
research
08/26/2021

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Recently, many datasets have been proposed to test the systematic genera...
research
11/28/2022

Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality

Recent datasets expose the lack of the systematic generalization ability...
research
09/29/2020

Think before you act: A simple baseline for compositional generalization

Contrarily to humans who have the ability to recombine familiar expressi...
research
10/14/2021

The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization

Despite successes across a broad range of applications, Transformers hav...

Please sign up or login with your details

Forgot password? Click here to reset