DeepAI AI Chat
Log In Sign Up

Structured Reordering for Modeling Latent Alignments in Sequence Transduction

06/06/2021
by   Bailin Wang, et al.
2

Despite success in many domains, neural models struggle in settings where train and test examples are drawn from different distributions. In particular, in contrast to humans, conventional sequence-to-sequence (seq2seq) models fail to generalize systematically, i.e., interpret sentences representing novel combinations of concepts (e.g., text segments) seen in training. Traditional grammar formalisms excel in such settings by implicitly encoding alignments between input and output segments, but are hard to scale and maintain. Instead of engineering a grammar, we directly model segment-to-segment alignments as discrete structured latent variables within a neural seq2seq model. To efficiently explore the large space of alignments, we introduce a reorder-first align-later framework whose central component is a neural reordering module producing separable permutations. We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations, and, thus, enabling end-to-end differentiable training of our model. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks (i.e., semantic parsing and machine translation).

READ FULL TEXT

page 1

page 2

page 3

page 4

09/02/2021

Sequence-to-Sequence Learning with Latent Neural Grammars

Sequence-to-sequence learning with neural networks has become the de fac...
06/07/2021

Lexicon Learning for Few-Shot Neural Sequence Modeling

Sequence-to-sequence transduction is the core problem in language proces...
04/22/2020

Discretized Bottleneck in VAE: Posterior-Collapse-Free Sequence-to-Sequence Learning

Variational autoencoders (VAEs) are important tools in end-to-end repres...
05/19/2022

LAGr: Label Aligned Graphs for Better Systematic Generalization in Semantic Parsing

Semantic parsing is the task of producing structured meaning representat...
09/26/2016

Online Segment to Segment Neural Transduction

We introduce an online neural sequence to sequence model that learns to ...
10/23/2020

A Differentiable Relaxation of Graph Segmentation and Alignment for AMR Parsing

Abstract Meaning Representations (AMR) are a broad-coverage semantic for...
10/14/2021

LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsing

Semantic parsing is the task of producing a structured meaning represent...