DeepAI AI Chat
Log In Sign Up

Structured Reordering for Modeling Latent Alignments in Sequence Transduction

by   Bailin Wang, et al.

Despite success in many domains, neural models struggle in settings where train and test examples are drawn from different distributions. In particular, in contrast to humans, conventional sequence-to-sequence (seq2seq) models fail to generalize systematically, i.e., interpret sentences representing novel combinations of concepts (e.g., text segments) seen in training. Traditional grammar formalisms excel in such settings by implicitly encoding alignments between input and output segments, but are hard to scale and maintain. Instead of engineering a grammar, we directly model segment-to-segment alignments as discrete structured latent variables within a neural seq2seq model. To efficiently explore the large space of alignments, we introduce a reorder-first align-later framework whose central component is a neural reordering module producing separable permutations. We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations, and, thus, enabling end-to-end differentiable training of our model. The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks (i.e., semantic parsing and machine translation).


page 1

page 2

page 3

page 4


Sequence-to-Sequence Learning with Latent Neural Grammars

Sequence-to-sequence learning with neural networks has become the de fac...

Lexicon Learning for Few-Shot Neural Sequence Modeling

Sequence-to-sequence transduction is the core problem in language proces...

Discretized Bottleneck in VAE: Posterior-Collapse-Free Sequence-to-Sequence Learning

Variational autoencoders (VAEs) are important tools in end-to-end repres...

LAGr: Label Aligned Graphs for Better Systematic Generalization in Semantic Parsing

Semantic parsing is the task of producing structured meaning representat...

Online Segment to Segment Neural Transduction

We introduce an online neural sequence to sequence model that learns to ...

A Differentiable Relaxation of Graph Segmentation and Alignment for AMR Parsing

Abstract Meaning Representations (AMR) are a broad-coverage semantic for...

LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsing

Semantic parsing is the task of producing a structured meaning represent...