DeepAI AI Chat
Log In Sign Up

Lexicon Learning for Few-Shot Neural Sequence Modeling

06/07/2021
by   Ekin Akyürek, et al.
0

Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural network models that provide the dominant solution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of systematic generalization arise from neural models' inability to disentangle lexical phenomena from syntactic ones. To address this, we augment neural decoders with a lexical translation mechanism that generalizes existing copy mechanisms to incorporate learned, decontextualized, token-level translation rules. We describe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generalization on a diverse set of sequence modeling tasks drawn from cognitive science, formal semantics, and machine translation.

READ FULL TEXT

page 6

page 12

12/28/2021

A Preordered RNN Layer Boosts Neural Machine Translation in Low Resource Settings

Neural Machine Translation (NMT) models are strong enough to convey sema...
11/19/2017

Incorporating Syntactic Uncertainty in Neural Machine Translation with Forest-to-Sequence Model

Incorporating syntactic information in Neural Machine Translation models...
06/06/2021

Structured Reordering for Modeling Latent Alignments in Sequence Transduction

Despite success in many domains, neural models struggle in settings wher...
01/02/2021

Decoding Time Lexical Domain Adaptationfor Neural Machine Translation

Machine translation systems are vulnerable to domain mismatch, especiall...
05/07/2021

Duplex Sequence-to-Sequence Learning for Reversible Machine Translation

Sequence-to-sequence (seq2seq) problems such as machine translation are ...
10/17/2018

Sequence to Sequence Mixture Model for Diverse Machine Translation

Sequence to sequence (SEQ2SEQ) models often lack diversity in their gene...