DeepAI AI Chat
Log In Sign Up

Online Segment to Segment Neural Transduction

by   Lei Yu, et al.

We introduce an online neural sequence to sequence model that learns to alternate between encoding and decoding segments of the input as it is read. By independently tracking the encoding and decoding representations our algorithm permits exact polynomial marginalization of the latent segmentation during training, and during decoding beam search is employed to find the best alignment path together with the predicted output sequence. Our model tackles the bottleneck of vanilla encoder-decoders that have to read and memorize the entire input sequence in their fixed-length hidden states before producing any output. It is different from previous attentive models in that, instead of treating the attention weights as output of a deterministic function, our model assigns attention weights to a sequential latent variable which can be marginalized out and permits online generation. Experiments on abstractive sentence summarization and morphological inflection show significant performance gains over the baseline encoder-decoders.


page 1

page 2

page 3

page 4


Synchronous Transformers for End-to-End Speech Recognition

For most of the attention-based sequence-to-sequence models, the decoder...

Prior Attention for Style-aware Sequence-to-Sequence Models

We extend sequence-to-sequence models with the possibility to control th...

The Neural Noisy Channel

We formulate sequence to sequence transduction as a noisy channel decodi...

Latent Sequence Decompositions

We present the Latent Sequence Decompositions (LSD) framework. LSD decom...

Structured Reordering for Modeling Latent Alignments in Sequence Transduction

Despite success in many domains, neural models struggle in settings wher...

Generating Text with Deep Reinforcement Learning

We introduce a novel schema for sequence to sequence learning with a Dee...

Sequential Segment-based Level Generation and Blending using Variational Autoencoders

Existing methods of level generation using latent variable models such a...