DeepAI AI Chat
Log In Sign Up

Plan, Attend, Generate: Planning for Sequence-to-Sequence Models

11/28/2017
by   Francis Dutil, et al.
Microsoft
0

We investigate the integration of a planning mechanism into sequence-to-sequence models using attention. We develop a model which can plan ahead in the future when it computes its alignments between input and output sequences, constructing a matrix of proposed future alignments and a commitment vector that governs whether to follow or recompute the plan. This mechanism is inspired by the recently proposed strategic attentive reader and writer (STRAW) model for Reinforcement Learning. Our proposed model is end-to-end trainable using primarily differentiable operations. We show that it outperforms a strong baseline on character-level translation tasks from WMT'15, the algorithmic task of finding Eulerian circuits of graphs, and question generation from the text. Our analysis demonstrates that the model computes qualitatively intuitive alignments, converges faster than the baselines, and achieves superior performance with fewer parameters.

READ FULL TEXT
06/13/2017

Plan, Attend, Generate: Character-level Neural Machine Translation with Planning in the Decoder

We investigate the integration of a planning mechanism into an encoder-d...
01/13/2020

ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training

In this paper, we present a new sequence-to-sequence pre-training model ...
11/21/2019

Improving Conditioning in Context-Aware Sequence to Sequence Models

Neural sequence to sequence models are well established for applications...
11/01/2018

Sequence-to-sequence Models for Small-Footprint Keyword Spotting

In this paper, we propose a sequence-to-sequence model for keyword spott...
06/15/2016

Strategic Attentive Writer for Learning Macro-Actions

We present a novel deep recurrent neural network architecture that learn...
07/04/2017

CharManteau: Character Embedding Models For Portmanteau Creation

Portmanteaus are a word formation phenomenon where two words are combine...
05/20/2018

Learning compositionally through attentive guidance

In this paper, we introduce Attentive Guidance (AG), a new mechanism to ...

Code Repositories

PAG

Plan, Attend, Generate: Planning for Sequence-to-Sequence Models


view repo