Assessing incrementality in sequence-to-sequence models

06/07/2019
by   Dennis Ulmer, et al.
0

Since their inception, encoder-decoder models have successfully been applied to a wide array of problems in computational linguistics. The most recent successes are predominantly due to the use of different variations of attention mechanisms, but their cognitive plausibility is questionable. In particular, because past representations can be revisited at any point in time, attention-centric methods seem to lack an incentive to build up incrementally more informative representations of incoming sentences. This way of processing stands in stark contrast with the way in which humans are believed to process language: continuously and rapidly integrating new information as it is encountered. In this work, we propose three novel metrics to assess the behavior of RNNs with and without an attention mechanism and identify key differences in the way the different model types process sentences.

READ FULL TEXT
research
12/21/2017

Variational Attention for Sequence-to-Sequence Models

The variational encoder-decoder (VED) encodes source information as a se...
research
10/28/2021

Understanding How Encoder-Decoder Architectures Attend

Encoder-decoder networks with attention have proven to be a powerful way...
research
07/01/2017

Efficient Attention using a Fixed-Size Memory Representation

The standard content-based attention mechanism typically used in sequenc...
research
06/13/2018

Double Path Networks for Sequence to Sequence Learning

Encoder-decoder based Sequence to Sequence learning (S2S) has made remar...
research
10/20/2016

Jointly Learning to Align and Convert Graphemes to Phonemes with Neural Attention Models

We propose an attention-enabled encoder-decoder model for the problem of...
research
01/03/2021

Recoding latent sentence representations – Dynamic gradient-based activation modification in RNNs

In Recurrent Neural Networks (RNNs), encoding information in a suboptima...
research
08/28/2018

Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue

We investigate how encoder-decoder models trained on a synthetic dataset...

Please sign up or login with your details

Forgot password? Click here to reset