Surprisal-Triggered Conditional Computation with Neural Networks

06/02/2020
by   Loren Lugosch, et al.
0

Autoregressive neural network models have been used successfully for sequence generation, feature extraction, and hypothesis scoring. This paper presents yet another use for these models: allocating more computation to more difficult inputs. In our model, an autoregressive model is used both to extract features and to predict observations in a stream of input observations. The surprisal of the input, measured as the negative log-likelihood of the current observation according to the autoregressive model, is used as a measure of input difficulty. This in turn determines whether a small, fast network, or a big, slow network, is used. Experiments on two speech recognition tasks show that our model can match the performance of a baseline in which the big network is always used with 15

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/13/2020

Generalized Autoregressive Neural Network Models

A time series is a sequence of observations taken sequentially in time. ...
research
08/12/2021

Dereverberation of Autoregressive Envelopes for Far-field Speech Recognition

The task of speech recognition in far-field environments is adversely af...
research
09/16/2019

Global Autoregressive Models for Data-Efficient Sequence Learning

Standard autoregressive seq2seq models are easily trained by max-likelih...
research
06/08/2023

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking

In many domains, autoregressive models can achieve low log-likelihood on...
research
05/17/2021

Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics

This paper introduces an alternative approach to sampling from autoregre...
research
02/20/2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

This paper presents the Imputer, a neural sequence model that generates ...

Please sign up or login with your details

Forgot password? Click here to reset