DeepAI
Log In Sign Up

Surprisal-Triggered Conditional Computation with Neural Networks

06/02/2020
by   Loren Lugosch, et al.
0

Autoregressive neural network models have been used successfully for sequence generation, feature extraction, and hypothesis scoring. This paper presents yet another use for these models: allocating more computation to more difficult inputs. In our model, an autoregressive model is used both to extract features and to predict observations in a stream of input observations. The surprisal of the input, measured as the negative log-likelihood of the current observation according to the autoregressive model, is used as a measure of input difficulty. This in turn determines whether a small, fast network, or a big, slow network, is used. Experiments on two speech recognition tasks show that our model can match the performance of a baseline in which the big network is always used with 15

READ FULL TEXT

page 1

page 2

page 3

page 4

02/13/2020

Generalized Autoregressive Neural Network Models

A time series is a sequence of observations taken sequentially in time. ...
08/12/2021

Dereverberation of Autoregressive Envelopes for Far-field Speech Recognition

The task of speech recognition in far-field environments is adversely af...
09/16/2019

Global Autoregressive Models for Data-Efficient Sequence Learning

Standard autoregressive seq2seq models are easily trained by max-likelih...
06/13/2021

The DEformer: An Order-Agnostic Distribution Estimating Transformer

Order-agnostic autoregressive distribution estimation (OADE), i.e., auto...
02/20/2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

This paper presents the Imputer, a neural sequence model that generates ...
04/20/2017

Fast Generation for Convolutional Autoregressive Models

Convolutional autoregressive models have recently demonstrated state-of-...