Surprisal-Triggered Conditional Computation with Neural Networks

06/02/2020
by   Loren Lugosch, et al.
0

Autoregressive neural network models have been used successfully for sequence generation, feature extraction, and hypothesis scoring. This paper presents yet another use for these models: allocating more computation to more difficult inputs. In our model, an autoregressive model is used both to extract features and to predict observations in a stream of input observations. The surprisal of the input, measured as the negative log-likelihood of the current observation according to the autoregressive model, is used as a measure of input difficulty. This in turn determines whether a small, fast network, or a big, slow network, is used. Experiments on two speech recognition tasks show that our model can match the performance of a baseline in which the big network is always used with 15

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/13/2020

Generalized Autoregressive Neural Network Models

A time series is a sequence of observations taken sequentially in time. ...
08/12/2021

Dereverberation of Autoregressive Envelopes for Far-field Speech Recognition

The task of speech recognition in far-field environments is adversely af...
09/16/2019

Global Autoregressive Models for Data-Efficient Sequence Learning

Standard autoregressive seq2seq models are easily trained by max-likelih...
05/17/2021

Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics

This paper introduces an alternative approach to sampling from autoregre...
06/13/2021

The DEformer: An Order-Agnostic Distribution Estimating Transformer

Order-agnostic autoregressive distribution estimation (OADE), i.e., auto...
02/20/2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

This paper presents the Imputer, a neural sequence model that generates ...
02/12/2015

MADE: Masked Autoencoder for Distribution Estimation

There has been a lot of recent interest in designing neural network mode...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.