DeepAI AI Chat
Log In Sign Up

State-Denoised Recurrent Neural Networks

by   Michael C. Mozer, et al.
University of Colorado Boulder

Recurrent neural networks (RNNs) are difficult to train on sequence processing tasks, not only because input noise may be amplified through feedback, but also because any inaccuracy in the weights has similar consequences as input noise. We describe a method for denoising the hidden state during training to achieve more robust representations thereby improving generalization performance. Attractor dynamics are incorporated into the hidden state to `clean up' representations at each step of a sequence. The attractor dynamics are trained through an auxillary denoising loss to recover previously experienced hidden states from noisy versions of those states. This state-denoised recurrent neural network SDRNN performs multiple steps of internal processing for each external sequence step. On a range of tasks, we show that the SDRNN outperforms a generic RNN as well as a variant of the SDRNN with attractor dynamics on the hidden state but without the auxillary loss. We argue that attractor dynamics---and corresponding connectivity constraints---are an essential component of the deep learning arsenal and should be invoked not only for recurrent networks but also for improving deep feedforward nets and intertask transfer.


page 1

page 2

page 3

page 4


Internal representation dynamics and geometry in recurrent neural networks

The efficiency of recurrent neural networks (RNNs) in dealing with seque...

Convolutional Bipartite Attractor Networks

In human perception and cognition, the fundamental operation that brains...

Tustin neural networks: a class of recurrent nets for adaptive MPC of mechanical systems

The use of recurrent neural networks to represent the dynamics of unstab...

LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

Recurrent neural networks, and in particular long short-term memory (LST...

Reversible Recurrent Neural Networks

Recurrent neural networks (RNNs) provide state-of-the-art performance in...

Backpropagation on Dynamical Networks

Dynamical networks are versatile models that can describe a variety of b...