Constant Error Carousel

What is a Constant Error Carousel?

A specified type of recurrent neural network architecture is called a Long Short-Term Memory (LTSM). Unlike a feedforward network, LTSM architecture incorporates feedback, allowing it to function as a general purpose computer. While there are many different forms of LTSM architecture, to understand the Constant Error Carousel, we will examine a simple framework.

Source

One cell consists of a cell unit, and three gates (input, forget, and output). The gates use a sigmoid activation function, and the input and cell state is usually transformed by tanh, another activation function. The gating mechanism can hold information for long durations, however vanilla LSTMs don't have a forget gate and instead add an unchanged cell state (e.g. a recurrent connection with a constant weight of 1). This addition is called the Constant Error Carousel because it solves the training problem of vanishing and exploding gradients. In networks that do contain a forget gate, the Constant Error Carousel may be reset by the forget gate. The addition of the Constant Error Carousel allows for the LSTM to learn long-term relationships while mitigating the risks of prolonged testing.