Optimal short-term memory before the edge of chaos in driven random recurrent networks

12/24/2019
by   Taichi Haruna, et al.
31

The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated by mean-field theory. The combination of a small input strength and mean-field assumptions makes it possible to derive an approximate expression for the conditional probability density of the state of a neuron given a past input signal. From this conditional probability density, we can analytically calculate short-term memory measures, such as memory capacity, mutual information, and Fisher information, and determine the relationships among these measures, which have not been clarified to date to the best of our knowledge. We show that the network contribution of these short-term memory measures peaks before the edge of chaos, where the dynamics of input-driven networks is stable but corresponding systems without input signals are unstable.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/23/2016

Memory and Information Processing in Recurrent Neural Networks

Recurrent neural networks (RNN) are simple dynamical systems whose compu...
research
03/11/2016

Determination of the edge of criticality in echo state networks through Fisher information maximization

It is a widely accepted fact that the computational capability of recurr...
research
02/14/2018

Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks

The inherent transient dynamics of recurrent neural networks (RNNs) have...
research
04/14/2012

Neuroevolution Results in Emergence of Short-Term Memory for Goal-Directed Behavior

Animals behave adaptively in the environment with multiply competing goa...
research
07/01/2013

Short Term Memory Capacity in Networks via the Restricted Isometry Property

Cortical networks are hypothesized to rely on transient network activity...
research
06/30/2020

Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules

Robust perception relies on both bottom-up and top-down signals. Bottom-...
research
01/22/2016

Orthogonal Echo State Networks and stochastic evaluations of likelihoods

We report about probabilistic likelihood estimates that are performed on...

Please sign up or login with your details

Forgot password? Click here to reset