Asymptotic Stability in Reservoir Computing

06/07/2022
by   Jonathan Dong, et al.
0

Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel limit to draw new insights on stability in reservoir computing. This limit corresponds to large reservoir sizes, and it already becomes relevant for reservoirs with a few hundred neurons. We obtain a quantitative characterization of the frontier between stability and chaos, which can greatly benefit hyperparameter tuning. In a broader sense, our results contribute to understanding the complex dynamics of Recurrent Neural Networks.

READ FULL TEXT

page 5

page 7

research
06/12/2020

Reservoir Computing meets Recurrent Kernels and Structured Transforms

Reservoir Computing is a class of simple yet efficient Recurrent Neural ...
research
01/03/2020

Memory-Loss is Fundamental for Stability and Distinguishes the Echo State Property Threshold in Reservoir Computing Beyond

Reservoir computing, a highly successful neuromorphic computing scheme u...
research
07/17/2023

Reducing hyperparameter dependence by external timescale tailoring

Task specific hyperparameter tuning in reservoir computing is an open is...
research
05/08/2019

Evaluating the Stability of Recurrent Neural Models during Training with Eigenvalue Spectra Analysis

We analyze the stability of recurrent networks, specifically, reservoir ...
research
10/03/2018

A characterization of the Edge of Criticality in Binary Echo State Networks

Echo State Networks (ESNs) are simplified recurrent neural network model...
research
05/19/2022

Analyzing Echo-state Networks Using Fractal Dimension

This work joins aspects of reservoir optimization, information-theoretic...
research
11/11/2022

Re-visiting Reservoir Computing architectures optimized by Evolutionary Algorithms

For many years, Evolutionary Algorithms (EAs) have been applied to impro...

Please sign up or login with your details

Forgot password? Click here to reset