Controlling Recurrent Neural Networks by Diagonal Conceptors

07/16/2021
by   J. P. de Jong, et al.
0

The human brain is capable of learning, memorizing, and regenerating a panoply of temporal patterns. A neuro-dynamical mechanism called conceptors offers a method for controlling the dynamics of a recurrent neural network by which a variety of temporal patterns can be learned and recalled. However, conceptors are matrices whose size scales quadratically with the number of neurons in the recurrent neural network, hence they quickly become impractical. In the work reported in this thesis, a variation of conceptors is introduced, called diagonal conceptors, which are diagonal matrices, thus reducing the computational cost drastically. It will be shown that diagonal conceptors achieve the same accuracy as conceptors, but are slightly more unstable. This instability can be improved, but requires further research. Nevertheless, diagonal conceptors show to be a promising practical alternative to the standard full matrix conceptors.

READ FULL TEXT

page 33

page 34

page 35

research
04/18/2017

Diagonal RNNs in Symbolic Music Modeling

In this paper, we propose a new Recurrent Neural Network (RNN) architect...
research
03/13/2014

Controlling Recurrent Neural Networks by Conceptors

The human brain is a dynamical system whose extremely complex sensor-dri...
research
02/23/2017

Inherent Biases of Recurrent Neural Networks for Phonological Assimilation and Dissimilation

A recurrent neural network model of phonological pattern learning is pro...
research
06/30/2016

Programming Patterns in Dataflow Matrix Machines and Generalized Recurrent Neural Nets

Dataflow matrix machines arise naturally in the context of synchronous d...
research
03/23/2023

Noise impact on recurrent neural network with linear activation function

In recent years, more and more researchers in the field of neural networ...
research
12/26/2017

Chaos-guided Input Structuring for Improved Learning in Recurrent Neural Networks

Anatomical studies demonstrate that brain reformats input information to...
research
02/25/2016

Practical Riemannian Neural Networks

We provide the first experimental results on non-synthetic datasets for ...

Please sign up or login with your details

Forgot password? Click here to reset