Transfer entropy-based feedback improves performance in artificial neural networks

06/13/2017
by   Sebastian Herzog, et al.
0

The structure of the majority of modern deep neural networks is characterized by uni- directional feed-forward connectivity across a very large number of layers. By contrast, the architecture of the cortex of vertebrates contains fewer hierarchical levels but many recurrent and feedback connections. Here we show that a small, few-layer artificial neural network that employs feedback will reach top level performance on a standard benchmark task, otherwise only obtained by large feed-forward structures. To achieve this we use feed-forward transfer entropy between neurons to structure feedback connectivity. Transfer entropy can here intuitively be understood as a measure for the relevance of certain pathways in the network, which are then amplified by feedback. Feedback may therefore be key for high network performance in small brain-like architectures.

READ FULL TEXT

page 3

page 6

research
07/24/2020

Linear discriminant initialization for feed-forward neural networks

Informed by the basic geometry underlying feed forward neural networks, ...
research
12/24/2019

FHDR: HDR Image Reconstruction from a Single LDR Image using Feedback Network

High dynamic range (HDR) image generation from a single exposure low dyn...
research
07/24/2021

Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers

Feed-forward neural networks consist of a sequence of layers, in which e...
research
04/29/2021

Learning in Feedforward Neural Networks Accelerated by Transfer Entropy

Current neural networks architectures are many times harder to train bec...
research
01/06/2023

Feedback-Gated Rectified Linear Units

Feedback connections play a prominent role in the human brain but have n...
research
09/06/2023

Split-Boost Neural Networks

The calibration and training of a neural network is a complex and time-c...
research
04/19/2023

Learning Hierarchically-Structured Concepts II: Overlapping Concepts, and Networks With Feedback

We continue our study from Lynch and Mallmann-Trenn (Neural Networks, 20...

Please sign up or login with your details

Forgot password? Click here to reset