Self-Organization Towards 1/f Noise in Deep Neural Networks

01/20/2023
by   Nicholas Chong Jia Le, et al.
0

Despite 1/f noise being ubiquitous in both natural and artificial systems, no general explanations for the phenomenon have received widespread acceptance. One well-known system where 1/f noise has been observed in is the human brain, with this 'noise' proposed by some to be important to the healthy function of the brain. As deep neural networks (DNNs) are loosely modelled after the human brain, and as they start to achieve human-level performance in specific tasks, it might be worth investigating if the same 1/f noise is present in these artificial networks as well. Indeed, we find the existence of 1/f noise in DNNs - specifically Long Short-Term Memory (LSTM) networks modelled on real world dataset - by measuring the Power Spectral Density (PSD) of different activations within the network in response to a sequential input of natural language. This was done in analogy to the measurement of 1/f noise in human brains with techniques such as electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI). We further examine the exponent values in the 1/f noise in "inner" and "outer" activations in the LSTM cell, finding some resemblance in the variations of the exponents in the fMRI signal. In addition, comparing the values of the exponent at "rest" compared to when performing "tasks" of the LSTM network, we find a similar trend to that of the human brain where the exponent while performing tasks is less negative.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/28/2021

Understanding Cognitive Fatigue from fMRI Scans with Self-supervised Learning

Functional magnetic resonance imaging (fMRI) is a neuroimaging technique...
research
04/22/2016

Bridging LSTM Architecture and the Neural Dynamics during Reading

Recently, the long short-term memory neural network (LSTM) has attracted...
research
10/15/2019

Jointly Discriminative and Generative Recurrent Neural Networks for Learning from fMRI

Recurrent neural networks (RNNs) were designed for dealing with time-ser...
research
08/20/2022

Net2Brain: A Toolbox to compare artificial vision models with human brain responses

We introduce Net2Brain, a graphical and command-line user interface tool...
research
04/23/2020

Evaluating Adversarial Robustness for Deep Neural Network Interpretability using fMRI Decoding

While deep neural networks (DNNs) are being increasingly used to make pr...
research
03/27/2023

Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain

Linking computational natural language processing (NLP) models and neura...

Please sign up or login with your details

Forgot password? Click here to reset