Biologically inspired sleep algorithm for artificial neural networks

08/01/2019
by   Giri P Krishnan, et al.
24

Sleep plays an important role in incremental learning and consolidation of memories in biological systems. Motivated by the processes that are known to be involved in sleep generation in biological networks, we developed an algorithm that implements a sleep-like phase in artificial neural networks (ANNs). After initial training phase, we convert the ANN to a spiking neural network (SNN) and simulate an offline sleep-like phase using spike-timing dependent plasticity rules to modify synaptic weights. The SNN is then converted back to the ANN and evaluated or trained on new inputs. We demonstrate several performance improvements after applying this processing to ANNs trained on MNIST, CUB200 and a motivating toy dataset. First, in an incremental learning framework, sleep is able to recover older tasks that were otherwise forgotten in the ANN without sleep phase due to catastrophic forgetting. Second, sleep results in forward transfer learning of unseen tasks. Finally, sleep improves generalization ability of the ANNs to classify images with various types of noise. We provide a theoretical basis for the beneficial role of the brain-inspired sleep-like phase for the ANNs and present an algorithmic way for future implementations of the various features of sleep in deep learning ANNs. Overall, these results suggest that biological sleep can help mitigate a number of problems ANNs suffer from, such as poor generalization and catastrophic forgetting for incremental learning.

READ FULL TEXT

page 6

page 7

page 12

page 13

research
11/18/2021

Continuous learning of spiking networks trained with local rules

Artificial neural networks (ANNs) experience catastrophic forgetting (CF...
research
03/18/2017

A wake-sleep algorithm for recurrent, spiking neural networks

We investigate a recently proposed model for cortical computation which ...
research
03/26/2020

Thalamo-cortical spiking model of incremental learning combining perception, context and NREM-sleep-mediated noise-resilience

The brain exhibits capabilities of fast incremental learning from few no...
research
02/02/2023

Simple method for detecting sleep episodes in rats ECoG using machine learning

In this paper we propose a new method for the automatic recognition of t...
research
10/24/2018

Sleep-like slow oscillations induce hierarchical memory association and synaptic homeostasis in thalamo-cortical simulations

The occurrence of sleep is widespread over the large majority of animal ...
research
05/25/2023

Dendritic Integration Based Quadratic Neural Networks Outperform Traditional Aritificial Ones

Incorporating biological neuronal properties into Artificial Neural Netw...

Please sign up or login with your details

Forgot password? Click here to reset