The Ant Swarm Neuro-Evolution Procedure for Optimizing Recurrent Networks

09/26/2019
by   AbdElRahman A. ElSaid, et al.
0

Hand-crafting effective and efficient structures for recurrent neural networks (RNNs) is a difficult, expensive, and time-consuming process. To address this challenge, we propose a novel neuro-evolution algorithm based on ant colony optimization (ACO), called ant swarm neuro-evolution (ASNE), for directly optimizing RNN topologies. The procedure selects from multiple modern recurrent cell types such as Delta-RNN, GRU, LSTM, MGU and UGRNN cells, as well as recurrent connections which may span multiple layers and/or steps of time. In order to introduce an inductive bias that encourages the formation of sparser synaptic connectivity patterns, we investigate several variations of the core algorithm. We do so primarily by formulating different functions that drive the underlying pheromone simulation process (which mimic L1 and L2 regularization in standard machine learning) as well as by introducing ant agents with specialized roles (inspired by how real ant colonies operate), i.e., explorer ants that construct the initial feed forward structure and social ants which select nodes from the feed forward connections to subsequently craft recurrent memory structures. We also incorporate a Lamarckian strategy for weight initialization which reduces the number of backpropagation epochs required to locally train candidate RNNs, speeding up the neuro-evolution process. Our results demonstrate that the sparser RNNs evolved by ASNE significantly outperform traditional one and two layer architectures consisting of modern memory cells, as well as the well-known NEAT algorithm. Furthermore, we improve upon prior state-of-the-art results on the time series dataset utilized in our experiments.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/06/2019

Investigating RNN Memory using Neuro-Evolution: Investigating Recurrent Neural Network Memory Structures using Neuro-Evolution

This paper presents a new algorithm, Evolutionary eXploration of Augment...
research
09/20/2019

An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution

Neuro-evolution and neural architecture search algorithms have gained in...
research
07/27/2023

Fading memory as inductive bias in residual recurrent networks

Residual connections have been proposed as architecture-based inductive ...
research
02/06/2019

Investigating Recurrent Neural Network Memory Structures using Neuro-Evolution

This paper presents a new algorithm, Evolutionary eXploration of Augment...
research
09/21/2020

An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution

Weight initialization is critical in being able to successfully train ar...
research
06/18/2017

Learning Hierarchical Information Flow with Recurrent Neural Modules

We propose ThalNet, a deep learning model inspired by neocortical commun...
research
03/17/2021

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

The predictive learning of spatiotemporal sequences aims to generate fut...

Please sign up or login with your details

Forgot password? Click here to reset