Evolving Parsimonious Networks by Mixing Activation Functions

03/21/2017
by   Alexander Hagg, et al.
0

Neuroevolution methods evolve the weights of a neural network, and in some cases the topology, but little work has been done to analyze the effect of evolving the activation functions of individual nodes on network size, which is important when training networks with a small number of samples. In this work we extend the neuroevolution algorithm NEAT to evolve the activation function of neurons in addition to the topology and weights of the network. The size and performance of networks produced using NEAT with uniform activation in all nodes, or homogenous networks, is compared to networks which contain a mixture of activation functions, or heterogenous networks. For a number of regression and classification benchmarks it is shown that, (1) qualitatively different activation functions lead to different results in homogeneous networks, (2) the heterogeneous version of NEAT is able to select well performing activation functions, (3) producing heterogeneous networks that are significantly smaller than homogeneous networks.

READ FULL TEXT
research
05/17/2016

Combinatorially Generated Piecewise Activation Functions

In the neuroevolution literature, research has primarily focused on evol...
research
05/22/2019

Effect of shapes of activation functions on predictability in the echo state network

We investigate prediction accuracy for time series of Echo state network...
research
09/11/2018

Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

This work presents deep asymmetric networks with a set of node-wise vari...
research
07/03/2018

Learning concise representations for regression by evolving networks of trees

We propose and study a method for learning interpretable representations...
research
06/13/2023

Safe Use of Neural Networks

Neural networks in modern communication systems can be susceptible to in...
research
12/02/2019

A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning

One of the distinguishing characteristics of modern deep learning system...

Please sign up or login with your details

Forgot password? Click here to reset