Activation Ensembles for Deep Neural Networks

02/24/2017
by   Mark Harmon, et al.
0

Many activation functions have been proposed in the past, but selecting an adequate one requires trial and error. We propose a new methodology of designing activation functions within a neural network at each layer. We call this technique an "activation ensemble" because it allows the use of multiple activation functions at each layer. This is done by introducing additional variables, α, at each activation layer of a network to allow for multiple activation functions to be active at each neuron. By design, activations with larger α values at a neuron is equivalent to having the largest magnitude. Hence, those higher magnitude activations are "chosen" by the network. We implement the activation ensembles on a variety of datasets using an array of Feed Forward and Convolutional Neural Networks. By using the activation ensemble, we achieve superior results compared to traditional techniques. In addition, because of the flexibility of this methodology, we more deeply explore activation functions and the features that they capture.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/19/2019

Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions

Most deep neural networks use simple, fixed activation functions, such a...
research
06/01/2019

Evolution of Novel Activation Functions in Neural Network Training with Applications to Classification of Exoplanets

We present analytical exploration of novel activation functions as conse...
research
01/13/2023

Neural network with optimal neuron activation functions based on additive Gaussian process regression

Feed-forward neural networks (NN) are a staple machine learning method w...
research
10/13/2019

Hardware/Software Codesign for Training/Testing Multiple Neural Networks on Multiple FPGAs

Most neural network designs for FPGAs are inflexible. In this paper, we ...
research
06/08/2023

Layer-level activation mechanism

In this work, we propose a novel activation mechanism aimed at establish...
research
11/10/2018

PolyNeuron: Automatic Neuron Discovery via Learned Polyharmonic Spline Activations

Automated deep neural network architecture design has received a signifi...
research
03/01/2023

A Lifted Bregman Formulation for the Inversion of Deep Neural Networks

We propose a novel framework for the regularised inversion of deep neura...

Please sign up or login with your details

Forgot password? Click here to reset