Parametric Variational Linear Units (PVLUs) in Deep Convolutional Networks

10/23/2021
by   Aarush Gupta, et al.
0

The Rectified Linear Unit is currently a state-of-the-art activation function in deep convolutional neural networks. To combat ReLU's dying neuron problem, we propose the Parametric Variational Linear Unit (PVLU), which adds a sinusoidal function with trainable coefficients to ReLU. Along with introducing nonlinearity and non-zero gradients across the entire real domain, PVLU allows for increased model generalization and robustness when implemented in the context of transfer learning. On a simple, non-transfer sequential CNN, PVLU led to relative error decrease of 16.3 augmentation, relative to ReLU. PVLU is also tested on transfer learning problems. The VGG-16 and VGG-19 models experience relative error reductions of 9.5 PVLU. When training on Gaussian-filtered CIFAR-10 images, similar improvements are noted for the VGG models. Most notably, PVLU fine tuning allows for relative error reductions up to and exceeding 10 ResNet models for both CIFAR-10 and CIFAR-100.

READ FULL TEXT
research
05/30/2016

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

The activation function is an important component in Convolutional Neura...
research
06/25/2017

Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Rectified linear unit (ReLU) is a widely used activation function for de...
research
05/11/2015

Improving neural networks with bunches of neurons modeled by Kumaraswamy units: Preliminary study

Deep neural networks have recently achieved state-of-the-art results in ...
research
03/22/2020

Dynamic ReLU

Rectified linear units (ReLU) are commonly used in deep neural networks....
research
05/22/2018

ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

This work introduces a novel activation unit that can be efficiently emp...
research
09/22/2017

EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks

For most state-of-the-art architectures, Rectified Linear Unit (ReLU) be...
research
08/07/2021

Impact of Aliasing on Generalization in Deep Convolutional Networks

We investigate the impact of aliasing on generalization in Deep Convolut...

Please sign up or login with your details

Forgot password? Click here to reset