Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

09/11/2018
by   Jinhyeok Jang, et al.
0

This work presents deep asymmetric networks with a set of node-wise variant activation functions. The nodes' sensitivities are affected by activation function selections such that the nodes with smaller indices become increasingly more sensitive. As a result, features learned by the nodes are sorted by the node indices in the order of their importance. Asymmetric networks not only learn input features but also the importance of those features. Nodes of lesser importance in asymmetric networks can be pruned to reduce the complexity of the networks, and the pruned networks can be retrained without incurring performance losses. We validate the feature-sorting property using both shallow and deep asymmetric networks as well as deep asymmetric networks transferred from famous networks.

READ FULL TEXT
research
11/21/2018

Neural Networks with Activation Networks

This work presents an adaptive activation method for neural networks tha...
research
03/21/2017

Evolving Parsimonious Networks by Mixing Activation Functions

Neuroevolution methods evolve the weights of a neural network, and in so...
research
07/29/2021

Otimizacao de pesos e funcoes de ativacao de redes neurais aplicadas na previsao de series temporais

Neural Networks have been applied for time series prediction with good e...
research
05/19/2020

Asymmetric scale functions for t-digests

The t-digest is a data structure that can be queried for approximate qua...
research
03/29/2021

Restricted Boltzmann Machines as Models of Interacting Variables

We study the type of distributions that Restricted Boltzmann Machines (R...
research
07/03/2023

An analysis on stochastic Lanczos quadrature with asymmetric quadrature nodes

The stochastic Lanczos quadrature method has garnered significant attent...
research
07/11/2022

The Mean Dimension of Neural Networks – What causes the interaction effects?

Owen and Hoyt recently showed that the effective dimension offers key st...

Please sign up or login with your details

Forgot password? Click here to reset