Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks

11/07/2013
by   Caglar Gulcehre, et al.
0

In this paper we propose and investigate a novel nonlinear unit, called L_p unit, for deep neural networks. The proposed L_p unit receives signals from several projections of a subset of units in the layer below and computes a normalized L_p norm. We notice two interesting interpretations of the L_p unit. First, the proposed unit can be understood as a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks (CNN), HMAX models and neocognitrons. Furthermore, the L_p unit is, to a certain degree, similar to the recently proposed maxout unit (Goodfellow et al., 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Secondly, we provide a geometrical interpretation of the activation function based on which we argue that the L_p unit is more efficient at representing complex, nonlinear separating boundaries. Each L_p unit defines a superelliptic boundary, with its exact shape defined by the order p. We claim that this makes it possible to model arbitrarily shaped, curved boundaries more efficiently by combining a few L_p units of different orders. This insight justifies the need for learning different orders for each unit in the model. We empirically evaluate the proposed L_p units on a number of datasets and show that multilayer perceptrons (MLP) consisting of the L_p units achieve the state-of-the-art results on a number of benchmark datasets. Furthermore, we evaluate the proposed L_p unit on the recently proposed deep recurrent neural networks (RNN).

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/03/2015

ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks

In this paper, we propose a deep neural network architecture for object ...
research
05/11/2015

Improving neural networks with bunches of neurons modeled by Kumaraswamy units: Preliminary study

Deep neural networks have recently achieved state-of-the-art results in ...
research
09/30/2015

Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

We seek to improve deep neural networks by generalizing the pooling oper...
research
04/08/2016

Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)

We propose a novel activation function that implements piece-wise orthog...
research
07/25/2017

Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks

In this paper, we introduce a novel type of Rectified Linear Unit (ReLU)...
research
03/14/2018

Building Sparse Deep Feedforward Networks using Tree Receptive Fields

Sparse connectivity is an important factor behind the success of convolu...
research
12/20/2013

Improving Deep Neural Networks with Probabilistic Maxout Units

We present a probabilistic variant of the recently introduced maxout uni...

Please sign up or login with your details

Forgot password? Click here to reset