Monotone Learning with Rectifier Networks

05/10/2018
by   Veit Elser, et al.
0

We introduce a new neural network model, together with a tractable and monotone online learning algorithm. Our model describes feed-forward networks for classification, with one output node for each class. The only nonlinear operation is rectification using a ReLU function with a bias. However, there is a rectifier on every edge rather than at the nodes of the network. There are also weights, but these are positive, static, and associated with the nodes. Our "rectified wire" networks are able to represent arbitrary Boolean functions. Only the bias parameters, on the edges of the network, are learned. Another departure in our approach, from standard neural networks, is that the loss function is replaced by a constraint. This constraint is simply that the value of the output node associated with the correct class should be zero. Our model has the property that the exact norm-minimizing parameter update, required to correctly classify a training item, is the solution to a quadratic program that can be computed with a few passes through the network. We demonstrate a training algorithm using this update, called sequential deactivation (SDA), on MNIST and some synthetic datasets. Upon adopting a natural choice for the nodal weights, SDA has no hyperparameters other than those describing the network structure. Our experiments explore behavior with respect to network size and depth in a family of sparse expander networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/10/2018

Monotone Learning with Rectified Wire Networks

We introduce a new neural network model, together with a tractable and m...
research
07/12/2022

Size and depth of monotone neural networks: interpolation and approximation

Monotone functions and data sets arise in a variety of applications. We ...
research
07/01/2015

Natural Neural Networks

We introduce Natural Neural Networks, a novel family of algorithms that ...
research
02/02/2022

The Role of Linear Layers in Nonlinear Interpolating Networks

This paper explores the implicit bias of overparameterized neural networ...
research
07/25/2018

Unbounded Output Networks for Classification

We proposed the expected energy-based restricted Boltzmann machine (EE-R...
research
06/10/2019

Latent Channel Networks

Latent Euclidean embedding models a given network by representing each n...
research
02/21/2015

Positive Neural Networks in Discrete Time Implement Monotone-Regular Behaviors

We study the expressive power of positive neural networks. The model use...

Please sign up or login with your details

Forgot password? Click here to reset