Towards Robust, Locally Linear Deep Networks

07/07/2019
by   Guang-He Lee, et al.
2

Deep networks realize complex mappings that are often understood by their locally linear behavior at or around points of interest. For example, we use the derivative of the mapping with respect to its inputs for sensitivity analysis, or to explain (obtain coordinate relevance for) a prediction. One key challenge is that such derivatives are themselves inherently unstable. In this paper, we propose a new learning problem to encourage deep networks to have stable derivatives over larger regions. While the problem is challenging in general, we focus on networks with piecewise linear activation functions. Our algorithm consists of an inference step that identifies a region around a point where linear approximation is provably stable, and an optimization step to expand such regions. We propose a novel relaxation to scale the algorithm to realistic models. We illustrate our method with residual and recurrent networks on image and sequence datasets.

READ FULL TEXT

page 6

page 10

page 20

page 21

research
11/11/2015

Piecewise Linear Activation Functions For More Efficient Deep Networks

This submission has been withdrawn by arXiv administrators because it is...
research
02/08/2014

On the Number of Linear Regions of Deep Neural Networks

We study the complexity of functions computable by deep feedforward neur...
research
06/16/2020

Measuring Model Complexity of Neural Networks with Curve Activation Functions

It is fundamental to measure model complexity of deep neural networks. T...
research
06/11/2020

On the asymptotics of wide networks with polynomial activations

We consider an existing conjecture addressing the asymptotic behavior of...
research
12/22/2015

Deep Learning with S-shaped Rectified Linear Activation Units

Rectified linear activation units are important components for state-of-...
research
02/10/2020

Reducing the Computational Burden of Deep Learning with Recursive Local Representation Alignment

Training deep neural networks on large-scale datasets requires significa...
research
06/16/2019

A Closer Look at Double Backpropagation

In recent years, an increasing number of neural network models have incl...

Please sign up or login with your details

Forgot password? Click here to reset