Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

01/04/2020
by   Xiao Zhang, et al.
28

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.

READ FULL TEXT

page 3

page 4

page 5

page 9

page 10

page 11

page 12

page 15

research
11/06/2017

Bounding and Counting Linear Regions of Deep Neural Networks

In this paper, we study the representational power of deep neural networ...
research
10/23/2020

On the Number of Linear Functions Composing Deep Neural Network: Towards a Refined Definition of Neural Networks Complexity

The classical approach to measure the expressive power of deep neural ne...
research
05/27/2019

Expression of Fractals Through Neural Network Functions

To help understand the underlying mechanisms of neural networks (NNs), s...
research
03/07/2020

Geometry and Topology of Deep Neural Networks' Decision Boundaries

Geometry and topology of decision regions are closely related with class...
research
02/12/2020

Fast Geometric Projections for Local Robustness Certification

Local robustness ensures that a model classifies all inputs within an ϵ-...
research
10/31/2018

Nearly-tight bounds on linear regions of piecewise linear neural networks

The developments of deep neural networks (DNN) in recent years have ushe...
research
12/08/2020

A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions

The expressiveness of deep neural network (DNN) is a perspective to unde...

Please sign up or login with your details

Forgot password? Click here to reset