A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions

12/08/2020
by   Yutong Xie, et al.
0

The expressiveness of deep neural network (DNN) is a perspective to understandthe surprising performance of DNN. The number of linear regions, i.e. pieces thata piece-wise-linear function represented by a DNN, is generally used to measurethe expressiveness. And the upper bound of regions number partitioned by a rec-tifier network, instead of the number itself, is a more practical measurement ofexpressiveness of a rectifier DNN. In this work, we propose a new and tighter up-per bound of regions number. Inspired by the proof of this upper bound and theframework of matrix computation in Hinz Van de Geer (2019), we propose ageneral computational approach to compute a tight upper bound of regions numberfor theoretically any network structures (e.g. DNN with all kind of skip connec-tions and residual structures). Our experiments show our upper bound is tighterthan existing ones, and explain why skip connections and residual structures canimprove network performance.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

07/14/2020

Bounding The Number of Linear Regions in Local Area for Neural Networks with ReLU Activations

The number of linear regions is one of the distinct properties of the ne...
04/01/2021

An upper bound on the Universality of the Quantum Approximate Optimization Algorithm

Using lie algebra, this brief text provides an upper bound on the univer...
10/31/2018

Nearly-tight bounds on linear regions of piecewise linear neural networks

The developments of deep neural networks (DNN) in recent years have ushe...
10/10/2020

Improve the Robustness and Accuracy of Deep Neural Network with L_2,∞ Normalization

In this paper, the robustness and accuracy of the deep neural network (D...
04/02/2019

Why ResNet Works? Residuals Generalize

Residual connections significantly boost the performance of deep neural ...
01/04/2020

Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

A deep neural network (DNN) with piecewise linear activations can partit...
04/11/2022

Nonlocal Effect on a Generalized Ohta-Kawasaki Model

We propose a nonlocal Ohta-Kawasaki model to study the nonlocal effect o...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.