Convergence of Deep ReLU Networks

07/27/2021
by   Yuesheng Xu, et al.
0

We explore convergence of deep neural networks with the popular ReLU activation function, as the depth of the networks tends to infinity. To this end, we introduce the notion of activation domains and activation matrices of a ReLU network. By replacing applications of the ReLU activation function by multiplications with activation matrices on activation domains, we obtain an explicit expression of the ReLU network. We then identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices. Sufficient and necessary conditions for convergence of these infinite products of matrices are studied. As a result, we establish necessary conditions for ReLU networks to converge that the sequence of weight matrices converges to the identity matrix and the sequence of the bias vectors converges to zero as the depth of ReLU networks increases to infinity. Moreover, we obtain sufficient conditions in terms of the weight matrices and bias vectors at hidden layers for pointwise convergence of deep ReLU networks. These results provide mathematical insights to the design strategy of the well-known deep residual networks in image classification.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/28/2021

Convergence of Deep Convolutional Neural Networks

Convergence of deep neural networks as the depth of the networks tends t...
research
05/13/2022

Convergence Analysis of Deep Residual Networks

Various powerful deep neural network architectures have made great contr...
research
12/14/2018

Products of Many Large Random Matrices and Gradients in Deep Neural Networks

We study products of random matrices in the regime where the number of t...
research
11/30/2021

Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks

We investigate the Fisher information matrix (FIM) of one hidden layer n...
research
05/02/2023

Hamming Similarity and Graph Laplacians for Class Partitioning and Adversarial Image Detection

Researchers typically investigate neural network representations by exam...
research
07/20/2021

An Embedding of ReLU Networks and an Analysis of their Identifiability

Neural networks with the Rectified Linear Unit (ReLU) nonlinearity are d...
research
06/01/2022

Rotate the ReLU to implicitly sparsify deep networks

In the era of Deep Neural Network based solutions for a variety of real-...

Please sign up or login with your details

Forgot password? Click here to reset