An Effective and Efficient Initialization Scheme for Training Multi-layer Feedforward Neural Networks

05/16/2020
by   Zebin Yang, et al.
0

Network initialization is the first and critical step for training neural networks. In this paper, we propose a novel network initialization scheme based on the celebrated Stein's identity. By viewing multi-layer feedforward neural networks as cascades of multi-index models, the projection weights to the first hidden layer are initialized using eigenvectors of the cross-moment matrix between the input's second-order score function and the response. The input data is then forward propagated to the next layer and such a procedure can be repeated until all the hidden layers are initialized. Finally, the weights for the output layer are initialized by generalized linear modeling. Such a proposed SteinGLM method is shown through extensive numerical results to be much faster and more accurate than other popular methods commonly used for training neural networks.

READ FULL TEXT
research
05/16/2020

An Effective and Efficient Initialization Scheme for Multi-layer Feedforward Neural Networks

Network initialization is the first and critical step for training neura...
research
05/16/2020

An Effective and Efficient Training Algorithm for Multi-layer Feedforward Neural Networks

Network initialization is the first and critical step for training neura...
research
09/18/2019

A Study on Binary Neural Networks Initialization

Initialization plays a crucial role in training neural models. Binary Ne...
research
04/20/2020

Revisiting Initialization of Neural Networks

Good initialization of weights is crucial for effective training of deep...
research
06/26/2023

Scaling and Resizing Symmetry in Feedforward Networks

Weights initialization in deep neural networks have a strong impact on t...
research
04/05/2020

Backprojection for Training Feedforward Neural Networks in the Input and Feature Spaces

After the tremendous development of neural networks trained by backpropa...
research
02/26/2021

Layer-Wise Interpretation of Deep Neural Networks Using Identity Initialization

The interpretability of neural networks (NNs) is a challenging but essen...

Please sign up or login with your details

Forgot password? Click here to reset