Revisiting Initialization of Neural Networks

04/20/2020

∙

Good initialization of weights is crucial for effective training of deep neural networks. In this paper we discuss an intialization scheme based on rigorous estimation of the local curvature. The proposed approach is more systematic and general than the state-of-art intialization from Glorot at al. and follow-up works.

READ FULL TEXT

Revisiting Initialization of Neural Networks

An Effective and Efficient Initialization Scheme for Training Multi-layer Feedforward Neural Networks

An Effective and Efficient Initialization Scheme for Multi-layer Feedforward Neural Networks

AutoInit: Automatic Initialization via Jacobian Tuning

Historical Document Image Segmentation with LDA-Initialized Deep Neural Networks

Curvature-based Comparison of Two Neural Networks

Neuron Campaign for Initialization Guided by Information Bottleneck Theory

A Bayesian approach for initialization of weights in backpropagation neural net with application to character recognition

Revisiting Initialization of Neural Networks

Related Research

An Effective and Efficient Initialization Scheme for Training Multi-layer Feedforward Neural Networks

An Effective and Efficient Initialization Scheme for Multi-layer Feedforward Neural Networks

AutoInit: Automatic Initialization via Jacobian Tuning

Historical Document Image Segmentation with LDA-Initialized Deep Neural Networks

Curvature-based Comparison of Two Neural Networks

Neuron Campaign for Initialization Guided by Information Bottleneck Theory

A Bayesian approach for initialization of weights in backpropagation neural net with application to character recognition