Guided Layer-wise Learning for Deep Models using Side Information

by   Pavel Sulimov, et al.

Training of deep models for classification tasks is hindered by local minima problems and vanishing gradients, while unsupervised layer-wise pretraining does not exploit information from class labels. Here, we propose a new regularization technique, called diversifying regularization (DR), which applies a penalty on hidden units at any layer if they obtain similar features for different types of data. For generative models, DR is defined as divergence over the variational posteriori distributions and included in the maximum likelihood estimation as a prior. Thus, DR includes class label information for greedy pretraining of deep belief networks which result in a better weight initialization for fine-tuning methods. On the other hand, for discriminative training of deep neural networks, DR is defined as a distance over the features and included in the learning objective. With our experimental tests, we show that DR can help the backpropagation to cope with vanishing gradient problems and to provide faster convergence and smaller generalization errors.


page 1

page 2

page 3

page 4


A Label Management Mechanism for Retinal Fundus Image Classification of Diabetic Retinopathy

Diabetic retinopathy (DR) remains the most prevalent cause of vision imp...

Towards the Localisation of Lesions in Diabetic Retinopathy

Convolutional Neural Networks (CNN) has successfully been used to classi...

Local minima in training of neural networks

There has been a lot of recent interest in trying to characterize the er...

A Comparison of Neural Network Training Methods for Text Classification

We study the impact of neural networks in text classification. Our focus...

Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation

Backpropagation has been widely used in deep learning approaches, but it...

Please sign up or login with your details

Forgot password? Click here to reset