Neural Network Regularization via Robust Weight Factorization

12/20/2014

∙

Regularization is essential when training large neural networks. As deep neural networks can be mathematically interpreted as universal function approximators, they are effective at memorizing sampling noise in the training data. This results in poor generalization to unseen data. Therefore, it is no surprise that a new regularization technique, Dropout, was partially responsible for the now-ubiquitous winning entry to ImageNet 2012 by the University of Toronto. Currently, Dropout (and related methods such as DropConnect) are the most effective means of regularizing large neural networks. These amount to efficiently visiting a large number of related models at training time, while aggregating them to a single predictor at test time. The proposed FaMe model aims to apply a similar strategy, yet learns a factorization of each weight matrix such that the factors are robust to noise.

READ FULL TEXT

Neural Network Regularization via Robust Weight Factorization

Regularizing Neural Networks by Stochastically Training Layer Ensembles

Regularization of Deep Neural Networks with Spectral Dropout

Dropout as data augmentation

Dropout Inference with Non-Uniform Weight Scaling

Modulating Regularization Frequency for Efficient Compression-Aware Model Training

LocalDrop: A Hybrid Regularization for Deep Neural Networks

Weight Expansion: A New Perspective on Dropout and Generalization

Neural Network Regularization via Robust Weight Factorization

Related Research

Regularizing Neural Networks by Stochastically Training Layer Ensembles

Regularization of Deep Neural Networks with Spectral Dropout

Dropout as data augmentation

Dropout Inference with Non-Uniform Weight Scaling

Modulating Regularization Frequency for Efficient Compression-Aware Model Training

LocalDrop: A Hybrid Regularization for Deep Neural Networks

Weight Expansion: A New Perspective on Dropout and Generalization