On Convergence and Generalization of Dropout Training

10/23/2020

∙

We study dropout in two-layer neural networks with rectified linear unit (ReLU) activations. Under mild overparametrization and assuming that the limiting kernel can separate the data distribution with a positive margin, we show that dropout training with logistic loss achieves ϵ-suboptimality in test error in O(1/ϵ) iterations.

READ FULL TEXT

On Convergence and Generalization of Dropout Training

Multi-Sample Dropout for Accelerated Training and Better Generalization

Improved Dropout for Shallow and Deep Learning

On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons

Altitude Training: Strong Bounds for Single-Layer Dropout

Improving Neural Network Generalization by Combining Parallel Circuits with Dropout

Triangular Dropout: Variable Network Width without Retraining

If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks

On Convergence and Generalization of Dropout Training

Related Research

Multi-Sample Dropout for Accelerated Training and Better Generalization

Improved Dropout for Shallow and Deep Learning

On the Convergence of Shallow Neural Network Training with Randomly Masked Neurons

Altitude Training: Strong Bounds for Single-Layer Dropout

Improving Neural Network Generalization by Combining Parallel Circuits with Dropout

Triangular Dropout: Variable Network Width without Retraining

If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks