Regularizing Neural Networks by Stochastically Training Layer Ensembles

by   Alex Labach, et al.

Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic regularization while explicitly averaging outputs. This provides stronger regularization with no additional computational cost at test time. We show consistent improvement on various image classification tasks using standard network topologies.


page 1

page 2

page 3

page 4


Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation

Ensembles of deep neural networks have demonstrated superior performance...

Neural Network Regularization via Robust Weight Factorization

Regularization is essential when training large neural networks. As deep...

Swapout: Learning an ensemble of deep architectures

We describe Swapout, a new stochastic training method, that outperforms ...

Learning Neural Network Subspaces

Recent observations have advanced our understanding of the neural networ...

An empirical analysis of dropout in piecewise linear networks

The recently introduced dropout training criterion for neural networks h...

Distributed Estimation via Network Regularization

We propose a new method for distributed estimation of a linear model by ...

Diverse Weight Averaging for Out-of-Distribution Generalization

Standard neural networks struggle to generalize under distribution shift...

1 Introduction

In order to generalize well to new inputs, modern deep neural networks require heavy regularization. While many techniques for achieving this have been proposed, dropout [1] and related methods have become some of the most widely used approaches [2]

. Dropout works by randomly removing neurons from a neural network with a certain probability at each training step, then using the full network at test time. It is commonly interpreted as implicitly averaging over an ensemble of networks, where the ensemble contains all possible networks obtained by removing a subset of neurons 

[1, 3, 4, 5]. The regularization power of dropout then comes from this averaging, where individual elements of the network ensemble may be overfitted, but averaging them reduces this effect.

We introduce STE (stochastically trained ensemble) layers, which add explicit averaging to dropout and related regularization methods 111Sample code can be found at

. In place of a standard dense neural network layer, we propose using multiple different weight matrices and bias vectors to transform inputs. Dropout or a similar method is applied to all resulting values, and corresponding outputs are then averaged together before an activation function is applied, optionally followed by dropout again. Our results suggest that training multiple sets of weights and biases together in this fashion along with dropout provides stronger regularization than dropout alone.

Our proposed method is related to various established ensemble methods in machine learning. This includes bagging, boosting, and mixture of expert methods, where multiple instances of a machine learning model are each trained on a subset of the input space or training set and collaborate during inference 

[6, 7]. While our method also stochastically trains multiple weight matrices on different subsets of the training set, after training, averaging is applied to produce a single weight matrix for use in inference. It therefore functions as a regularization method for a single instance of a model rather than a technique for using multiple instances of a model during inference.

A recent related method is fraternal dropout [8]

, which regularizes recurrent neural networks by applying two sets of dropout masks, then using a loss function that promotes similarity in the resulting outputs and invariance to dropout masks. While our proposed method has similar goals, we insert averaging directly into dense layers and use a standard loss function, achieving regularization through ensemble averaging rather than explicitly promoting invariance to dropout masks. Our method also operates on dense layers rather than recurrent neural networks. Finally, our method varies from both model averaging methods and fraternal dropout in that it applies to individual layers rather than entire networks and in that we directly average weights and biases at test time rather than activations or outputs.

2 Proposed Method





Figure 1: Diagram of an STE layer using dropout for noise. Inputs are transformed using multiple independent weight matrices and bias vectors, then noise is added using dropout, corresponding outputs are averaged together, and finally the activation function is applied. Note that dropout can additionally be used on the layer output. At test time, weight matrices and biases are averaged to produce a single weight matrix and bias vector.

To introduce our notation, a standard dense neural network layer with inputs and outputs implements the following function:


where is the layer input, is a weight matrix, is a bias vector, is an activation function applied elementwise, and is the layer output. The weights and biases are learned parameters.

STE layers act as a replacement for dense layers by transforming using different weight matrices and bias vectors, applying some noise, and then averaging the results to produce a vector of length before applying the activation function. We refer to as the averaging factor of the layer. Mathematically, an STE layer with inputs and outputs is of the form:


where represents an operation that implements the transformation , but with some type of noise added during training. The vectors produced by each application of are then averaged together. All weight matrices and bias vectors are initialized independently.

At test time, the noise operation must become or some affine transformation of this. This allows the following property to hold.

Theorem 1.

At test time, an STE layer with inputs and outputs operates as a dense layer with the same number of inputs and outputs.


At test time, produces for some and . The layer output is therefore given by:


which is equivalent to a dense layer with the weight matrix and the bias vector
. ∎

This means that an STE layer acts as a regularization method, incurring no additional computational cost at test time compared to a dense layer of the same dimensions, regardless of the value of . In contrast, model averaging methods require running inference on multiple model instances, which is computationally costly. It is important to note that the injection of different noise values to each member of the ensemble during training is exactly what makes it impossible to combine weight matrices in a similar fashion during training, ensuring diversity across the ensemble.

We mainly investigate the use of dropout [1] to provide noise, leading to the following equation during training:


where each is an independent length- vector of i.i.d. samples from Bernoulli() and represents element-wise multiplication. This method is illustrated in Figure 1. At test time the element-wise multiplication by is replaced with multiplication by . This is equivalent to a dense layer with the weight matrix and the bias vector .

We also explore applying a dropout mask directly to weights, which leads to the following equation:


where each is an independent matrix of i.i.d. samples from Bernoulli(). This is a generalization of dropconnect [9], which represents the special case . We differ from [9] and use the same approach as with dropout at test time, replacing multiplication by mask elements with multiplication by .

width= Dataset No regularization Standard dropout STE layers using dropout STE layers using dropconnect Loss Accuracy Loss Accuracy Loss Accuracy Loss Accuracy CIFAR-10 1.3510.009 53.20.06 1.2660.01 56.40.3 1.2360.003 57.00.2 1.2370.006 56.90.4 CIFAR-100 3.2240.01 25.30.6 2.9710.009 29.80.3 2.8360.004 32.00.2 2.8450.003 32.20.1 SVHN 0.7590.009 81.50.07 0.5680.005 85.70.1 0.4890.002 86.30.09 0.5070.001 85.80.1

Table 1:

Comparison of STE layers to standard dense layers in an MLP with and without dropout. Results are averaged over five tests with sample standard deviations provided. Lower loss and higher accuracy are better.

width= Dataset No regularization Standard dropout STE layers using dropout STE layers using dropconnect Loss Accuracy Loss Accuracy Loss Accuracy Loss Accuracy CIFAR-10 1.2340.02 57.10.8 1.1660.02 60.70.5 1.0300.007 64.60.4 1.0370.02 64.00.01 CIFAR-100 3.1650.04 24.90.8 3.0380.04 26.50.5 2.9370.03 27.20.3 3.0170.01 26.00.6 SVHN 0.5050.01 85.30.4 0.4790.003 86.60.2 0.4680.004 86.60.2 0.4720.004 86.30.1

Table 2: Comparison of STE layers to standard dense layers in LeNet-5 with and without dropout. Results are averaged over five tests with sample standard deviations provided. Lower loss and higher accuracy are better.

Whereas dropout and related methods create pseudo-ensembles of models [5], STE layers additionally use an explicit ensemble of separate weights and biases. different initial sets of weights are sampled, and the use of independent noise for each causes each to follow a different path towards some local minimum. As with bagging methods [6]

, we understand the role of averaging outputs as reducing the variance caused by neural network learning arriving at suboptimal minima. A major difference from bagging is that the weights in STE layers can be directly averaged at test time, as opposed to running multiple models and averaging their outputs. This is possible because STE layers train weight ensembles together in a single layer, causing them to learn in such a way that groups of activations within a neural network correspond to each other and can be averaged. We empirically analyze this effect in the following section and show that the resulting weight matrices remain diverse.

STE layers can also be seen as promoting a network representation that is more robust to different weight initializations and noise than dense layers. By incorporating multiple weight initialization samples and independent applications of noise, the sensitivity to any particular realization is reduced. This has a similar effect to regularization approaches that explicitly promote invariance with respect to dropout masks [8].

3 Experiments

We evaluate using STE layers instead of dense hidden layers in a multi-layer perceptron (MLP) as well as replacing fully connected layers in two standard convolutional neural network topologies: LeNet-5 

[10] and AlexNet [11]. These networks are tested on three image classification datasets: CIFAR-10 [12], CIFAR-100 [12], and SVHN [13]. Our result metrics are cross-entropy loss and top-1 accuracy in percent. We normalize all datasets to have zero mean and unit standard deviation.

We primarily compare STE layers to dense layers that use standard dropout regularization with . We test STE layers using either dropout or dropconnect internally, both with , but additionally apply dropout to the layer outputs, again with . For comparison, we also report results obtained without any regularization methods applied to the dense layers. We use for the MLP and LeNet-5 tests, but for AlexNet tests, since higher values were found to degrade training. Both dense and STE layer weights are initialized using Glorot uniform initialization [14]

and biases are initialized to zero. We always use a dense layer as the output layer, not an STE layer. The MLP used for testing has two hidden layers with 2 048 neurons each and ReLU activation functions. Since AlexNet is designed for larger images than those in the datasets used, when training and testing it, we upsample images by a factor of seven using bilinear interpolation.

We train using SGD optimization with Nesterov momentum of 0.9, a minibatch size of 128, and learning rate decay

after each minibatch such that the learning rate at iteration is reduced by a factor . To avoid bias, learning rates and decay rates are tuned to produce optimal results with standard dropout regularization rather than our proposed method. For the MLP, we use an initial learning rate of with a decay rate of

and train for 128 epochs. For LeNet-5, we use an initial learning rate of

with a decay rate of and train for 256 epochs. For AlexNet, we use an initial learning rate of with a decay rate of and train for 150 epochs. Ten percent of data is randomly held back for validation. For testing, we restore the weights from the epoch with lowest validation loss.

width= Dataset No regularization Standard dropout STE layers using dropout STE layers using dropconnect Loss Accuracy Loss Accuracy Loss Accuracy Loss Accuracy CIFAR-10 0.8430.02 72.40.7 0.6040.006 80.90.3 0.5770.01 81.10.5 0.6000.01 81.00.2 CIFAR-100 2.7090.03 35.50.4 1.9210.02 50.60.5 1.7300.02 53.70.4 1.8580.01 50.50.7 SVHN 0.3910.02 90.30.3 0.2650.01 93.50.2 0.2310.003 94.00.05 0.2370.005 93.90.2

Table 3: Comparison of STE layers to standard dense layers in AlexNet with and without dropout. Results are averaged over five tests with sample standard deviations provided. Lower loss and higher accuracy are better.

3.1 Results and Analysis

width=0.5 Model MLP LeNet-5 AlexNet Dense STE Dense STE Dense STE CIFAR-10 10.5M 83.9M 1.7M 13.9M 45.5M 88.5M CIFAR-100 10.7M 84.1M 1.7M 13.9M 45.9M 88.9M SVHN 10.5M 83.9M 1.7M 13.9M 45.5M 88.5M

Table 4: The number of parameters trained for the experiments described in this paper. Note that at test time, STE layer parameters are averaged so that there is no additional cost compared to standard dense layers.
Figure 2: Two sample activation vectors within an STE layer before averaging, using a trained LeNet-5 model with two test images from CIFAR-10 as inputs. Columns correspond to values that will be averaged together.

Results on the test partition of each dataset are shown in Tables 3, 2 and 1. In all cases, lower test loss is achieved using STE layers compared to standard dense layers with dropout regularization. Top-1 accuracy remains the same in some cases, but for the most part, it improves when using STE layers. This improvement can be dramatic, as seen with the 3.9% increase with LeNet-5 on CIFAR-10. In general, STE layers are shown to improve performance, which we interpret as them providing more effective regularization, allowing the neural networks to generalize more effectively on the given, relatively small, datasets.

For the most part, using dropconnect as the noise method within STE layers shows worse performance than using dropout. However, STE layers with dropconnect still outperform dense layers with dropout. It may be possible to improve its relative performance by tuning

, which we did not investigate in the interest of minimizing tuned hyperparameters in our proposed method. Other noise methods could also be investigated, such as multiplicative Gaussian noise 

[15]. However, given that dropout is generally thought to be versatile and effective for stochastically training neural networks, it is unclear whether this could provide significant benefits.

Using STE layers allows for much larger networks to be trained with no additional cost at test time. Table 4 shows the number of parameters trained in our experiments when using dense layers versus STE layers. The averaging causes the weights of neurons that are averaged together to become correlated, simplifying the learning task and allowing for weights to be directly averaged together. However, the use of dropout or other noise maintains diversity within each ensemble. This diversity allows weight averaging at test time to produce a regularized network. The correlation between activations is illustrated in fig. 2, which shows activations within a trained STE layer before averaging. Activations that are averaged together are clearly correlated, but far from identical, demonstrating that the weight matrices within the ensemble have arrived at different minima within the optimization space.

Note that for the sake of comparison, only the averaging factor was tuned for STE layer-specific tests. Further tuning of optimizer parameters, noise parameters, adding additional regularization, etc. could improve results in practice.

4 Conclusions

We find that STE layers can effectively use ensemble averaging to regularize neural networks, with no additional cost at test time. We show consistent improvements compared to dropout regularization using standard neural network topologies and standard image classification datasets. While model averaging methods are common in machine learning, our proposed method differs from most established methods in that it allows weights and biases to be directly averaged at test time, thereby functioning as a regularization method. STE layers are based on dense neural network layers, but there is clear potential for similar methods to work in neural networks with different kinds of layers, including convolutional neural networks, recurrent neural networks, residual neural networks, and neural networks with attention mechanisms.


  • [1] Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580, 2012.
  • [2] Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee, “Survey of dropout methods for deep neural networks,” arXiv preprint arXiv:1904.13310, 2019.
  • [3] David Warde-Farley, Ian J. Goodfellow, Aaron Courville, and Yoshua Bengio, “An empirical analysis of dropout in piecewise linear networks,” in Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  • [4] Pierre Baldi and Peter J Sadowski, “Understanding dropout,” in Advances in Neural Information Processing Systems 26, pp. 2814–2822. Curran Associates, Inc., 2013.
  • [5] Philip Bachman, Ouais Alsharif, and Doina Precup, “Learning with pseudo-ensembles,” in Advances in Neural Information Processing Systems 27, pp. 3365–3373. Curran Associates, Inc., 2014.
  • [6] Thomas G. Dietterich, “Ensemble methods in machine learning,” in

    Multiple Classifier Systems

    , Berlin, Heidelberg, 2000, pp. 1–15, Springer Berlin Heidelberg.
  • [7] Saeed Masoudnia and Reza Ebrahimpour, “Mixture of experts: a literature survey,” Artificial Intelligence Review, vol. 42, no. 2, pp. 275–293, Aug 2014.
  • [8] Konrad Żołna, Devansh Arpit, Dendi Suhubdy, and Yoshua Bengio, “Fraternal dropout,” arXiv preprint arXiv:1711.00066, 2018.
  • [9] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus, “Regularization of neural networks using dropconnect,” in International conference on machine learning, 2013, pp. 1058–1066.
  • [10] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner, “Gradient based learning applied to document recognition,” in Proceedings of the IEEE, 1998, pp. 2278–2324.
  • [11] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,

    Imagenet classification with deep convolutional neural networks,”

    in Advances in neural information processing systems, 2012, pp. 1097–1105.
  • [12] Alex Krizhevsky, “Learning multiple layers of features from tiny images,” University of Toronto, 05 2012.
  • [13] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in

    NIPS Workshop on Deep Learning and Unsupervised Feature Learning

    , 2011.
  • [14] Xavier Glorot and Yoshua Bengio, “Understanding the difficulty of training deep feedforward neural networks.,” in Proceedings of the International Conference on Artificial Intelligence and Statistics. AISTATS, 2010, vol. 9, pp. 249–256.
  • [15] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.,” Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.