1 Introduction
Noise is normally seen as intrinsically undesirable. The word itself bears a very negative connotation. It is not surprising then that many early mathematical models in neuroscience aimed to factor out noise by any means. A few decades ago, the use of stochastic resonance (Wiesenfeld et al., 1995)
in neuroscientific models initiated a new interest in neurosience regarding random fluctuations and the role they play in the brain. Theories about neuronal noise are now flourishing and previous deterministic models are improved by the incorporation of noise
(Yarom & Hounsgaard, 2011).Biological brains have always been a strong inspiration when it comes to developing learning algorithms. Considering the amount of noise which takes place in the brain during learning, one can wonder if this has any beneficial effect. Many techniques in machine learning have made use of noise to improve performance recently, namely, Denoising Autoencoders
(Vincent et al., 2008), dropout (Hinton et al., 2012) and its relative, DropConnect (Wan et al., 2013). Those successful approaches suggest that neuronal noise plays a fundamental role in the process of learning and should be studied more thoroughly.Using dropout can be viewed as training a huge number of neural networks with shared parameters and applying bagging at test time for better generalization
(Baldi & Sadowski, 2013). Binary noise can also be viewed as preventing neurons from coadapting, which improves the generalization of the model even more. In this paper, we propose an alternative view and suggest noise schemes like dropout are implicitly incorporating a form of sophisticated data augmentation. In Section 3, we formulate a method to generate data which replicates dropout noise within a deterministic network, and demonstrate in Section 5 that there is no significant loss of accuracy.Finally, capitalizing on the idea of data augmentation, we present in section 4
an extension of dropout which uses random noise levels to improve the variety of samples. This simple extension improves classification performance across different network architectures, yielding competitive results on the MNIST permutation invariant classification task.
2 Dropout
The main goal when using dropout is to regularize the neural network we are training. The technique consists of dropping neurons randomly with some probability
. Those random modifications of the network’s stucture are believed to avoid coadaptation of neurons by making it impossible for two subsequent neurons to rely solely on each other (Srivastava et al., 2014). The most accepted interpretation of dropout is that is implicitely bagging at test time a large number of neural networks which share parameters.Assume is a linear projection of a dimensional input into a dimensional space:
(1) 
(2)  
(3) 
3 From a data augmentation perspective
In many previous works it has been shown that augmenting data by using domain specific transformations helps in learning better models (LeCun et al., 1998; Simard et al., 2003; Krizhevsky et al., 2012; Ciresan et al., 2012). In this work, we analyze dropout in the context of data augmentation. Considering the task of classification, given a set of training samples, the objective would be to learn a mapping function which maps every input to its corresponding output label. To generalize, the mapping function needs to be able to correctly map not just the training samples but also any other samples drawn from the data distribution. This means that it must not only map input space subregions represented by training samples, but all highprobability subregions of the natural distribution.
One way to learn such a mapping function is by augmenting the training data such that it covers a larger portion of the natural distribution. Domainbased data augmentation helps to artificially boost training data coverage which makes it possible to train a better mapping function. We hypothesize that noise based regularization techniques result in a similar effect of increasing training data coverage at every hidden layer and this work presents multiple experimental observations to support our hypothesis.
3.1 Projecting noise back into the input space
We assume that for a given , there exist an , such that
(4) 
Similarly to adversarial examples from Goodfellow et al. (2014b), an can be found by minimizing the squared error
(5) 
Equation 5 can be generalized to a network with hidden layers. To lighten notation we first define
(6)  
(7) 
We can now compute the back projection corresponding to all hidden layer activations at once, which results in minizing the loss
(8) 
We can show by contradiction that one is unlikely to find a single that significantly reduces . The proof is detailed in appendix subsection 8.1. Fortunately, it is easy to find a different for each hidden layer, by providing multiple inputs , where is the number of hidden layers. As each is the back projection of a transformation in the representation space defined by the th hidden layer, it suggests viewing dropout as a sophisticated data augmentation procedure that samples data around training examples with respect to different level of representations. This raises the question whether we can train the network deterministically on the rather than using dropout. The answer is not trivial, because

When using as inputs, dropout is not effectively applied to every layer at the same time. The local stochasticity preventing coadaptation is then present at a specific layer only once for every . This could be not aggressive enough to avoid coadaptation.

The gradients of the linear projections will differ greatly. In the case of dropout, is always equal to its input transformation, i.e. , whereas the deterministic version of the training will update according to ^{1}^{1}1Because we train on samples from , one for each hidden layer.
Although we proved a single minimising 8 is difficult to find for a large network, we show experimentally in section 5 that it is possible to do so within reasonable approximation for a relatively small two hidden layer network. We further show that dropout can be replicated by projecting the noise back on the input space without a significant loss of accuracy.
4 Improving the set of augmentations
When dealing with domainbased transformations, we intuitively look for the richest set of transformations. In computer vision for instance, translations, rotations, scalings, shearings and elastic transformations are often combined. Looking at dropout from a data augmentation perspective, this intuition raises the following question: given that noise scheme used is implicitely applying some transformations in the input space, which one would produce the richest set of transformations?
With noise schemes like dropout, there are two important components which influence the transformations; The probability distribution of
and the features of the neural network used to encode . Modifying the probability distribution is the most straighforward way to improve the set of transformations and will be the main focus of this paper. However, features of the neural network play a key role in the transformations and we will outline some possible avenues in the conclusion section.4.1 Random noise levels
While using dropout, the proportion of neurons dropped is very close to probability
. It follows naturally from Binomial distribution’s expectation. The transformations induced are as different as the values
can take. Despite this, their magnitude is as constant as the proportion of neurons dropped. That means, every transformations displaces the sample to a relatively constant distance but in random directions in a high dimensional space.A simple way to vary the transformation magnitude randomly is to replace
by a random variable. Let
and where defines the layer, the sample, and the layer’s neuron. It is important to use the same for all neurons of a layer, otherwise we would have .To compensate for the change of level of activations during test, a scaling is normally applied. One could also simply apply the inverse scaling during training, turning equation 3 into
(9) 
To adapt the equation to random dropout level, we simply need to replace with
(10) 
No scaling needs to be done during test anymore.
Figure 1 shows the differences between density function on proportion of neurons dropped for dropout and random dropout. Transformations induced by random dropout are clearly more diverse than those induced by dropout.
5 Experiments
5.1 Visualizations of noise projected back into the input space
Visualizing the noise projected back into the input space helps to understand what kind of transformations are induced by dropout. Unsupervised models learn more general features than supervised fully connected neural networks and produce thus more visually appealing transformations. For this reason, we trained autoencoders with dropout on the hidden layer to generate samples of transformations.
The autoencoder is very similar to denoising autoencoders, the only difference is that a Bernoulli mask is applied to the hidden activations rather than to the input. There is thus no noise applied to the input explicitly. Models are trained for 300 epochs, with minibatch size of 100,
= 0.4, a learning rate of 0.001 on MNIST, 0.0001 on CIFAR10 and a momentum of 0.7 on MNIST, 0.5 on CIFAR10. For CIFAR10, we do preprocessing with PCA dimensionality reduction and retain 781 features.Once the model is trained, we use gradient descent to compute as described in 3.1. We iterate for 10 epochs with a learning rate of 100 for both MNIST and CIFAR10. Figure 5 shows well how close are from the natural input space and we clearly see that the classes are still distinguishable.
in such a way that other features highly dependant on the same subregion still have the same activation level.
To help understand how each feature influences the transformation, we isolate five most active features for a given input and shut them off, each separately. For each feature shut off, we compute using gradient descent. Figure 3 shows the results found for MNIST. One could think the features dropped by the noise are simply removed from the input. It turns out removing the feature would affect the activation of other neurons. Because of this, features are rather destroyed in the input in such a way that other features highly dependant on the same subregion still have the same activation level.
5.2 Equivalence of dropout and noisy samples
We ran a series of experiments with fully connected feed forward neural networks on the MNIST and CIFAR10 datasets to study the effect of replacing dropout by corresponding noisy inputs. Each network consists of two hidden layers with rectified linear units followed by a
softmax layer. We experimented with four different architectures each one with a different number of units in the hidden layers: 25002500, 25001250, 15001500 and 1500750.The MNIST dataset consists of training samples and test samples. We split the training set into a set of samples for training and for validation. Each network is trained for epochs and the best model based on validation error is selected. Later the best model is further trained on the complete training set of samples (training + validation split) for another epochs. The mean error percentage for the last epochs is used for comparison.
We also ran experiments on the CIFAR10 permutation invariant task using the same network architectures described above. The dataset consists of training samples and test samples. We use PCA based dimensionality reduction without whitening as preprocessing scheme, retaining 781 features. We used the same approach as in the MNIST experiments to train the networks and for reporting the performances.
At each epoch, an is generated for each training sample. It proved to be possible to find good approximations for the entire network at once for a 2hidden layer network. Thus, we trained on and solely rather than , and as it gave a significant speed up. For simplicity, the network is trained on for an epoch than on for an epoch. All are generated with parameter values of the model at the beginning of the epoch.
Noisy inputs are found using stochastic gradient descent. 20 learning steps are done with a learning rate of 300.0 for first hidden layer and 30 for second hidden layer. The results for these experiments are shown in figure 4.
Results suggest that Dropout can be replicated by projecting the noise back into the input and training a neural network deterministically on this generated data. There is not significant drop in accuracy, it is even slightly better than Dropout in the case of CIFAR10. This supports the idea that dropout can be seen as data augmentation.
5.3 Richer noise schemes
We ran a series of experiments with fully connected feed forward neural networks on the MNIST and CIFAR10 datasets to compare dropout. Each networks consist of two hidden layers with rectified linear units followed by a softmax layer. We experimented with three different network architectures each one with a different number of units in the hidden layers: 2500625, 25001250 and 25002500. Each network is trained and validated the same way as mentioned in previous section.
First, we evaluated the dropout noise scheme by training the networks with a fixed hidden noise level of and the input noise level varying from to with increments of for each experiment. In the second experiment, we fixed the input noise level at and the hidden noise level is varied from to , again with an increment of . In the final set of experiments we use the random dropout noise scheme using the same noise level at input and hidden layers. The noise level in this case is a range where is varied from to with increment . The classification performances corresponding to the all the experiments on both the datasets are reported in Figure 6.
Random dropout improves the performance of the models over dropout with no additional computational cost.
6 Related Work
To the best of our knowledge there is no work analyzing dropout from a data augmentation perspective. Nonetheless, there is a plethora of excellent works about dropout, some describing his regularisation properties and others developing new kind of noise schemes based on different intuitions.
Regularization properties of noise have been known for more than a decade, Bishop (1995) showed for instance that the regularization term induced by noise belongs to the class of generalized Tikhonov regularizers for sumofsquares error functions. More recently, Baldi & Sadowski (2013) proved that dropout noise scheme specifically applies a regularisation term very similar to usual weight decay.
Not exactly about regularization, but uncertainty, Gal & Ghahramani (2015) gave a very inspiring interpretation of dropout as a Bayesian approximation.
Different noise schemes are used by Poole et al. (2014) and applied at different positions in autoencoders; input, preactivation and postactivations. They report better results than denoising autoencoders and also support that Gaussian noise yields better results than dropout on MNIST classification task.
Bachman et al. (2014) emphasise bagging interpretation of dropout and propose a generalization called pseudoensembles and a related regularizer which makes it possible to train semisupervised networks.
Some recent work reports that noise level schedules, inspired by simulated annealing, help in supervised and unsupervised tasks (Geras & Sutton, 2014; Chandra & Sharma, 2014; Rennie et al., 2014). We propose an alternative that avoids a schedule and rather uses random noise levels such that the model cannot adapt to slowly changing noise distribution.
A similar approach of sampling the noise level was used in Geras & Sutton (2014)
in the context of unsupervised learning using an autoencoder (on input not features). However, they show that the approach is not very useful in their case.
Finally, work by Graham et al. (2015) is related to random noise as both their submatrix multiplications and random noise level are inducing an independence between neurons of a single layer. They found that the independence is not to damageable if they use enough different submatrix patterns.
7 Conclusion
We have presented and justified a novel interpretation of dropout as priorknowledge free data augmentation. We described a new procedure to generate samples by back projecting the dropout noise into the input space. Our results suggest neural networks can be trained without dropout on such noisy samples and still yield good results. Nonetheless, experiments should be performed on larger networks in order to determine whether this observation is just a particular property of relatively small networks. Furthermore, trained networks should be analyzed to determine if coadaptation is still avoided when using perlayer noise backprojection on deep neural networks.
Presenting only random dropout, the list of possible substitute to dropout in this work is far from exhaustive. As described in section 4, important knobs to modify induced data augmentation by noise are model’s features and the noise scheme applied on them. Using semisupervised cost can influence the implicit transformations by forcing the network to learn more general features. A network could also be trained on samples generated from another network, similarly to generative adverserial networks (Goodfellow et al., 2014a).
Acknowledgements
This work was supported by the German Federal Ministry of Education and Research (BMBF) in the project 01GQ0841 (BFNT Frankfurt), an NSERC Discovery grant, a Google faculty research award and Ubisoft. We are grateful towards the developers of Theano
(Bergstra et al., 2010; Bastien et al., 2012), Fuel and Blocks (van Merriënboer et al., 2015).References
 Bachman et al. (2014) Bachman, Phil, Alsharif, Ouais, and Precup, Doina. Learning with pseudoensembles. In Advances in Neural Information Processing Systems, pp. 3365–3373, 2014.
 Baldi & Sadowski (2013) Baldi, Pierre and Sadowski, Peter J. Understanding dropout. In Advances in Neural Information Processing Systems, pp. 2814–2822, 2013.
 Bastien et al. (2012) Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian, Bergeron, Arnaud, Bouchard, Nicolas, WardeFarley, David, and Bengio, Yoshua. Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590, 2012.
 Bergstra et al. (2010) Bergstra, James, Breuleux, Olivier, Bastien, Frédéric, Lamblin, Pascal, Pascanu, Razvan, Desjardins, Guillaume, Turian, Joseph, WardeFarley, David, and Bengio, Yoshua. Theano: a cpu and gpu math expression compiler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4, pp. 3. Austin, TX, 2010.
 Bishop (1995) Bishop, Chris M. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108–116, 1995.
 Chandra & Sharma (2014) Chandra, B and Sharma, Rajesh Kumar. Adaptive noise schedule for denoising autoencoder. In Neural Information Processing, pp. 535–542. Springer, 2014.

Ciresan et al. (2012)
Ciresan, Dan, Meier, Ueli, and Schmidhuber, Jürgen.
Multicolumn deep neural networks for image classification.
In
Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on
, pp. 3642–3649. IEEE, 2012.  Gal & Ghahramani (2015) Gal, Yarin and Ghahramani, Zoubin. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. arXiv preprint arXiv:1506.02142, 2015.
 Geras & Sutton (2014) Geras, Krzysztof J and Sutton, Charles. Scheduled denoising autoencoders. arXiv preprint arXiv:1406.3269, 2014.
 Goodfellow et al. (2014a) Goodfellow, Ian, PougetAbadie, Jean, Mirza, Mehdi, Xu, Bing, WardeFarley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680, 2014a.
 Goodfellow et al. (2014b) Goodfellow, Ian J, Shlens, Jonathon, and Szegedy, Christian. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
 Graham et al. (2015) Graham, Ben, Reizenstein, Jeremy, and Robinson, Leigh. Efficient batchwise dropout training using submatrices. arXiv preprint arXiv:1502.02478, 2015.
 Hinton et al. (2012) Hinton, Geoffrey E., Srivastava, Nitish, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Improving neural networks by preventing coadaptation of feature detectors. CoRR, abs/1207.0580, 2012.
 Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
 LeCun et al. (1998) LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner, Patrick. Gradientbased learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
 Poole et al. (2014) Poole, Ben, SohlDickstein, Jascha, and Ganguli, Surya. Analyzing noise in autoencoders and deep networks. arXiv preprint arXiv:1406.1831, 2014.
 Rennie et al. (2014) Rennie, Steven, Goel, Vaibhava, and Thomas, Samuel. Annealed dropout training of deep networks. In Spoken Language Technology (SLT), IEEE Workshop on. IEEE, 2014.
 Simard et al. (2003) Simard, Patrice Y, Steinkraus, Dave, and Platt, John C. Best practices for convolutional neural networks applied to visual document analysis. In 2013 12th International Conference on Document Analysis and Recognition, volume 2, pp. 958–958. IEEE Computer Society, 2003.
 Srivastava et al. (2014) Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014.
 van Merriënboer et al. (2015) van Merriënboer, Bart, Bahdanau, Dzmitry, Dumoulin, Vincent, Serdyuk, Dmitriy, WardeFarley, David, Chorowski, Jan, and Bengio, Yoshua. Blocks and fuel: Frameworks for deep learning. arXiv preprint arXiv:1506.00619, 2015.
 Vincent et al. (2008) Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, PierreAntoine. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096–1103. ACM, 2008.
 Wan et al. (2013) Wan, Li, Zeiler, Matthew, Zhang, Sixin, Cun, Yann L, and Fergus, Rob. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML13), pp. 1058–1066, 2013.
 Wiesenfeld et al. (1995) Wiesenfeld, Kurt, Moss, Frank, et al. Stochastic resonance and the benefits of noise: from ice ages to crayfish and squids. Nature, 373(6509):33–36, 1995.
 Yarom & Hounsgaard (2011) Yarom, Yosef and Hounsgaard, Jorn. Voltage fluctuations in neurons: signal or noise? Physiological reviews, 91(3):917–929, 2011.
8 Appendix
8.1 Proof of unlikeliness
We can show, with a proof by contradiction, that it’s unlikely to find a single that minimizes well .
By the associative property of function composition, we can rewrite equation 7
(11) 
Suppose there exist an such that
(12)  
(13) 
Based on 11 and 13, we have that . The proof is concluded by replacing the latter in 12 and then expanding the composed functions.
(14) 
Equation 14 can only be true if does not apply any modification to , that means when . It happens with a probability where is the Bernoulli success probability, is the number of hidden units and is the mean sparsity level, i.e. mean percentage of active hidden units, of the th hidden layer. This probability is very low for standard hyperparameters values. For instance, with , and , the probability is as low as .