1 Introduction and Motivation
Ioffe & Szegedy (2015) identified an important problem involved in training deep networks, viz.
, Internal Covariate Shift. It refers to the problem of shifting distribution of the input of every hidden layer in a deep neural network. This idea is borrowed from the concept of covariate shift
(Shimodaira, 2000), where this problem is faced by a single inputoutput learning system. Consider the last layer of a deep network being used for classification; this layer essentially tries to learn , whereis the class label random variable (r.v.) and
is the layer input r.v. However, learning a fixed becomes a problem if changes continuously. As a result, this slows down training convergence.Batch Normalization (BN) addresses this problem by normalizing the distribution of every hidden layer’s input. In order to do so, it calculates the preactivation mean and standard deviation using minibatch statistics at each iteration of training and uses these estimates to normalize the input to the next layer. While this approach leads to a significant performance jump by addressing internal covariate shift, its estimates of mean and standarddeviation of hidden layer input for validation rely on minibatch statistics, which are not representative of the entire data distribution (especially during initial training iterations). This is because the minibatch statistics of input to hidden layers depends on the output from previous layers, which in turn depend on the previous layer parameters that keep shifting during training, and a moving average of these estimates are used for validation. Finally, due to involvement of batch statistics, BN is inapplicable with batchsize .
In this paper, we propose a simple parametric normalization technique for addressing internal covariate shift that does not depend on batch statistics for normalizing the input to hidden layers and is less severely affected by the problem of shifting parameters during validation. In fact, we show that it is unnecessary to explicitly calculate mean and standarddeviation from minibatches for normalizing the input to hidden layers even for training. Instead, a data independent estimate of these normalization components are available in closed form for every hidden layer, assuming the preactivation values follow Gaussian distribution and that the weight matrix of hidden layers are roughly incoherent. We show how to forward propagate the normalization property (of the data distribution) to all hidden layers by exploiting the knowledge of the distribution of the preactivation values (Gaussian) and some algebraic manipulations. Hence we call our approach Normalization Propagation.
2 Background
It has long been known in Deep Learning community that input whitening and decorrelation helps in speeding up the training process. In fact, it is explicitly mentioned in
(LeCun et al., 2012) that this whitening should be performed before every layer so that the input to the next layer has zero mean. From the perspective of Internal Covariate Shift, what is required for the network to learn a hypothesis at any given layer at every point in time during training, is for the distributionof the input to that layer to be fixed. While whitening could be used for achieving this task at every layer, it would be a very expensive choice (cubic order of input size) since whitening dictates computing the Singular Value Decomposition (SVD) of the input data matrix. However,
Desjardins et al. (2015) suggest to overcome this problem by approximating this SVD by: a) using subsampled training data to compute this SVD; b) computing it every few number of iteration and relying on the assumption that this SVD approximately holds for the iterations in between. In addition, each hidden layer’s input is then whitened by reparametrizing a subset of network parameters that are involved in gradient descent. As mentioned in Ioffe & Szegedy (2015), this reparametrizing may lead to effectively cancelling/attenuating the effect of the gradient update step since these two operations are done independently.Batch Normalization (Ioffe & Szegedy, 2015) addresses both the above problems. First, they propose a strategy for normalizing the data at hidden layers such that the gradient update step accounts for this normalization. Secondly, this normalization is performed for units of each hidden layer independently (thus avoiding whitening) using minibatch statistics. Specifically, this is achieved by normalizing the preactivation of all hidden layers as,
(1) 
where denotes the element of
and the expectation/variance is calculated over the training minibatch
. Notice since is a part of this normalization, it becomes a part of the gradient descent step as well. However, a problem common to both the above approaches is that of shifting network parameters upon which their approximation of input normalization for hidden layers depends.3 Normalization Propagation (NormProp) Derivation
We will now describe the idea behind NormProp. At a glance the problem at hand seems cyclic because estimating the mean and standard deviation of the input distribution to any hidden layer requires the input distribution of its previous layer (and hence its parameters) to be fixed to the optimal value before hand. However, as we will now show, we can sidestep this naive approach and get an approximation of the true unbiased estimate using the knowledge that the preactivation to every hidden layer follows a Gaussian distribution and some algebraic manipulation over the properties of the weight matrix. For the derivation below, we will focus on networks with ReLU activation, and later discuss how to extend our algorithm to other activation functions.
3.1 Data Normalization
Real world data generally follows Gaussian like distribution. Therefore, consider a data distribution in such that all the samples are normalized, i.e.,
(2) 
Then our goal is to find a way to propagate this normalization to all the hidden layers without the need of explicit data dependent normalization. Depending on whether this input is passed through a convolutional layer or a fully connected layer, a part of the input or the entire input gets multiplied to a weight matrix. Irrespective of the case, lets use
to denote this input for ease of notation; which can thus be the entire data vector or a subset of its element depending on the case. The preactivation is given by
, where and is the number of filters (we will ignore bias for now). As also claimed by Ioffe & Szegedy (2015); Hyvärinen & Oja (2000), we assume the preactivation () has a Gaussian form. So if we ensure that the preactivation of the first hidden layer is a normalized Gaussian, then the hidden layer’s output () will be a Rectified Gaussian distribution. As mentioned in section 2, we can choose to directly normalize the postactivation output . However, as we will now show, it is easier to find closed form estimates for normalizing instead.3.2 Mean and Standarddeviation Normalization for First Hidden Layer
Clearly, since the input data is mean subtracted, the preactivation to the first hidden layer also has zero mean from linearity,i.e., . Now we want to ensure the variance of is . Let the covariance matrix of be denoted by . Then the following proposition bounds how far is from a canonical distribution.
Proposition 1.
(Canonical Error Bound) Let where and such that and (
is the identity matrix) . Then the covariance matrix of
is approximately canonical satisfying,(3) 
where is the covariance matrix of , is the coherence^{1}^{1}1Coherence is defined as of the rows of , is the closest approximation of the covariance matrix to a canonical ellipsoid and diagonalizes a vector to a diagonal matrix. The corresponding optimal .
The above proposition tells us two things. First, the covariance matrix of the preactivation is approximately canonical (diagonal covariance matrix) if the above error bound can be made tight, and that this tightness can be controlled by certain properties of the weight matrix . Second, if we want to normalize each element of the vector to have unit standarddeviation, then our best bet is to divide each by the corresponding weight length if we ensure tight canonical error bound. This is because the closest estimate of a diagonal variance for is ( in our case).
For any dictionary (weight matrix ), the bound above can be made tighter my minimizing coherence . In our approach, we also need to normalize each element of the vector to have unit standarddeviation which is achieved by dividing each by . Notice this automatically makes each hidden weight vector to effectively have unit length. As a result, the error bound only depends on the coherence of . On the other hand, it is generally observed that useful filters that constitute a good representation of real world data are roughly incoherent (Wright et al., 2010; Makhzani & Frey, 2013); thus ensuring the R.H.S is small thereby minimizing the error bound.
At this point, we have normalized the preactivation to have zero mean and unit variance (divide each preactivation element by corresponding ). As a result, the output of the first hidden layer () is Rectified Gaussian distribution. Notice that the above bound ensures the dimensions of and hence () are roughly uncorrelated. Thus, if we subtract the distribution mean from and divide by its standard deviation, we will have reduced the dynamics of the second layer to be identical to that of the first layer. The mean and standard deviation of the aforementioned Rectified Gaussian is,
Remark 1.
(PostReLU distribution) Let and . Then and
Hence in order to normalize the postactivation to have zero mean and unit standard, the above calculated values can be used. Finally, in the case of Pooling (in ConvNets), we essentially take a block of postactivated units and take average or maximum of these values. If we consider each such unit to be independent then the distribution after pooling, will have a different mean and standard deviation. However, in reality, each of these units are highly correlated since they involve computation over either overlapping or spatially close patches. Therefore, we found that the distribution statistics do not get affected significantly and hence we do not recompute mean and standard deviation postpooling.
3.3 Propagation to Higher Layers
With the above two operations, the dynamics of the second hidden layer become identical to that of the first hidden layer. By induction, repeating these two operations for every layer, viz.–1) divide every hidden layer’s preReLUactivation by its corresponding , where is the corresponding layer’s weight matrix, 2) subtract and divide and (respectively) from every hidden layer’s postReLUactivation– we ensure that the input to every layer is a canonical distribution. While training, all these normalization operations are backpropagated.
3.4 Effect of NormProp on Jacobian of Hidden Layers
It has been discussed in Saxe et al. (2013); Ioffe & Szegedy (2015) that Jacobian of hidden layers with singular values close to one improves training convergence in deep networks. While BN has been shown to intuitively achieve this condition, we will now show more rigorously that NormProp (approximately) indeed achieves this condition.
Let be a vector such that the element of is given by . The output of a hidden unit using NormProp is given by where , , and . Let be such that the row of equals . Thus the output can be rewritten as . Let denote the Jacobian of this output with respect to the previous layer input . Then the row of is given by
(4) 
where denotes the row of . Let be an indicator vector such that the element of is given by
(5) 
where is the indicator operator. Let be a matrix such that every column of is occupied by the vector . Then the entire Jacobian matrix can be written as . In order to analyze the singular values of , we want to calculate . From proposition 1, the covariance matrix of the preactivation is given by , where . Since the length of each is , , and (therefore) is approximately an identity matrix if the rows of are incoherent. Thus,
(6) 
Finally taking an expectation of over the distribution of which is Normal, we get,
(7) 
where denotes the density of
– Normal distribution. From the definition of
, it is straight forward to see that the result of the integral is a matrix with its diagonal equal to a vector of , hence, , where is the identity matrix. Thus the singular values of the Jacobian are which, being close to , approximately achieves dynamical isometry (Saxe et al., 2013) and should thus prevent the problem of exploding or diminishing gradients while training deep networks suggesting faster convergence. In the next section, we will use this value during the practical implementation of NormProp for improving the Jacobian to be approximately .4 NormProp: Implementation Details
We have all the ingredients required to filter out the steps for Normalization Propagation for training any deep neural network with ReLU activation though out its hidden layers. Like BN, NormProp can be used alongside any optimization algorithm (eg. Stochastic Gradient Descent with/without momentum) for training deep networks.
4.1 Normalize Data
Since the core idea of NormProp is to propagate the data normalization through hidden layers, we offer two alternative choices either one of which can be used for normalizing the input to a NormProp network. As we will describe, both options are justified in their respective scenario.
1. Global Data Normalization: In cases when the entire dataset – approximately representing the true data distribution – is available at hand, we compute the global mean and standard deviation for each feature element. Then the first step for NormProp is to subtract elementwise mean calculated over the entire dataset from each sample. Similarly divide each feature element by the elementwise standarddeviation. Ideally it is required by NormProp that all input dimensions be statistically uncorrelated, a property achieved by whitening for instance, but we suggest elementwise normalization as an approximation since it is computationally cheaper. Notice this precludes the dilemma of what range the input should be scaled to before passing through the network.
2. Batch Data Normalization: In many real world scenario, streaming data is available and thus it is not possible to compute an unbiased estimate of global mean and standard deviation at any given point in time. In such cases, we propose to instead batchnormalize every minibatch training data fed to the network. Again, we perform the normalization of each feature element independently for computational purposes. Notice this normalization is only performed at the data level, all hidden layers are still normalized by the NormProp strategy which is not affected by shifting model parameters as compared to BN. Moreover, Batch Data Normalization also serves as a regularization since each data sample gets a different representation each time depending on the minibatch it comes with. Thus by using the Batch Data Normalization strategy we actually benefit from the regularization aspect of BN but also overcome its drawbacks by computing the hidden layer mean and standarddeviation without depending on batch statistics. Notice this strategy is most effective when the incoming data is well shuffled.
4.2 Initialize Network Parameters
We use Normalized Initialization (Glorot & Bengio, 2010)
for setting the initial values of all the weight matrices, both fully connected and convolutional. Bias vectors are initialized to zeros and scaling vectors (described in the next subsection) can either be initialized to
or as described in the next subsection.4.3 Propagate Normalization
Similar to BN, we also make use of gradientbasedlearnable scaling and bias parameters and during implementation. We will now describe our normalization in detail for both fully connected and convolutional layers.
4.3.1 Fully Connected Layers
Consider any fully connected layer characterized by a weight matrix , bias , scaling , input and activation ReLU. Here denotes the number of filters and denotes the input dimension. Then without NormProp, the output unit of the hidden layer would traditionally be:
(8) 
Now in the case of NormProp, the output becomes,
(9) 
Here we initialize each to in order to make the Jacobian close to one as suggested by our analysis in section 3.4 for ReLU activation. Thus we call this number the Jacobian factor. We found this initializing using Jacobian factor helps training with larger learning rates without diverging. However, one can also choose to treat the initialization value as a hyperparameter.
4.3.2 Convolutional Layers
Consider any convolutional layer characterized by a filter matrix , bias , scaling , input
and activation ReLU along with any arbitrary choice of stridesize. Here,
denotes the number of filters, –depth, –height, – width for input/filters and L,B– height and width of image. Then without NormProp, the output feature map of the hidden layer using the filter would traditionally be:(10) 
where denotes the convolution operation. Now in the case of NormProp, the output feature map becomes,
(11) 
where each element of is again initialized to . Notice each is multiplied to all outputs from the same corresponding filter and similarly all the scalars as well as the bias vector are broadcasted to all the dimensions. Pooling is done after this normalization process the same way as done traditionally.
4.4 Training
The network is trained using Back Propagation. While doing so, all the normalizations also get backpropagated at every layer.
Optimization: We use Stochastic Gradient Descent with momentum (set to ) for training. Data shuffling also leads to performance improvement (this however, is true in general while training deep networks).
Learning Rate: We found learning speeds up by reducing the learning rate by half whenever the training error starts saturating. Also, we found larger initial learning rate for larger batch size improves performance.
Constraints: After every training iteration, we scale each hidden weightvector/filtermap to have unit length, i.e., we use constraint on all hidden weights, both convolutional and fully connected. This is done because the scale of weight vectors do not affect network representation, so constraining the weights should reduce the parameter search space.
Regularizations:
We use weight decay along with the loss function; we found a small coefficient value of
is necessary during training. We found Dropout does not help during training; we believe this might be because Dropout changes the distribution of output of the layer it is applied, which affects NormProp.4.5 Validation and Testing
Validation and test procedures are identical for NormProp. While validation/testing, each sample is first normalized using mean and standard deviation which are calculated depending on how the train data is normalized during training. In case we use Global Data Normalization during training, we simply use the same global estimate of mean and standard deviation to normalize each test/validation sample. On the other hand, if Batch Data Normalization is used during training, a running estimate of mean and standard deviation is maintained during training which is then used to normalize every test/validation sample. Finally, the input is forward propagated though the network with learned parameters using the same strategy described in section 4.3.
4.6 Extension to other Activation Functions
Even though our paper shows how to overcome the problem of Internal Covariate Shift specifically for networks with ReLU activation throughout, we have in essence proposed a general framework for propagating normalization done at data level to all hidden layers. All that is needed for extending NormProp to other activation functions is to compute the distribution mean () and standard deviation () of output after the activation function of choice, similar to what is shown in remark 1. Thus the general form of output for any given activation becomes^{2}^{2}2Using the appropriate Jacobian Factor allows the use of larger learning rate; however, NormProp works without it as well. (shown for convolution layer as an example),
(12) 
This activation can be both parameter based or fixed. For instance, a parameter based activation is Parametric ReLU (PReLU, He et al. (2015)) (with parameter ) given by,
(13) 
Then the post PReLU distribution statistics is given by,
Remark 2.
Let and . Then and
Notice the distribution mean and standard deviation depends on the parameter and thus will be involved in the normalization process. In case of nonparameter based activations (eg. Tanh, Sigmoid), one can either choose to analytically compute the statistics (like we did for ReLU) or compute these values empirically by simulation since the input distribution to the activation is a fixed Normal distribution. Thus NormProp is a general framework which can be extended to any activation function of choice.
5 Empirical Results and Observations
We want to verify the following: a) performance comparison of NormProp when using Global Data Normalization vs. Batch Data Normalization; b) NormProp alleviates the problem of Internal Covariate Shift more accurately compared to BN; c) thus, convergence stability of NormProp is better than BN; d) effect of batchsize on the behaviour of NormProp, especially batchsize (BN not applicable). Finally we report classification result on various datasets using NormProp and BN.
Datasets: We use the following datasets,
1) CIFAR (Krizhevsky, 2009)– It consists of real world color images in 10 classes split into train and test images. We use images from train set for validation and remaining for training.
2) CIFAR– It has the same number of train and test samples as CIFAR but it has
classes. For training, we use hyperparameters same as those for CIFAR
.3) SVHN (Netzer et al., 2011)– It consists of color images of house numbers collected by Google Street View. It has train images, test images and an additional train images. Similar to the protocol in (Goodfellow et al., 2013), we select 400 samples per class from the train set and 200 samples per class from the extra set as validation and use the remaining images of the train and extra sets for training.
Experimental Protocols (For experiments in sections 5.1 through 5.4): We use CIFAR with the following Network in Network (Lin et al., 2014) architecture^{3}^{3}3
We use the following shorthand for a) conv layer: C(number of filters, filter size, stride size, padding); b) pooling: P(kernel size, stride, padding, pool mode)
. For any specified initial learning rate, we reduce it by half every epochs. We use Stochastic Gradient Descent with momentum . We use test set during validation for convergence analysis.5.1 Global vs. Batch Data Normalization
Since we offer two alternate ways to normalize data (section 4.1) fed to a NormProp network, we evaluate both strategies with different batch sizes to see the difference in performance. We use batch sizes^{4}^{4}4Notice this batch size has nothing to do with the data normalization strategies in discussion. Different batch sizes are used only for adding more variation in experiments. and using initial learning rates and respectively. The results are shown in figure 1. The performance ^{5}^{5}5Even though the numbers are very close, the best accuracy of is achieved by Batch Data Normalization using batch size . using both strategies is very similar for both batch sizes, converging in only epochs. This shows the robustness and applicability of NormProp in both streaming data as well as block data scenario. However, since Batch Data Normalization strategy is a more practical choice, we stick to this strategy for the rest of the experiments when using NormProp.
5.2 NormProp vs. BN– Internal Covariate Shift
The fundamental goal of our paper (as well as that of Batch Normalization Ioffe & Szegedy, 2015) is to alleviate the problem of Internal Covariate Shift. This implies preventing the distribution of hidden layer inputs from shifting while the network is being trained. In deep networks, the features generated by higher layers are completely dependent on the lower features since all the information in data is propagated from lower to higher layers. Thus the problem of Internal Covariate Shift in lower hidden layers is expected to affect the overall performance more severely as compared to the same problem in higher layers.
In order to study the effect of normalization by NormProp and BN on hidden layers, we train two separate networks using each strategy and an additional network without any normalization as our baseline. After every training epoch, we record the mean of the input distribution (over the validation set) to a single randomly chosen (but fixed) unit in each hidden layer. We use batch size and an initial learning rate of for NormProp and BN, and for training the network without any normalization (larger learning rates cause divergence). For the layer convolutional networks we train, the input mean to the last layers against training epoch are shown in figure 2. There are three important observations in these figures: a) NormProp achieves significantly more stable input distribution for lower hidden layers compared to BN, thus facilitating good lower level representation; b) the input distribution for all hidden layers converge after epochs for NormProp. On the other hand, the input distribution to the second layer for BN remains unconverged even after epochs; c) on an average, input distribution to all layers converge closer to zero for NormProp (avg. ) as compared to BN (avg. ). Finally the performance of the network trained without any normalization is incomparable to the normalized ones due to large variations in the hidden layer input distribution (especially the lower layers). This experiment also serves to show the Canonical Error Bound (proposition 1) is small since the input statistics to hidden layers are roughly preserved.
5.3 Convergence Stability of NormProp vs. BN
As a result of alleviating Internal Covariate Shift more accurately during validation as compared to BN, NormProp is expected to achieve a more stable convergence. We confirm this intuition by recording the validation accuracy while the network is being trained. We use batch size and initial learning rates . The plot is shown in figure 3^{6}^{6}6We observed identical trends on SVHN and CIFAR. Additionally, we also experimented optimizing with SGD without momentum and RMS prop (Tieleman & Hinton, 2012)
. We found in general (for most minibatch sizes) the performance of RMS prop was worse than SGD with momentum while that of SGD without momentum was worse than both. On the other hand, RMSProp generally performed better than SGD but SGD with batch size 1 was very similar to SGDMomentum.
. Clearly NormProp achieves a more stable convergence in general, but especially during initial training. This is because NormProp achieves a more stable hidden layer input distribution computed for validation.5.4 Effect of Batchsize on NormProp
We want to see the effect of batchsize used during training with NormProp. Since it is also possible to train with batch size (using Global Data Normalization at data layer), we compare the validation performance of NormProp during training for various batch sizes including . The plots are shown in figure 4. The performance of NormProp is largely unaffected by batch size although lower batch sizes seem to yield better performance.
5.5 Results on various Datasets
We evaluate NormProp and BN on CIFAR, CIFAR and SVHN datasets, but also report existing stateoftheart (SOTA) results. For all the datasets and both methods, we use the same architecture as mentioned in the experimental protocol above except for CIFAR, the last convolutional layer is instead of . For CIFAR datasets we use batch size and an initial learning rate of and reduce it by half after every epochs and train for epochs. Since SVHN is a much larger dataset, we only train for epochs with batch size and an initial learning rate of and reduce it by half after every epochs. We use Stochastic gradient descent with momentum (). For CIFAR and CIFAR, we train using both without data augmentation and with data augmentation (horizontal flipping only); and no data augmentation for SVHN. We did not preprocess any of the datasets. The results are shown in table 1. We find NormProp consistently achieves either better or competitive performance compared to BN, but also beats existing SOTA results.
Datasets and Methods  Test Error () 

CIFAR  
without data augmentation  
NormProp  9.11 
Batch Normalization  9.41 
NIN + ALP units (Agostinelli et al., 2015)  9.59 
NIN (Lin et al., 2014)  10.47 
DSN (Lee et al., 2015)  9.69 
Maxout (Goodfellow et al., 2013)  11.68 
with data augmentation  
NormProp  7.47 
Batch Normalization  7.25 
NIN + ALP units (Agostinelli et al., 2015)  7.51 
NIN (Lin et al., 2014)  8.81 
DSN (Lee et al., 2015)  7.97 
Maxout (Goodfellow et al., 2013)  9.38 
CIFAR  
without data augmentation  
NormProp  32.19 
Batch Normalization  35.32 
NIN + ALP units (Agostinelli et al., 2015)  34.40 
NIN (Lin et al., 2014)  35.68 
DSN (Lee et al., 2015)  34.57 
Maxout (Goodfellow et al., 2013)  38.57 
with data augmentation  
NormProp  29.24 
Batch Normalization  30.26 
NIN + ALP units (Agostinelli et al., 2015)  30.83 
NIN (Lin et al., 2014)   
DSN (Lee et al., 2015)   
Maxout (Goodfellow et al., 2013)   
SVHN  
NormProp  1.88 
Batch Normalization  2.25 
NIN + ALP units (Agostinelli et al., 2015)   
NIN (Lin et al., 2014)  2.35 
DSN (Lee et al., 2015)  1.92 
Maxout (Goodfellow et al., 2013)  2.47 
5.6 Training Speed
Since there is no need for estimating the running average values of input mean and standard deviation for hidden layers for NormProp algorithm, it expected to be faster compared to Batch Normalization. So we record the time taken for NormProp and BN for 1 epoch of training on CIFAR dataset using the experimental protocol used for above experiments. On an NVIDIA GeForce GTX Titan X GPU with Intel i73930K CPU and 32GB Ram machine, NormProp takes sec while BN takes sec.
6 Conclusion
We have proposed a novel algorithm for addressing the problem of Internal Covariate Shift involved during training deep neural networks that overcomes certain drawbacks of Batch Normalization (BN). Specifically, we propose a parametric approach (NormProp) that avoids estimating the mean and standard deviation of hidden layers’ input distribution using input data minibatch statistics (that involve shifting network parameters). Instead, NormProp relies on normalizing the statistics of the given dataset and conditioning the weight matrix which ensures normalization done for the dataset is propagated to all hidden layers. Thus NormProp does not need to maintain a moving average estimate of batch statistics of hidden layer inputs for validation/test phase, thus being more representative of the entire data distribution (especially during initial training period when parameters change drastically). This also enables the use of batch size for training. Although we have shown how to apply NormProp in detail for networks with ReLU activation, we have discussed (section 4.6) how to extend it for other activations as well. We have empirically shown NormProp achieves more stable convergence and hidden layer input distribution over validation set during training, and better/competitive classification performance compared with BN while being faster by omitting the need to compute minibatch estimate of mean/standarddeviation for hidden layers’ input. In conclusion, our approach is applicable alongside any activation function and cost objectives for improving training convergence.
References
 Agostinelli et al. (2015) Agostinelli, Forest, Hoffman, Matthew, Sadowski, Peter, and Baldi, Pierre. Learning activation functions to improve deep neural networks. In ICLR, 2015.
 Desjardins et al. (2015) Desjardins, Guillaume, Simonyan, Karen, Pascanu, Razvan, et al. Natural neural networks. In Advances in Neural Information Processing Systems, pp. 2062–2070, 2015.

Glorot & Bengio (2010)
Glorot, Xavier and Bengio, Yoshua.
Understanding the difficulty of training deep feedforward neural
networks.
In
International conference on artificial intelligence and statistics
, pp. 249–256, 2010.  Goodfellow et al. (2013) Goodfellow, Ian J., Wardefarley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. In In ICML, 2013.

He et al. (2015)
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian.
Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification.
InProceedings of the IEEE International Conference on Computer Vision
, pp. 1026–1034, 2015.  Hyvärinen & Oja (2000) Hyvärinen, Aapo and Oja, Erkki. Independent component analysis: algorithms and applications. Neural networks, 13(4):411–430, 2000.
 Ioffe & Szegedy (2015) Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, Francis R. and Blei, David M. (eds.), ICML, volume 37 of JMLR Proceedings, pp. 448–456. JMLR.org, 2015.
 Krizhevsky (2009) Krizhevsky, Alex. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
 LeCun et al. (2012) LeCun, Yann A, Bottou, Léon, Orr, Genevieve B, and Müller, KlausRobert. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9–48. Springer, 2012.
 Lee et al. (2015) Lee, ChenYu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeplysupervised nets. In ICML, 2015.
 Lin et al. (2014) Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. In ICLR, 2014.
 Makhzani & Frey (2013) Makhzani, Alireza and Frey, Brendan. ksparse autoencoders. CoRR, abs/1312.5663, 2013.
 Netzer et al. (2011) Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5. Granada, Spain, 2011.
 Saxe et al. (2013) Saxe, Andrew M, McClelland, James L, and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
 Shimodaira (2000) Shimodaira, Hidetoshi. Improving predictive inference under covariate shift by weighting the loglikelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000.

Tieleman & Hinton (2012)
Tieleman, Tijmen and Hinton, Geoffrey.
Lecture 6.5rmsprop: Divide the gradient by a running average of its
recent magnitude.
COURSERA: Neural Networks for Machine Learning
, 4:2, 2012. 
Wright et al. (2010)
Wright, John, Ma, Yi, Mairal, Julien, Sapiro, Guillermo, Huang, Thomas S, and
Yan, Shuicheng.
Sparse representation for computer vision and pattern recognition.
Proceedings of the IEEE, 98(6):1031–1044, 2010.
Appendix A Proofs
Proposition 1.
Let where and such that and ( is the identity matrix) . Then the covariance matrix of is approximately canonical satisfying,
(14) 
where is the covariance matrix of , is the coherence of the rows of , is the closest approximation of the covariance matrix to a canonical ellipsoid and diagonalizes a vector to a diagonal matrix. The corresponding optimal .
Proof.
Notice that,
(15) 
On the other hand, the covariance of is given by,
(16) 
Since has spherical covariance, the offdiagonal elements of are zero and the diagonal elements are the variance of any individual unit, since all units are identical. Thus,
(17) 
Thus,
(18) 
in the above equation denotes elementwise square of elements of . Finally minimizing w.r.t , leads to . Substituting this into equation 18, we get,
(19) 
∎
Remark 1.
Let and . Then and
Proof.
For the definition of and , we have,
(20) 
where is sampled from a HalfNormal distribution such that ; thus leading to the claimed result. In order to compute variance, notice that . Then,
(21) 
Substituting yields the claimed result. ∎
Remark 2.
Let and . Then and
Proof.
For the definition of and , half the mass of is concentrated on with HalfNormal distribution, while the other half of the mass is concentrated on with HalfNormal distribution scaled with . Thus,
(22) 
where is sampled from a HalfNormal distribution such that ; thus leading to the claimed result. Similarly in order to compute variance, notice that . Then,
(23) 
Substituting yields the claimed result. ∎
Comments
There are no comments yet.