# Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

While the authors of Batch Normalization (BN) identify and address an important problem involved in training deep networks-- Internal Covariate Shift-- the current solution has certain drawbacks. Specifically, BN depends on batch statistics for layerwise input normalization during training which makes the estimates of mean and standard deviation of input (distribution) to hidden layers inaccurate for validation due to shifting parameter values (especially during initial training epochs). Also, BN cannot be used with batch-size 1 during training. We address these drawbacks by proposing a non-adaptive normalization technique for removing internal covariate shift, that we call Normalization Propagation. Our approach does not depend on batch statistics, but rather uses a data-independent parametric estimate of mean and standard-deviation in every layer thus being computationally faster compared with BN. We exploit the observation that the pre-activation before Rectified Linear Units follow Gaussian distribution in deep networks, and that once the first and second order statistics of any given dataset are normalized, we can forward propagate this normalization without the need for recalculating the approximate statistics for hidden layers.

## Authors

• 21 publications
• 25 publications
• 1 publication
• 11 publications
• ### Why Regularized Auto-Encoders learn Sparse Representation?

While the authors of Batch Normalization (BN) identify and address an im...
05/21/2015 ∙ by Devansh Arpit, et al. ∙ 0

• ### Normalization of Neural Networks using Analytic Variance Propagation

We address the problem of estimating statistics of hidden units in a neu...
03/28/2018 ∙ by Alexander Shekhovtsov, et al. ∙ 0

• ### How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)

Batch Normalization (BatchNorm) is a widely adopted technique that enabl...
05/29/2018 ∙ by Shibani Santurkar, et al. ∙ 0

• ### Solving internal covariate shift in deep learning with linked neurons

This work proposes a novel solution to the problem of internal covariate...
12/07/2017 ∙ by Carles Roger Riera Molina, et al. ∙ 0

• ### Generalized Batch Normalization: Towards Accelerating Deep Neural Networks

Utilizing recently introduced concepts from statistics and quantitative ...
12/08/2018 ∙ by Xiaoyong Yuan, et al. ∙ 0

• ### An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs

Batch Normalization (BN) techniques have been proposed to reduce the so-...
01/09/2020 ∙ by You Huang, et al. ∙ 25

• ### Controlling Covariate Shift using Equilibrium Normalization of Weights

We introduce a new normalization technique that exhibits the fast conver...
12/11/2018 ∙ by Aaron Defazio, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction and Motivation

Ioffe & Szegedy (2015) identified an important problem involved in training deep networks, viz.

, Internal Covariate Shift. It refers to the problem of shifting distribution of the input of every hidden layer in a deep neural network. This idea is borrowed from the concept of covariate shift

(Shimodaira, 2000), where this problem is faced by a single input-output learning system. Consider the last layer of a deep network being used for classification; this layer essentially tries to learn , where

is the class label random variable (r.v.) and

is the layer input r.v. However, learning a fixed becomes a problem if changes continuously. As a result, this slows down training convergence.

Batch Normalization (BN) addresses this problem by normalizing the distribution of every hidden layer’s input. In order to do so, it calculates the pre-activation mean and standard deviation using mini-batch statistics at each iteration of training and uses these estimates to normalize the input to the next layer. While this approach leads to a significant performance jump by addressing internal covariate shift, its estimates of mean and standard-deviation of hidden layer input for validation rely on mini-batch statistics, which are not representative of the entire data distribution (especially during initial training iterations). This is because the mini-batch statistics of input to hidden layers depends on the output from previous layers, which in turn depend on the previous layer parameters that keep shifting during training, and a moving average of these estimates are used for validation. Finally, due to involvement of batch statistics, BN is inapplicable with batch-size .

In this paper, we propose a simple parametric normalization technique for addressing internal covariate shift that does not depend on batch statistics for normalizing the input to hidden layers and is less severely affected by the problem of shifting parameters during validation. In fact, we show that it is unnecessary to explicitly calculate mean and standard-deviation from mini-batches for normalizing the input to hidden layers even for training. Instead, a data independent estimate of these normalization components are available in closed form for every hidden layer, assuming the pre-activation values follow Gaussian distribution and that the weight matrix of hidden layers are roughly incoherent. We show how to forward propagate the normalization property (of the data distribution) to all hidden layers by exploiting the knowledge of the distribution of the pre-activation values (Gaussian) and some algebraic manipulations. Hence we call our approach Normalization Propagation.

## 2 Background

It has long been known in Deep Learning community that input whitening and decorrelation helps in speeding up the training process. In fact, it is explicitly mentioned in

(LeCun et al., 2012) that this whitening should be performed before every layer so that the input to the next layer has zero mean. From the perspective of Internal Covariate Shift, what is required for the network to learn a hypothesis at any given layer at every point in time during training, is for the distribution

of the input to that layer to be fixed. While whitening could be used for achieving this task at every layer, it would be a very expensive choice (cubic order of input size) since whitening dictates computing the Singular Value Decomposition (SVD) of the input data matrix. However,

Desjardins et al. (2015) suggest to overcome this problem by approximating this SVD by: a) using sub-sampled training data to compute this SVD; b) computing it every few number of iteration and relying on the assumption that this SVD approximately holds for the iterations in between. In addition, each hidden layer’s input is then whitened by re-parametrizing a subset of network parameters that are involved in gradient descent. As mentioned in Ioffe & Szegedy (2015), this re-parametrizing may lead to effectively cancelling/attenuating the effect of the gradient update step since these two operations are done independently.

Batch Normalization (Ioffe & Szegedy, 2015) addresses both the above problems. First, they propose a strategy for normalizing the data at hidden layers such that the gradient update step accounts for this normalization. Secondly, this normalization is performed for units of each hidden layer independently (thus avoiding whitening) using mini-batch statistics. Specifically, this is achieved by normalizing the pre-activation of all hidden layers as,

 ^ui=ui−EB[ui]√varB(ui) (1)

where denotes the element of

and the expectation/variance is calculated over the training mini-batch

. Notice since is a part of this normalization, it becomes a part of the gradient descent step as well. However, a problem common to both the above approaches is that of shifting network parameters upon which their approximation of input normalization for hidden layers depends.

## 3 Normalization Propagation (NormProp) Derivation

We will now describe the idea behind NormProp. At a glance the problem at hand seems cyclic because estimating the mean and standard deviation of the input distribution to any hidden layer requires the input distribution of its previous layer (and hence its parameters) to be fixed to the optimal value before hand. However, as we will now show, we can side-step this naive approach and get an approximation of the true unbiased estimate using the knowledge that the pre-activation to every hidden layer follows a Gaussian distribution and some algebraic manipulation over the properties of the weight matrix. For the derivation below, we will focus on networks with ReLU activation, and later discuss how to extend our algorithm to other activation functions.

### 3.1 Data Normalization

Real world data generally follows Gaussian like distribution. Therefore, consider a data distribution in such that all the samples are normalized, i.e.,

 Ex∈X[x]=0Ex∈X[x2j]=1∀j∈{1,…,n} (2)

Then our goal is to find a way to propagate this normalization to all the hidden layers without the need of explicit data dependent normalization. Depending on whether this input is passed through a convolutional layer or a fully connected layer, a part of the input or the entire input gets multiplied to a weight matrix. Irrespective of the case, lets use

to denote this input for ease of notation; which can thus be the entire data vector or a subset of its element depending on the case. The pre-activation is given by

, where and is the number of filters (we will ignore bias for now). As also claimed by Ioffe & Szegedy (2015); Hyvärinen & Oja (2000), we assume the pre-activation () has a Gaussian form. So if we ensure that the pre-activation of the first hidden layer is a normalized Gaussian, then the hidden layer’s output () will be a Rectified Gaussian distribution. As mentioned in section 2, we can choose to directly normalize the post-activation output . However, as we will now show, it is easier to find closed form estimates for normalizing instead.

### 3.2 Mean and Standard-deviation Normalization for First Hidden Layer

Clearly, since the input data is mean subtracted, the pre-activation to the first hidden layer also has zero mean from linearity,i.e., . Now we want to ensure the variance of is . Let the covariance matrix of be denoted by . Then the following proposition bounds how far is from a canonical distribution.

###### Proposition 1.

(Canonical Error Bound) Let where and such that and (

is the identity matrix) . Then the covariance matrix of

is approximately canonical satisfying,

 minα∥Σ−diag(α)∥F ≤σ2μ ⎷m∑i,j=1;i≠j∥Wi∥22∥Wj∥22 (3)

where is the covariance matrix of , is the coherence111Coherence is defined as of the rows of , is the closest approximation of the covariance matrix to a canonical ellipsoid and diagonalizes a vector to a diagonal matrix. The corresponding optimal .

The above proposition tells us two things. First, the covariance matrix of the pre-activation is approximately canonical (diagonal covariance matrix) if the above error bound can be made tight, and that this tightness can be controlled by certain properties of the weight matrix . Second, if we want to normalize each element of the vector to have unit standard-deviation, then our best bet is to divide each by the corresponding weight length if we ensure tight canonical error bound. This is because the closest estimate of a diagonal variance for is ( in our case).

For any dictionary (weight matrix ), the bound above can be made tighter my minimizing coherence . In our approach, we also need to normalize each element of the vector to have unit standard-deviation which is achieved by dividing each by . Notice this automatically makes each hidden weight vector to effectively have unit length. As a result, the error bound only depends on the coherence of . On the other hand, it is generally observed that useful filters that constitute a good representation of real world data are roughly incoherent (Wright et al., 2010; Makhzani & Frey, 2013); thus ensuring the R.H.S is small thereby minimizing the error bound.

At this point, we have normalized the pre-activation to have zero mean and unit variance (divide each pre-activation element by corresponding ). As a result, the output of the first hidden layer () is Rectified Gaussian distribution. Notice that the above bound ensures the dimensions of and hence () are roughly uncorrelated. Thus, if we subtract the distribution mean from and divide by its standard deviation, we will have reduced the dynamics of the second layer to be identical to that of the first layer. The mean and standard deviation of the aforementioned Rectified Gaussian is,

###### Remark 1.

(Post-ReLU distribution) Let and . Then and

Hence in order to normalize the post-activation to have zero mean and unit standard, the above calculated values can be used. Finally, in the case of Pooling (in Conv-Nets), we essentially take a block of post-activated units and take average or maximum of these values. If we consider each such unit to be independent then the distribution after pooling, will have a different mean and standard deviation. However, in reality, each of these units are highly correlated since they involve computation over either overlapping or spatially close patches. Therefore, we found that the distribution statistics do not get affected significantly and hence we do not recompute mean and standard deviation post-pooling.

### 3.3 Propagation to Higher Layers

With the above two operations, the dynamics of the second hidden layer become identical to that of the first hidden layer. By induction, repeating these two operations for every layer, viz.–1) divide every hidden layer’s pre-ReLU-activation by its corresponding , where is the corresponding layer’s weight matrix, 2) subtract and divide and (respectively) from every hidden layer’s post-ReLU-activation– we ensure that the input to every layer is a canonical distribution. While training, all these normalization operations are back-propagated.

### 3.4 Effect of NormProp on Jacobian of Hidden Layers

It has been discussed in Saxe et al. (2013); Ioffe & Szegedy (2015) that Jacobian of hidden layers with singular values close to one improves training convergence in deep networks. While BN has been shown to intuitively achieve this condition, we will now show more rigorously that NormProp (approximately) indeed achieves this condition.

Let be a vector such that the element of is given by . The output of a hidden unit using NormProp is given by where , , and . Let be such that the row of equals . Thus the output can be rewritten as . Let denote the Jacobian of this output with respect to the previous layer input . Then the row of is given by

 Ji≜1c1∂ReLU(~Wix)∂~Wix∂~Wix∂x=1c1∂ReLU(~Wix)∂~Wix~Wi (4)

where denotes the row of . Let be an indicator vector such that the element of is given by

 1xi≜∂ReLU(~Wix)∂~Wix=1(~Wix>0) (5)

where is the indicator operator. Let be a matrix such that every column of is occupied by the vector . Then the entire Jacobian matrix can be written as . In order to analyze the singular values of , we want to calculate . From proposition 1, the covariance matrix of the pre-activation is given by , where . Since the length of each is , , and (therefore) is approximately an identity matrix if the rows of are incoherent. Thus,

 (6)

Finally taking an expectation of over the distribution of which is Normal, we get,

 Ex[JJT]≈1c21Ex[diag(1x)]=1c21∫diag(1x)p(x)d(x) (7)

where denotes the density of

Normal distribution. From the definition of

, it is straight forward to see that the result of the integral is a matrix with its diagonal equal to a vector of , hence, , where is the identity matrix. Thus the singular values of the Jacobian are which, being close to , approximately achieves dynamical isometry (Saxe et al., 2013) and should thus prevent the problem of exploding or diminishing gradients while training deep networks suggesting faster convergence. In the next section, we will use this value during the practical implementation of NormProp for improving the Jacobian to be approximately .

## 4 NormProp: Implementation Details

We have all the ingredients required to filter out the steps for Normalization Propagation for training any deep neural network with ReLU activation though out its hidden layers. Like BN, NormProp can be used alongside any optimization algorithm (eg. Stochastic Gradient Descent with/without momentum) for training deep networks.

### 4.1 Normalize Data

Since the core idea of NormProp is to propagate the data normalization through hidden layers, we offer two alternative choices either one of which can be used for normalizing the input to a NormProp network. As we will describe, both options are justified in their respective scenario.

1. Global Data Normalization: In cases when the entire dataset – approximately representing the true data distribution – is available at hand, we compute the global mean and standard deviation for each feature element. Then the first step for NormProp is to subtract element-wise mean calculated over the entire dataset from each sample. Similarly divide each feature element by the element-wise standard-deviation. Ideally it is required by NormProp that all input dimensions be statistically uncorrelated, a property achieved by whitening for instance, but we suggest element-wise normalization as an approximation since it is computationally cheaper. Notice this precludes the dilemma of what range the input should be scaled to before passing through the network.

2. Batch Data Normalization: In many real world scenario, streaming data is available and thus it is not possible to compute an unbiased estimate of global mean and standard deviation at any given point in time. In such cases, we propose to instead batch-normalize every mini-batch training data fed to the network. Again, we perform the normalization of each feature element independently for computational purposes. Notice this normalization is only performed at the data level, all hidden layers are still normalized by the NormProp strategy which is not affected by shifting model parameters as compared to BN. Moreover, Batch Data Normalization also serves as a regularization since each data sample gets a different representation each time depending on the mini-batch it comes with. Thus by using the Batch Data Normalization strategy we actually benefit from the regularization aspect of BN but also overcome its drawbacks by computing the hidden layer mean and standard-deviation without depending on batch statistics. Notice this strategy is most effective when the incoming data is well shuffled.

### 4.2 Initialize Network Parameters

We use Normalized Initialization (Glorot & Bengio, 2010)

for setting the initial values of all the weight matrices, both fully connected and convolutional. Bias vectors are initialized to zeros and scaling vectors (described in the next subsection) can either be initialized to

or as described in the next subsection.

### 4.3 Propagate Normalization

Similar to BN, we also make use of gradient-based-learnable scaling and bias parameters and during implementation. We will now describe our normalization in detail for both fully connected and convolutional layers.

#### 4.3.1 Fully Connected Layers

Consider any fully connected layer characterized by a weight matrix , bias , scaling , input and activation ReLU. Here denotes the number of filters and denotes the input dimension. Then without NormProp, the output unit of the hidden layer would traditionally be:

 oi=ReLU(WTix+βi) (8)

Now in the case of NormProp, the output becomes,

 oi=1√12(1−1π)[% ReLU(γi(WTix)∥Wi∥2+βi)−√12π] (9)

Here we initialize each to in order to make the Jacobian close to one as suggested by our analysis in section 3.4 for ReLU activation. Thus we call this number the Jacobian factor. We found this initializing using Jacobian factor helps training with larger learning rates without diverging. However, one can also choose to treat the initialization value as a hyper-parameter.

#### 4.3.2 Convolutional Layers

Consider any convolutional layer characterized by a filter matrix , bias , scaling , input

and activation ReLU along with any arbitrary choice of stride-size. Here,

denotes the number of filters, –depth, –height, – width for input/filters and L,B– height and width of image. Then without NormProp, the output feature map of the hidden layer using the filter would traditionally be:

 oi=ReLU(Wi∗x+βi) (10)

where denotes the convolution operation. Now in the case of NormProp, the output feature map becomes,

 (11)

where each element of is again initialized to . Notice each is multiplied to all outputs from the same corresponding filter and similarly all the scalars as well as the bias vector are broadcasted to all the dimensions. Pooling is done after this normalization process the same way as done traditionally.

### 4.4 Training

The network is trained using Back Propagation. While doing so, all the normalizations also get back-propagated at every layer.

Optimization: We use Stochastic Gradient Descent with momentum (set to ) for training. Data shuffling also leads to performance improvement (this however, is true in general while training deep networks).

Learning Rate: We found learning speeds up by reducing the learning rate by half whenever the training error starts saturating. Also, we found larger initial learning rate for larger batch size improves performance.

Constraints: After every training iteration, we scale each hidden weight-vector/filter-map to have unit length, i.e., we use constraint on all hidden weights, both convolutional and fully connected. This is done because the scale of weight vectors do not affect network representation, so constraining the weights should reduce the parameter search space.

Regularizations:

We use weight decay along with the loss function; we found a small coefficient value of

is necessary during training. We found Dropout does not help during training; we believe this might be because Dropout changes the distribution of output of the layer it is applied, which affects NormProp.

### 4.5 Validation and Testing

Validation and test procedures are identical for NormProp. While validation/testing, each sample is first normalized using mean and standard deviation which are calculated depending on how the train data is normalized during training. In case we use Global Data Normalization during training, we simply use the same global estimate of mean and standard deviation to normalize each test/validation sample. On the other hand, if Batch Data Normalization is used during training, a running estimate of mean and standard deviation is maintained during training which is then used to normalize every test/validation sample. Finally, the input is forward propagated though the network with learned parameters using the same strategy described in section 4.3.

### 4.6 Extension to other Activation Functions

Even though our paper shows how to overcome the problem of Internal Covariate Shift specifically for networks with ReLU activation throughout, we have in essence proposed a general framework for propagating normalization done at data level to all hidden layers. All that is needed for extending NormProp to other activation functions is to compute the distribution mean () and standard deviation () of output after the activation function of choice, similar to what is shown in remark 1. Thus the general form of output for any given activation becomes222Using the appropriate Jacobian Factor allows the use of larger learning rate; however, NormProp works without it as well. (shown for convolution layer as an example),

 oi=1c1[σ(γi(Wi∗x)∥Wi∥F+βi)−c2] (12)

This activation can be both parameter based or fixed. For instance, a parameter based activation is Parametric ReLU (PReLU, He et al. (2015)) (with parameter ) given by,

 PReLUa(x)={xifx>0axifx≤0 (13)

Then the post PReLU distribution statistics is given by,

###### Remark 2.

Let and . Then and

Notice the distribution mean and standard deviation depends on the parameter and thus will be involved in the normalization process. In case of non-parameter based activations (eg. Tanh, Sigmoid), one can either choose to analytically compute the statistics (like we did for ReLU) or compute these values empirically by simulation since the input distribution to the activation is a fixed Normal distribution. Thus NormProp is a general framework which can be extended to any activation function of choice.

## 5 Empirical Results and Observations

We want to verify the following: a) performance comparison of NormProp when using Global Data Normalization vs. Batch Data Normalization; b) NormProp alleviates the problem of Internal Covariate Shift more accurately compared to BN; c) thus, convergence stability of NormProp is better than BN; d) effect of batch-size on the behaviour of NormProp, especially batch-size (BN not applicable). Finally we report classification result on various datasets using NormProp and BN.

Datasets: We use the following datasets,
1) CIFAR- (Krizhevsky, 2009)– It consists of real world color images in 10 classes split into train and test images. We use images from train set for validation and remaining for training.
2) CIFAR-– It has the same number of train and test samples as CIFAR- but it has

classes. For training, we use hyperparameters same as those for CIFAR-

.
3) SVHN (Netzer et al., 2011)– It consists of color images of house numbers collected by Google Street View. It has train images, test images and an additional train images. Similar to the protocol in (Goodfellow et al., 2013), we select 400 samples per class from the train set and 200 samples per class from the extra set as validation and use the remaining images of the train and extra sets for training.

Experimental Protocols (For experiments in sections 5.1 through 5.4): We use CIFAR- with the following Network in Network (Lin et al., 2014) architecture333

We use the following shorthand for a) conv layer: C(number of filters, filter size, stride size, padding); b) pooling: P(kernel size, stride, padding, pool mode)

. For any specified initial learning rate, we reduce it by half every epochs. We use Stochastic Gradient Descent with momentum . We use test set during validation for convergence analysis.

### 5.1 Global vs. Batch Data Normalization

Since we offer two alternate ways to normalize data (section 4.1) fed to a NormProp network, we evaluate both strategies with different batch sizes to see the difference in performance. We use batch sizes444Notice this batch size has nothing to do with the data normalization strategies in discussion. Different batch sizes are used only for adding more variation in experiments. and using initial learning rates and respectively. The results are shown in figure 1. The performance 555Even though the numbers are very close, the best accuracy of is achieved by Batch Data Normalization using batch size . using both strategies is very similar for both batch sizes, converging in only epochs. This shows the robustness and applicability of NormProp in both streaming data as well as block data scenario. However, since Batch Data Normalization strategy is a more practical choice, we stick to this strategy for the rest of the experiments when using NormProp.

### 5.2 NormProp vs. BN– Internal Covariate Shift

The fundamental goal of our paper (as well as that of Batch Normalization Ioffe & Szegedy, 2015) is to alleviate the problem of Internal Covariate Shift. This implies preventing the distribution of hidden layer inputs from shifting while the network is being trained. In deep networks, the features generated by higher layers are completely dependent on the lower features since all the information in data is propagated from lower to higher layers. Thus the problem of Internal Covariate Shift in lower hidden layers is expected to affect the overall performance more severely as compared to the same problem in higher layers.

In order to study the effect of normalization by NormProp and BN on hidden layers, we train two separate networks using each strategy and an additional network without any normalization as our baseline. After every training epoch, we record the mean of the input distribution (over the validation set) to a single randomly chosen (but fixed) unit in each hidden layer. We use batch size and an initial learning rate of for NormProp and BN, and for training the network without any normalization (larger learning rates cause divergence). For the layer convolutional networks we train, the input mean to the last layers against training epoch are shown in figure 2. There are three important observations in these figures: a) NormProp achieves significantly more stable input distribution for lower hidden layers compared to BN, thus facilitating good lower level representation; b) the input distribution for all hidden layers converge after epochs for NormProp. On the other hand, the input distribution to the second layer for BN remains un-converged even after epochs; c) on an average, input distribution to all layers converge closer to zero for NormProp (avg. ) as compared to BN (avg. ). Finally the performance of the network trained without any normalization is in-comparable to the normalized ones due to large variations in the hidden layer input distribution (especially the lower layers). This experiment also serves to show the Canonical Error Bound (proposition 1) is small since the input statistics to hidden layers are roughly preserved.

### 5.3 Convergence Stability of NormProp vs. BN

As a result of alleviating Internal Covariate Shift more accurately during validation as compared to BN, NormProp is expected to achieve a more stable convergence. We confirm this intuition by recording the validation accuracy while the network is being trained. We use batch size and initial learning rates . The plot is shown in figure 3666We observed identical trends on SVHN and CIFAR-. Additionally, we also experimented optimizing with SGD without momentum and RMS prop (Tieleman & Hinton, 2012)

. We found in general (for most mini-batch sizes) the performance of RMS prop was worse than SGD with momentum while that of SGD without momentum was worse than both. On the other hand, RMSProp generally performed better than SGD but SGD with batch size 1 was very similar to SGD-Momentum.

. Clearly NormProp achieves a more stable convergence in general, but especially during initial training. This is because NormProp achieves a more stable hidden layer input distribution computed for validation.

### 5.4 Effect of Batch-size on NormProp

We want to see the effect of batch-size used during training with NormProp. Since it is also possible to train with batch size (using Global Data Normalization at data layer), we compare the validation performance of NormProp during training for various batch sizes including . The plots are shown in figure 4. The performance of NormProp is largely unaffected by batch size although lower batch sizes seem to yield better performance.

### 5.5 Results on various Datasets

We evaluate NormProp and BN on CIFAR-, CIFAR- and SVHN datasets, but also report existing state-of-the-art (SOTA) results. For all the datasets and both methods, we use the same architecture as mentioned in the experimental protocol above except for CIFAR-, the last convolutional layer is instead of . For CIFAR datasets we use batch size and an initial learning rate of and reduce it by half after every epochs and train for epochs. Since SVHN is a much larger dataset, we only train for epochs with batch size and an initial learning rate of and reduce it by half after every epochs. We use Stochastic gradient descent with momentum (). For CIFAR- and CIFAR-, we train using both without data augmentation and with data augmentation (horizontal flipping only); and no data augmentation for SVHN. We did not pre-process any of the datasets. The results are shown in table 1. We find NormProp consistently achieves either better or competitive performance compared to BN, but also beats existing SOTA results.

### 5.6 Training Speed

Since there is no need for estimating the running average values of input mean and standard deviation for hidden layers for NormProp algorithm, it expected to be faster compared to Batch Normalization. So we record the time taken for NormProp and BN for 1 epoch of training on CIFAR- dataset using the experimental protocol used for above experiments. On an NVIDIA GeForce GTX Titan X GPU with Intel i7-3930K CPU and 32GB Ram machine, NormProp takes sec while BN takes sec.

## 6 Conclusion

We have proposed a novel algorithm for addressing the problem of Internal Covariate Shift involved during training deep neural networks that overcomes certain drawbacks of Batch Normalization (BN). Specifically, we propose a parametric approach (NormProp) that avoids estimating the mean and standard deviation of hidden layers’ input distribution using input data mini-batch statistics (that involve shifting network parameters). Instead, NormProp relies on normalizing the statistics of the given dataset and conditioning the weight matrix which ensures normalization done for the dataset is propagated to all hidden layers. Thus NormProp does not need to maintain a moving average estimate of batch statistics of hidden layer inputs for validation/test phase, thus being more representative of the entire data distribution (especially during initial training period when parameters change drastically). This also enables the use of batch size for training. Although we have shown how to apply NormProp in detail for networks with ReLU activation, we have discussed (section 4.6) how to extend it for other activations as well. We have empirically shown NormProp achieves more stable convergence and hidden layer input distribution over validation set during training, and better/competitive classification performance compared with BN while being faster by omitting the need to compute mini-batch estimate of mean/standard-deviation for hidden layers’ input. In conclusion, our approach is applicable alongside any activation function and cost objectives for improving training convergence.

## References

• Agostinelli et al. (2015) Agostinelli, Forest, Hoffman, Matthew, Sadowski, Peter, and Baldi, Pierre. Learning activation functions to improve deep neural networks. In ICLR, 2015.
• Desjardins et al. (2015) Desjardins, Guillaume, Simonyan, Karen, Pascanu, Razvan, et al. Natural neural networks. In Advances in Neural Information Processing Systems, pp. 2062–2070, 2015.
• Glorot & Bengio (2010) Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In

International conference on artificial intelligence and statistics

, pp. 249–256, 2010.
• Goodfellow et al. (2013) Goodfellow, Ian J., Warde-farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. In In ICML, 2013.
• He et al. (2015) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian.

Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.

In

Proceedings of the IEEE International Conference on Computer Vision

, pp. 1026–1034, 2015.
• Hyvärinen & Oja (2000) Hyvärinen, Aapo and Oja, Erkki. Independent component analysis: algorithms and applications. Neural networks, 13(4):411–430, 2000.
• Ioffe & Szegedy (2015) Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Bach, Francis R. and Blei, David M. (eds.), ICML, volume 37 of JMLR Proceedings, pp. 448–456. JMLR.org, 2015.
• Krizhevsky (2009) Krizhevsky, Alex. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
• LeCun et al. (2012) LeCun, Yann A, Bottou, Léon, Orr, Genevieve B, and Müller, Klaus-Robert. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9–48. Springer, 2012.
• Lee et al. (2015) Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen. Deeply-supervised nets. In ICML, 2015.
• Lin et al. (2014) Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. In ICLR, 2014.
• Makhzani & Frey (2013) Makhzani, Alireza and Frey, Brendan. k-sparse autoencoders. CoRR, abs/1312.5663, 2013.
• Netzer et al. (2011) Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp.  5. Granada, Spain, 2011.
• Saxe et al. (2013) Saxe, Andrew M, McClelland, James L, and Ganguli, Surya. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
• Shimodaira (2000) Shimodaira, Hidetoshi. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference, 90(2):227–244, 2000.
• Tieleman & Hinton (2012) Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.

COURSERA: Neural Networks for Machine Learning

, 4:2, 2012.
• Wright et al. (2010) Wright, John, Ma, Yi, Mairal, Julien, Sapiro, Guillermo, Huang, Thomas S, and Yan, Shuicheng.

Sparse representation for computer vision and pattern recognition.

Proceedings of the IEEE, 98(6):1031–1044, 2010.

## Appendix A Proofs

###### Proposition 1.

Let where and such that and ( is the identity matrix) . Then the covariance matrix of is approximately canonical satisfying,

 minα∥Σ−diag(α)∥F≤σ2μ ⎷m∑i,j=1;i≠j∥Wi∥22∥Wj∥22 (14)

where is the covariance matrix of , is the coherence of the rows of , is the closest approximation of the covariance matrix to a canonical ellipsoid and diagonalizes a vector to a diagonal matrix. The corresponding optimal .

###### Proof.

Notice that,

 Eu[u]=WEx[x]=0 (15)

On the other hand, the covariance of is given by,

 Σ=Eu[(u−Eu[u])(u−Eu[u])T]=Ex[(Wx−WEx[x])(Wx−WEx[x])T]=Ex[W(x−Ex[x])(x−Ex[x])TWT]=WEx[(x−Ex[x])(x−Ex[x])T]WT (16)

Since has spherical covariance, the off-diagonal elements of are zero and the diagonal elements are the variance of any individual unit, since all units are identical. Thus,

 Eu[(u−Eu[u])(u−Eu[u])T]=σ2WWT (17)

Thus,

 (18)

in the above equation denotes element-wise square of elements of . Finally minimizing w.r.t , leads to . Substituting this into equation 18, we get,

 ∥Σ−diag(α)∥2F≤σ4m∑i,j=1;i≠jμ2∥Wi∥22∥Wj∥22 (19)

###### Remark 1.

Let and . Then and

###### Proof.

For the definition of and , we have,

 E[Y]=12.0+12E[Z]=12E[Z] (20)

where is sampled from a Half-Normal distribution such that ; thus leading to the claimed result. In order to compute variance, notice that . Then,

 var(Y)=E[Y2]−E[Y]2=0.5E[Z2]−14E[Z]2=0.5(var(Z)+E[Z]2)−14E[Z]2 (21)

Substituting yields the claimed result. ∎

###### Remark 2.

Let and . Then and

###### Proof.

For the definition of and , half the mass of is concentrated on with Half-Normal distribution, while the other half of the mass is concentrated on with Half-Normal distribution scaled with . Thus,

 E[Y]=−a2E[Z]+12E[Z]=(1−a)12E[Z] (22)

where is sampled from a Half-Normal distribution such that ; thus leading to the claimed result. Similarly in order to compute variance, notice that . Then,

 var(Y)=E[Y2]−E[Y]2=0.5E[Z2](1+a2)−(1−a)214E[Z]2=0.5(1+a2)(var(Z)+E[Z]2)−(1−a)214E[Z]2 (23)

Substituting yields the claimed result. ∎