1 Introduction
The broad success of deep learning largely owes to the recent advances on largescale datasets
[43], powerful computing resources (e.g., GPUs and TPUs), sophisticated network architectures [15, 16] and optimization algorithms [4, 24]. Among these factors, the efficient optimization techniques, such as stochastic gradient descent (SGD) with momentum
[38], Adagrad [10] and Adam [24], make it possible to train very deep neural networks (DNNs) with a largescale dataset, and consequently deliver more powerful and robust DNN models in practice. The generalization performance of the trained DNN models as well as the efficiency of training process depend essentially on the employed optimization techniques.There are two major goals for a good DNN optimizer: accelerating the training process and improving the model generalization capability. The first goal aims to spend less time and cost to reach a good local minima, while the second goal aims to ensure that the learned DNN model can make accurate predictions on test data. A variety of optimization algorithms [38, 10, 24, 10, 24] have been proposed to achieve these goals. SGD [4, 5] and its extension SGD with momentum (SGDM) [38] are among the most commonly used ones. They update the parameters along the opposite direction of their gradients in one training step. Most of the current DNN optimization methods are based on SGD and improve SGD to better overcome the gradient vanishing or explosion problems. A few successful techniques have been proposed, such as weight initialization strategies [11, 14]
, efficient active functions (e.g., ReLU
[35][36, 37], adaptive learning rate optimization algorithms [10, 24], and so on.In addition to the above techniques, the sample/feature statistics such as mean and variance can also be used to normalize the network activations or weights to make the training process more stable. The representative methods operating on activations include batch normalization (BN) [19], instance normalization (IN) [47, 18], layer normalization (LN) [29] and group normalization (GN) [51]. Among them, BN is the most widely used optimization technique which normalizes the features along the sample dimension in a minibatch for training. BN smooths the optimization landscape [45] and it can speed up the training process and boost model generalization performance when a proper batch size is used [53, 15]. However, BN works not very well when the training batch size is small, which limits its applications to memory consuming tasks, such as object detection [13, 42], video classification [21, 2], etc.
Another line of statistics based methods operate on weights. The representative ones include weight normalization (WN) [44, 17] and weight standardization (WS) [39]
. These methods reparameterize weights to restrict weight vectors during training. For example, WN decouples the length of weight vectors from their direction to accelerate the training of DNNs. WS uses the weight vectors’ mean and variance to standardize them to have zero mean and unit variance. Similar to BN, WS can also smooth the loss landscape and speed up training. Nevertheless, such methods operating on weight vectors cannot directly adopt the pretrained models (e.g., on ImageNet) because their weights may not meet the condition of zero mean and unit variance.
Different from the above techniques which operate on activations or weight vectors, we propose a very simple yet effective DNN optimization technique, namely gradient centralization (GC), which operates on the gradients of weight vectors. As illustrated in Fig. 1(a), GC simply centralizes the gradient vectors to have zero mean. It can be easily embedded into the current gradient based optimization algorithms (e.g., SGDM [38], Adam [24]) using only one line of code. Though simple, GC demonstrates various desired properties, such as accelerating the training process, improving the generalization performance, and the compatibility for finetuning pretrained models. The main contributions of this paper are highlighted as follows:

We propose a new general network optimization technique, namely gradient centralization (GC), which can not only smooth and accelerate the training process of DNN but also improve the model generalization performance.

We analyze the theoretical properties of GC, and show that GC constrains the loss function by introducing a new constraint on weight vector, which regularizes both the weight space and output feature space so that it can boost model generalization performance. Besides, the constrained loss function has better Lipschitzness than the original one, which makes the training process more stable and efficient.
Finally, we perform comprehensive experiments on various applications, including general image classification, finegrained image classification, object detection and instance segmentation. The results demonstrate that GC can consistently improve the performance of learned DNN models in different applications. It is a simple, general and effective network optimization method.
. (b) Illustration of the GC operation on gradient matrix/tensor of weights in the fullyconnected layer (left) and convolutional layer (right). GC computes the column/slice mean of gradient matrix/tensor and centralizes each column/slice to have zero mean.
2 Related Work
In order to accelerate the training and boost the generalization performance of DNNs, a variety of optimization techniques [19, 51, 44, 39, 38, 36] have been proposed to operate on activation, weight and gradient. In this section we briefly review the related work from these three aspects.
Activation: The activation normalization layer has become a common setting in DNN, such as batch normalization (BN) [19] and group normalization (GN) [51]. BN was originally introduced to solve the internal covariate shift by normalizing the activations along the sample dimension. It allows higher learning rates [3], accelerates the training speed and improves the generalization accuracy [33, 45]. However, BN does not perform well when the training batch size is small, and GN is proposed to address this problem by normalizing the activations or feature maps in a divided group for each input sample. In addition, layer normalization (LN) [29] and instance normalization (IN) [47, 18]
have been proposed for RNN and style transfer learning, respectively.
Weight: Weight normalization (WN) [44] reparameterizes the weight vectors and decouples the length of a weight vector from its direction. It speeds up the convergence of SGDM algorithm to a certain degree. Weight standardization (WS) [39]
adopts the Zscore standardization to reparameterize the weight vectors. Like BN, WS can also smooth the loss landscape and improve training speed. Besides, binarized DNN
[40, 9, 8] quantifies the weight into binary values, which can improve the generalization capability for certain DNNs. However, a shortcoming of those methods operating on weights is that they cannot be directly used to finetune pretrained models since the pretrained weight may not meet their constraints. As a consequence, we have to design specific pretraining methods for them in order to finetune the model.Gradient: A commonly used operation on gradient is to compute the momentum of gradient [38]. By using the momentum of gradient, SGDM accelerates SGD in the relevant direction and dampens oscillations. Besides, regularization based weight decay, which introduces regularization into the gradient of weight, has long been a standard trick to improve the generalization performance of DNNs [27, 54]. To make DNN training more stable and avoid gradient explosion, gradient clipping [36, 37, 1, 23] has been proposed to train a very deep DNNs. In addition, the projected gradient methods [12, 28] and Riemannian approach [7, 49] project the gradient on a subspace or a Riemannian manifold to regularize the learning of weights.
3 Gradient Centralization
3.1 Motivation
BN [19] is a powerful DNN optimization technique, which uses the first and second order statistics to perform Zscore standardization on activations. It has been shown in [45] that BN reduces the Lipschitz constant of loss function and makes the gradients more Lipschitz smooth so that the optimization landscape becomes smoother. WS [39] can also reduce the Lipschitzness of loss function and smooth the optimization landscape through Zscore standardization on weight vectors. BN and WS operate on activations and weight vectors, respectively, and they implicitly constrict the gradient of weights, which improves the Lipschitz property of loss for optimization.
Apart from operating on activation and weight, can we directly operate on gradient to make the training process more effective and stable? One intuitive idea is that we use Zscore standardization to normalize gradient, like what has been done by BN and WS on activation and weight. Unfortunately, we found that normalizing gradient cannot improve the stability of training. Instead, we propose to compute the mean of gradient vectors and centralize the gradients to have zero mean. As we will see in the following development, the so called gradient centralization (GC) method can have good Lipschitz property, smooth the DNN training and improve the model generalization performance.
3.2 Notations
We define some basic notations. For fully connected layers (FC layers), the weight matrix is denoted as , and for convolutional layers (Conv layers) the weight tensor is denoted as , where is the number of input channels, is the number of output channels, and , are the kernel size of convolution layers. For the convenience of expression, we unfold the weight tensor of Conv layer into a matrix/tensor and use a unified notation for weight matrix in FC layer () and Conv layers (). Denote by () the th column vector of weight matrix and the objective function. and denote the gradient of w.r.t. the weight matrix and weight vector , respectively. The size of gradient matrix is the same as weight matrix . Let be the input activations for this layer and be its output activations. denotes an dimensional unit vector and
denotes an identity matrix.
3.3 Formulation of GC
For a FC layer or a Conv layer, suppose that we have obtained the gradient through backward propagation, then for a weight vector whose gradient is (), the GC operator, denoted by , is defined as follows:
(1) 
where . The formulation of GC is very simple. As shown in Fig. 1(b), we only need to compute the mean of the column vectors of the weight matrix, and then remove the mean from each column vector. We can also have a matrix formulation of Eq. (1):
(2) 
The physical meaning of will be explained later in Section 4.1. In practical implementation, we can directly remove the mean value from each weight vector to accomplish the GC operation. The computation is very simple and efficient.
3.4 Embedding of GC to SGDM/Adam
GC can be easily embedded into the current DNN optimization algorithms such as SGDM [38, 5] and Adam [24]. After obtaining the centralized gradient , we can directly use it to update the weight matrix. Algorithm 1 and Algorithm 2 show how to embed GC into the two most popular optimization algorithms, SGDM and Adam, respectively. Moreover, if we want to use weight decay, we can set , where is the weight decay factor. It only needs to add one line of code into most existing DNN optimization algorithms to execute GC with negligible additional computational cost. For example, it costs only
sec extra training time in one epoch on CIFAR100 with ResNet50 model in our experiments (71 sec for one epoch).
4 Properties of GC
As we will see in the section of experimental result, GC can accelerate the training process and improve the generalization performance of DNNs. In this section, we perform theoretical analysis to explain why GC works.
4.1 Improving Generalization Performance
One important advantage of GC is that it can improve the generalization performance of DNNs. We explain this advantage from two aspects: weight space regularization and output feature space regularization.
Weight space regularization: Let’s first explain the physical meaning of in Eq.(2). Actually, it is easy to prove that:
(3) 
The above equations show that
is the projection matrix for the hyperplane with normal vector
in weight space, and is the projected gradient.The property of projected gradient has been investigated in some previous works [12, 28, 7, 49], which indicate that projecting the gradient of weight will constrict the weight space in a hyperplane or a Riemannian manifold. Similarly, the role of GC can also be viewed from the perspective of projected gradient descant. We give a geometric illustration of SGD with GC in Fig. 2. As shown in Fig. 2, in the th step of SGD with GC, the gradient is first projected on the hyperplane determined by , where is the weight vector in the th iteration, and then the weight is updated along the direction of projected gradient . From , we have , i.e., is a constant during training. Mathematically, the latent objective function w.r.t. one weight vector can be written as follows:
(4) 
Clearly, this is a constrained optimization problem on weight vector . It regularizes the solution space of , reducing the possibility of overfitting on training data. As a result, GC can improve the generalization capability of trained DNN models, especially when the number of training samples is limited.
It is noted that WS [39] uses a constraint for weight optimization. It reparameterizes weights to meet this constraint. However, this constraint largely limits its practical applications because the initialized weight may not satisfy this constraint. For example, a pretrained DNN on ImageNet usually cannot meet for its initialized weight vectors. If we use WS to finetune this DNN, the advantages of pretrained models will disappear. Therefore, we have to retrain the DNN on ImageNet with WS before we finetune it. This is very cumbersome. Fortunately the weight constraint of GC in Eq. (4) fits any initialization of weight, e.g., ImageNet pretrained initialization, because it involves the initialized weight into the constraint so that is always true. This greatly extends the applications of GC.
Output feature space regularization: For SGD based algorithms, we have . It can be derived that . For any input feature vector , we have the following theorem:
Theorem 4.1: Suppose that SGD (or SGDM) with GC is used to update the weight vector , for any input feature vectors and , we have
(5) 
where is the initial weight vector and is a scalar.
Please find the proof in the Appendix. Theorem 4.1 indicates that a constant intensity change (i.e., ) of an input feature causes a change of output activation; interestingly, this change is only related to and but not the current weight vector . is the scaled mean of the initial weight vector . In particular, if the mean of is close to zero, then the output activation is not sensitive to the intensity change of input features, and the output feature space becomes more robust to training sample variations.
Indeed, the mean of is very close to zero by the commonly used weight initialization strategies, such as Xavier initialization [11], Kaiming initialization [14] and even ImageNet pretrained weight initialization. Fig. 3 shows the absolute value (log scale) of the mean of weight vectors for Conv layers in ResNet50 with Kaiming normal initialization and ImageNet pretrained weight initialization. We can see that the mean values of most weight vectors are very small and close to zero (less than ). This ensures that if we train the DNN model with GC, the output features will not be sensitive to the variation of the intensity of input features. This property regularizes the output feature space and boosts the generalization performance of DNN training.
4.2 Accelerating Training Process
Optimization landscape smoothing: It has been shown in [45, 39] that both BN and WS smooth the optimization landscape. Although BN and WS operate on activations and weights, they implicitly constrict the gradient of weights, making the gradient of weight more predictive and stable for fast training. Specifically, BN and WS use the gradient magnitude to capture the Lipschitzness of function . For the loss and its gradients, will be and , respectively, and will be . The upper bounds of and ( is the Hessian matrix of ) have been given in [45, 39] to illustrate the optimization landscape smoothing property of BN and WS. Similar conclusion can be made for our proposed GC by comparing the Lipschitzness of original loss function with the constrained loss function in Eq. (4) and the Lipschitzness of their gradients. We have the following theorem:
Theorem 4.2: Suppose is the gradient of loss function w.r.t. weight vector . With the defined in Eq.(2), we have the following conclusion for the loss function and its gradient, respectively:
(6) 
The proof of Theorem 4.2 can be found in the Appendix. Theorem 4.2 shows that for the loss function and its gradient , the constrained loss function in Eq. (4) by GC leads to a better Lipschitzness than the original loss function so that the optimization landscape becomes smoother. This means that GC has similar advantages to BN and WS on accelerating training. A good Lipschitzness on gradient implies that the gradients used in training are more predictive and wellbehaved so that the optimization landscape can be smoother for faster and more effective training.
Gradient explosion suppression: Another benefit of GC for DNN training is that GC can avoid gradient explosion and make training more stable. This property is similar to gradient clipping [36, 37, 23, 1]. Too large gradients will make the weights change abruptly during training so that the loss may severely oscillate and hard to converge. It has been shown that gradient clipping can suppress large gradient so that the training can be more stable and faster [36, 37]. There are two popular gradient clipping approaches: elementwise value clipping [36, 23] and norm clipping [37, 1], which apply thresholding to elementwise value and gradient norm to gradient matrix, respectively. In order to investigate the influence of GC on clipping gradient, in Fig. 4 we plot the max value and norm of gradient matrix of the first convolutional layer and the fullyconnected layer in ResNet50 (trained on CIFAR100) with and without GC. It can be seen that both the max value and the norm of the gradient matrix become smaller by using GC in training. This is in accordance to our conclusion in Theorem 4.2 that GC can make training process smoother and faster.
5 Experimental Results
5.1 Setup of Experiments
Extensive experiments are performed to validate the effectiveness of GC. To make the results as comprehensive and clear as possible, we arrange the experiments as follows:

We start from experiments on the MiniImageNet dataset [48] to demonstrate that GC can accelerate the DNN training process and improve the model generalization performance. We also evaluate the combinations of GC with BN and WS to show that GC can improve them for DNN optimization.

We then use the CIFAR100 dataset [26] to evaluate GC with various DNN optimizers (e.g., SGDM, Adam, Adagrad), various DNN architectures (e.g., ResNet, DenseNet, VGG), and and different hyperparameters.

We then perform experiments on ImageNet [43] to demonstrate that GC also works well on large scale image classification, and show that GC can also work well with normalization methods other than BN, such as GN.

At last, we perform experiments on the COCO dataset [31] to show that GC also works well on other tasks such as objection detection and segmentation.
GC can be applied to either Conv layer or FC layer, or both of them. In all of our following experiments, if not specified, we always apply GC to both Conv and FC layers. Except for Section 5.3
where we embed GC into different DNN optimizers for test, in all other sections we embed GC into SGDM for experiments, and the momentum is set to 0.9. All experiments are conducted under the Pytorch 1.3 framework and the GPUs are NVIDIA Tesla P100.
We would like to stress that no additional hyperparameter is introduced in our GC method. Only one line of code is needed to embed GC into the existing optimizers, while keeping all other settings remain unchanged. We compare the performances of DNN models trained with and without GC to validate the effectiveness of GC.
5.2 Results on MiniImagenet
MiniImageNet [48] is a subset of the ImageNet dataset [43], which was originally proposed for few shot learning. We use the train/test splits provided by [41, 20]. It consists of 100 classes and each class has 500 images for training and 100 images for testing. The image resolution is . We resize the images into , which is the standard ImageNet training input size. The DNN we used here is ResNet50, which is trained on 4 GPUs with batch size 128. Other settings are the same as training ImageNet. We repeat the experiments for 10 times and report the average results over the 10 runs.
BN, WS and GC operate on activations, weights and gradients, respectively, and they can be used together to train DNNs. Actually, it is necessary to normalize the feature space by methods such as BN; otherwise, the model is hard to be well trained. Therefore, we evaluate four combinations here: BN, BN+GC, BN+WS and BN+WS+GC. The optimizer is SGDM with momentum 0.9. Fig. 5 presents the training loss and testing accuracy curves of these four combinations. Compared with BN, the training loss of BN+GC decreases much faster and the testing accuracy increases more rapidly. For both BN and BN+WS, GC can further speed up their training speed. Moreover, we can see that BN+GC achieves the highest testing accuracy, validating that GC can accelerate training and enhance the generalization performance simultaneously.
5.3 Experiments on CIFAR100
Model  R18  R101  X29  V11  D121 

w/o GC  76.870.26  78.82 0.42  79.700.30  70.94 0.34  79.310.33 
w/ GC  78.820.31  80.210.31  80.530.33  71.690.37  79.680.40 
Algorithm  SGDM  Adam  Adagrad  SGDW  AdamW 

w/o GC  78.230.42  71.640.56  70.34 0.31  74.020.27  74.120.42 
w/ GC  79.140.33  72.800.62  71.580.37  76.820.29  75.070.37 
CIFAR100 [26] consists of 50K training images and 10K testing images from 100 classes. The size of input image is . Since the image resolution is small, we found that applying GC to the Conv layer is good enough on this dataset. All DNN models are trained for 200 epochs using one GPU with batch size 128. The experiments are repeated for 10 times and the results are reported in mean std format.
Different networks: We testify GC on different DNN architectures, including ResNet18 (R18), ResNet101 (R101) [15], ResNeXt29 4x64d (X29) [52], VGG11 (V11) [46] and DenseNet121 (D121) [16]. SGDM is used as the network optimizer. The weight decay is set to 0.0005. The initial learning rate is and it is multiplied by for every 60 epochs. Table 1 shows the testing accuracies of these DNNs. It can be seen that the performance of all DNNs is improved by GC, which verifies that GC is a general optimization technique for different DNN architectures.
Different optimizers: We embed GC into different DNN optimizers, including SGDM [38], Adagrad [10], Adam [24], SGDW and AdamW [32], to test their performance. SGDW and AdamW optimizers directly apply weight decay on weight, instead of using weight decay regularization. Weight decay is set to for SGDW and AdamW, and for other optimizers. The initial learning rate is set to , and for SGDM/SGDW, Adagrad, Adam/AdamW, respectively, and the learning rate is multiplied by for every 60 epochs. The other hyperparameters are set by their default settings on Pytorch. The DNN model used here is ResNet50. Table 2 shows the testing accuracies. It can be seen that GC boosts the generalization performance of all the five optimizers. It is also found that adaptive learning rate based algorithms Adagrad and Adam have poor generalization performance on CIFAR100, while GC can improve their performance by .
Different hyperparameter settings: In order to illustrate that GC can achieve consistent improvement with different hyperparameters, we present the results of GC with different settings of weight decay and learning rates on the CIFAR100 dataset. ResNet50 is used as the backbone. Table 3 shows the testing accuracies with different settings of weight decay, including 0, , , and . The optimizer is SGDM with learning rate . It can be seen that the performance of weight decay is consistently improved by GC. Table 4 shows the testing accuracies with different learning rates for SGDM and Adam. For SGDM, the learning rates are , and , and for Adam, the learning rates are , and . The weight decay is set to . Other settings are the same as those in the manuscript. We can see that GC consistently improves the performance.
Weight decay  0  

w/o GC  71.620.31  73.910.35  75.570.33  78.230.42  77.430.30 
w/ GC  72.830.29  76.560.31  77.620.37  79.140.33  78.100.36 
Algorithm  SGDM  SGDM  SGDM  Adam  Adam  Adam 

Learning rate  0.05  0.1  0.2  0.0005  0.001  0.0015 
w/o GC  76.810.27  78.230.42  76.530.32  73.880.46  71.640.56  70.630.44 
w/ GC  78.120.33  79.140.33  77.710.35  74.320.55  72.800.62  71.220.49 
Datesets  R50BN  R50GN  R101BN  R101GN 

w/o GC  23.71  24.50  22.37  23.34 
w/ GC  23.21  23.53  21.82  22.14 
5.4 Results on ImageNet
We then evaluate GC on the largescale ImageNet dataset [43] which consists of million images for training and 50K images for validation from 1000 categories. We use the common training settings and embed GC to SGDM on Conv layer. The ResNet50 and ResNet101 are used as the backbone networks. For the former, we use 4 GPUs with batch size 64 per GPU, and for the latter, we use 8 GPUs with batch size 32 per GPU.
We evaluate four models here: ResNet50 with BN (R50BN), ResNet50 with GN (R50GN), ResNet101 with BN (R101BN) and ResNet101 with GN (R101GN). Table 5 shows the final Top1 errors of these four DNN models trained with and without GC. We can see that GC can improve the performance by on ImageNet. Fig. 6 plots the training and validation error curves of ResNet50 (GN is used for feature normalization). We can see that GC can largely speed up the training with GN.
Datasets  #Category  #Training  #Testing 

FGVC Aircraft  100  6,667  3,333 
Stanford Cars  196  8,144  8,041 
Stanford Dogs  120  12,000  8,580 
CUB2002011  200  5,994  5,794 
Datesets  FGVC Aircraft  Stanford Cars  Stanford Dogs  CUB2002011 

w/o GC  86.620.31  88.660.22  76.160.25  82.070.26 
w/ GC  87.770.27  90.030.26  78.230.24  83.400.30 
5.5 Results on Finegrained Image Classification
In order to show that GC can also work with the pretrained models, we conduct experiments on four challenging finegrained image classification datasets, including FGVC Aircraft [34], Stanford Cars [25], Stanford Dogs [22] and CUB2002011 [50]. The detailed statistics of these four datasets are summarized in Table 6. We use the official pretrained ResNet50 provided by Pytorch as the baseline DNN for all these four datasets. The original images are resized into and we crop the center region with as input for both training and testing. The models are pretrained on ImageNet. We use SGDM with momentum of to finetune ResNet50 for 100 epochs on 4 GPUs with batch size 256. The initial learning rate is for the last FC layer and 0.01 for all pretrained Conv layers. The learning rate is multiplied by at the 50th and 80th epochs. Please note that our goal is to validate the effectiveness of GC but not to push stateoftheart results, so we only use simple training tricks. We repeat the experiments for 10 times and report the result in mean std format.
Fig. 7 shows the training and testing accuracies of SGDM and SGDM+GC for the first 40 epochs on the four finegrained image classification datasets. Table 7 shows the final testing accuracies. We can see that both the training and testing accuracies of SGDM are improved by GC. For the final classification accuracy, GC improves SGDM by on these four datasets. This sufficiently demonstrates the effectiveness of GC on finetuning pretrained models.
Method  Backbone  AP  Backbone  AP  

w/o GC  R50  36.4  58.4  39.1  X10132x4d  40.1  62.0  43.8 
w/ GC  R50  37.0  59.0  40.2  X10132x4d  40.7  62.7  43.9 
w/o GC  R101  38.5  60.3  41.6  X10164x4d  41.3  63.3  45.2 
w/ GC  R101  38.9  60.8  42.2  X10164x4d  41.6  63.8  45.4 
Method  Backbone  Backbone  

w/o GC  R50  37.4  59.0  40.6  34.1  55.5  36.1  R50 (4c1f)  37.5  58.2  41.0  33.9  55.0  36.1 
w/ GC  R50  37.9  59.6  41.2  34.7  56.1  37.0  R50 (4c1f)  38.4  59.5  41.8  34.6  55.9  36.7 
w/o GC  R101  39.4  60.9  43.3  35.9  57.7  38.4  R101GN  41.1  61.7  44.9  36.9  58.7  39.3 
w/ GC  R101  40.0  61.5  43.7  36.2  58.1  38.7  R101GN  41.7  62.3  45.3  37.4  59.3  40.3 
w/o GC  X10132x4d  41.1  62.8  45.0  37.1  59.4  39.8  R50GN+WS  40.0  60.7  43.6  36.1  57.8  38.6 
w/ GC  X10132x4d  41.6  63.1  45.5  37.4  59.8  39.9  R50GN+WS  40.6  61.3  43.9  36.6  58.2  39.1 
w/o GC  X10164x4d  42.1  63.8  46.3  38.0  60.6  40.9  
w/ GC  X10164x4d  42.8  64.5  46.8  38.4  61.0  41.1 
5.6 Objection Detection and Segmentation
Finally, we evaluate GC on object detection and segmentation tasks to show that GC can also be applied to more tasks beyond image classification. The models are pretrained on ImageNet. The training batch size for object detection and segmentation is usually very small (e.g., 1 or 2) because of the high resolution of input image. Therefore, the BN layer is usually frozen [15] and the benefits from BN cannot be enjoyed during training. One alternative is to use GN instead. The models are trained on COCO dataset (118K images) and evaluated on COCO dataset (40K images) [31]. COCO dataset can be used for multiple tasks, including image classification, object detection, semantic segmentation and instance segmentation.
We use the MMDetection [6] toolbox, which contains comprehensive models on object detection and segmentation tasks, as the detection framework. The official implementations and settings are used for all experiments. All the pretrained models are provided from their official websites, and we finetune them on COCO set with 8 GPUs and 2 images per GPU. The optimizers are SGDM and SGDM+GC. The backbone networks include ResNet50 (R50), ResNet101 (R101), ResNeXt10132x4d (X10132x4d), ResNeXt10164x4d (X10132x4d). The Feature Pyramid Network (FPN) [30] is also used. The learning rate schedule is for both Faster RCNN [42] and Mask RCNN [13], except R50 with GN and R101 with GN, which use learning rate schedule.
Tabel 8 shows the Average Precision (AP) results of Faster RCNN. We can see that all the backbone networks trained with GC can achieve a performance gain about on object detection. Tabel 9 presents the Average Precision for bounding box () and instance segmentation (). It can be seen that the increases by for object detection task and the increases by for instance segmentation task. Moreover, we find that if 4conv1fc bounding box head, like R50 (4c1f), is used, the performance can increase more by GC. And GC can also boost the performance of GN (see R101GN) and improve the performance of WS (see R50GN+WS). Overall, we can see that GC boosts the generalization performance of all evaluated models. This demonstrates that it is a simple yet effective optimization technique, which is general to many tasks beyond image classification.
6 Conclusions
How to efficiently and effectively optimize a DNN is one of the key issues in deep learning research. Previous methods such as batch normalization (BN) and weight standardization (WS) mostly operate on network activations or weights to improve DNN training. We proposed a different approach which operates directly on gradients. Specifically, we removed the mean from the gradient vectors and centralized them to have zero mean. The socalled Gradient Centralization (GC) method demonstrated several desired properties of deep network optimization. We showed that GC actually improves the loss function with a constraint on weight vectors, which regularizes both weight space and output feature space. We also showed that this constrained loss function has better Lipschitzness than the original one so that it has a smoother optimization landscape. Comprehensive experiments were performed and the results demonstrated that GC can be well applied to different tasks with different optimizers and network architectures, improving their training efficiency and generalization performance.
Appendix
A1. Proof of Theorem 4.1
Proof. First we show below a simple property of :
where is the dimension of .
For each SGD step with GC, we have:
It can be easily derived that:
where is the number of iterations. Then for the output activations of and , there is
(7)  
Therefore,
(8) 
For SGD with momentum, the conclusion is the same, because we can obtain a term in the third row of Eq.(7), where is the momentum in the th iteration, and this term is also equal to zero.
The proof is completed.
A2. Proof of Theorem 4.2
Proof. Because is a unit vector, there is . We can easily prove that:
(9)  
Then for , we have:
(10)  
For , we also have
(11)  
The proof is completed.
References
 [1] Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. pp. 308–318. ACM (2016)
 [2] AbuElHaija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., Vijayanarasimhan, S.: Youtube8m: A largescale video classification benchmark. arXiv preprint arXiv:1609.08675 (2016)
 [3] Bjorck, J., Gomes, C., Selman, B., Weinberger, K.Q.: Understanding batch normalization pp. 7694–7705 (2018)
 [4] Bottou, L.: Stochastic gradient learning in neural networks. Proceedings of NeuroNımes 91(8), 12 (1991)

[5]
Bottou, L.: Largescale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp. 177–186. Springer (2010)
 [6] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)
 [7] Cho, M., Lee, J.: Riemannian approach to batch normalization. In: Advances in Neural Information Processing Systems. pp. 5225–5235 (2017)
 [8] Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: Advances in neural information processing systems. pp. 3123–3131 (2015)
 [9] Courbariaux, M., Hubara, I., Soudry, D., ElYaniv, R., Bengio, Y.: Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or1. arXiv preprint arXiv:1602.02830 (2016)

[10]
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research
12(Jul), 2121–2159 (2011) 
[11]
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249–256 (2010)
 [12] Gupta, H., Jin, K.H., Nguyen, H.Q., McCann, M.T., Unser, M.: Cnnbased projected gradient descent for consistent ct image reconstruction. IEEE transactions on medical imaging 37(6), 1440–1453 (2018)

[13]
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask rcnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
 [14] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)

[15]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
 [16] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
 [17] Huang, L., Liu, X., Liu, Y., Lang, B., Tao, D.: Centered weight normalization in accelerating training of deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2803–2811 (2017)
 [18] Huang, X., Belongie, S.: Arbitrary style transfer in realtime with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1501–1510 (2017)
 [19] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

[20]
Iscen, A., Tolias, G., Avrithis, Y., Chum, O.: Label propagation for deep semisupervised learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5070–5079 (2019)

[21]
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., FeiFei, L.: Largescale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 1725–1732 (2014)
 [22] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fgvc: Stanford dogs. In: San Diego: CVPR Workshop on FGVC. vol. 1 (2011)

[23]
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image superresolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646–1654 (2016)
 [24] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
 [25] Krause, J., Stark, M., Deng, J., FeiFei, L.: 3d object representations for finegrained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 554–561 (2013)
 [26] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Tech. rep., Citeseer (2009)
 [27] Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in neural information processing systems. pp. 950–957 (1992)
 [28] Larsson, M., Arnab, A., Kahl, F., Zheng, S., Torr, P.: A projected gradient descent method for crf inference allowing endtoend training of arbitrary pairwise potentials. In: International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. pp. 564–579. Springer (2017)
 [29] Lei Ba, J., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
 [30] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
 [31] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
 [32] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
 [33] Luo, P., Wang, X., Shao, W., Peng, Z.: Towards understanding regularization in batch normalization (2018)
 [34] Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)

[35]
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML10). pp. 807–814 (2010)
 [36] Pascanu, R., Mikolov, T., Bengio, Y.: Understanding the exploding gradient problem. CoRR, abs/1211.5063 2 (2012)

[37]
Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International conference on machine learning. pp. 1310–1318 (2013)
 [38] Qian, N.: On the momentum term in gradient descent learning algorithms. Neural networks 12(1), 145–151 (1999)
 [39] Qiao, S., Wang, H., Liu, C., Shen, W., Yuille, A.: Weight standardization. arXiv preprint arXiv:1903.10520 (2019)

[40]
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnornet: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision. pp. 525–542. Springer (2016)
 [41] Ravi, S., Larochelle, H.: Optimization as a model for fewshot learning (2016)
 [42] Ren, S., He, K., Girshick, R., Sun, J.: Faster rcnn: Towards realtime object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
 [43] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211–252 (2015)
 [44] Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems. pp. 901–909 (2016)
 [45] Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? (no, it is not about internal covariate shift) pp. 2483–2493 (2018)
 [46] Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)
 [47] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
 [48] Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in neural information processing systems. pp. 3630–3638 (2016)
 [49] Vorontsov, E., Trabelsi, C., Kadoury, S., Pal, C.: On orthogonality and learning recurrent networks with long term dependencies. In: Proceedings of the 34th International Conference on Machine LearningVolume 70. pp. 3570–3578. JMLR. org (2017)
 [50] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltechucsd birds2002011 dataset (2011)
 [51] Wu, Y., He, K.: Group normalization. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 3–19 (2018)
 [52] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1492–1500 (2017)
 [53] Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 (2016)
 [54] Zhang, G., Wang, C., Xu, B., Grosse, R.: Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 (2018)
References
 [1] Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. pp. 308–318. ACM (2016)
 [2] AbuElHaija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B., Vijayanarasimhan, S.: Youtube8m: A largescale video classification benchmark. arXiv preprint arXiv:1609.08675 (2016)
 [3] Bjorck, J., Gomes, C., Selman, B., Weinberger, K.Q.: Understanding batch normalization pp. 7694–7705 (2018)
 [4] Bottou, L.: Stochastic gradient learning in neural networks. Proceedings of NeuroNımes 91(8), 12 (1991)
 [5] Bottou, L.: Largescale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp. 177–186. Springer (2010)
 [6] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)
 [7] Cho, M., Lee, J.: Riemannian approach to batch normalization. In: Advances in Neural Information Processing Systems. pp. 5225–5235 (2017)
 [8] Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: Advances in neural information processing systems. pp. 3123–3131 (2015)
 [9] Courbariaux, M., Hubara, I., Soudry, D., ElYaniv, R., Bengio, Y.: Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or1. arXiv preprint arXiv:1602.02830 (2016)
 [10] Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12(Jul), 2121–2159 (2011)
 [11] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249–256 (2010)
 [12] Gupta, H., Jin, K.H., Nguyen, H.Q., McCann, M.T., Unser, M.: Cnnbased projected gradient descent for consistent ct image reconstruction. IEEE transactions on medical imaging 37(6), 1440–1453 (2018)
 [13] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask rcnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
 [14] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)
 [15] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
 [16] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
 [17] Huang, L., Liu, X., Liu, Y., Lang, B., Tao, D.: Centered weight normalization in accelerating training of deep neural networks. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 2803–2811 (2017)
 [18] Huang, X., Belongie, S.: Arbitrary style transfer in realtime with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1501–1510 (2017)
 [19] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
 [20] Iscen, A., Tolias, G., Avrithis, Y., Chum, O.: Label propagation for deep semisupervised learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5070–5079 (2019)
 [21] Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., FeiFei, L.: Largescale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 1725–1732 (2014)
 [22] Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fgvc: Stanford dogs. In: San Diego: CVPR Workshop on FGVC. vol. 1 (2011)
 [23] Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image superresolution using very deep convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1646–1654 (2016)
 [24] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
 [25] Krause, J., Stark, M., Deng, J., FeiFei, L.: 3d object representations for finegrained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 554–561 (2013)
 [26] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Tech. rep., Citeseer (2009)
 [27] Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Advances in neural information processing systems. pp. 950–957 (1992)
 [28] Larsson, M., Arnab, A., Kahl, F., Zheng, S., Torr, P.: A projected gradient descent method for crf inference allowing endtoend training of arbitrary pairwise potentials. In: International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. pp. 564–579. Springer (2017)
 [29] Lei Ba, J., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv:1607.06450 (2016)
 [30] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2117–2125 (2017)
 [31] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
 [32] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
 [33] Luo, P., Wang, X., Shao, W., Peng, Z.: Towards understanding regularization in batch normalization (2018)
 [34] Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)
 [35] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML10). pp. 807–814 (2010)
 [36] Pascanu, R., Mikolov, T., Bengio, Y.: Understanding the exploding gradient problem. CoRR, abs/1211.5063 2 (2012)
 [37] Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International conference on machine learning. pp. 1310–1318 (2013)
 [38] Qian, N.: On the momentum term in gradient descent learning algorithms. Neural networks 12(1), 145–151 (1999)
 [39] Qiao, S., Wang, H., Liu, C., Shen, W., Yuille, A.: Weight standardization. arXiv preprint arXiv:1903.10520 (2019)
 [40] Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnornet: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision. pp. 525–542. Springer (2016)
 [41] Ravi, S., Larochelle, H.: Optimization as a model for fewshot learning (2016)
 [42] Ren, S., He, K., Girshick, R., Sun, J.: Faster rcnn: Towards realtime object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015)
 [43] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision 115(3), 211–252 (2015)
 [44] Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In: Advances in Neural Information Processing Systems. pp. 901–909 (2016)
 [45] Santurkar, S., Tsipras, D., Ilyas, A., Madry, A.: How does batch normalization help optimization? (no, it is not about internal covariate shift) pp. 2483–2493 (2018)
 [46] Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)
 [47] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
 [48] Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: Advances in neural information processing systems. pp. 3630–3638 (2016)
 [49] Vorontsov, E., Trabelsi, C., Kadoury, S., Pal, C.: On orthogonality and learning recurrent networks with long term dependencies. In: Proceedings of the 34th International Conference on Machine LearningVolume 70. pp. 3570–3578. JMLR. org (2017)
 [50] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltechucsd birds2002011 dataset (2011)
 [51] Wu, Y., He, K.: Group normalization. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 3–19 (2018)
 [52] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1492–1500 (2017)
 [53] Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 (2016)
 [54] Zhang, G., Wang, C., Xu, B., Grosse, R.: Three mechanisms of weight decay regularization. arXiv preprint arXiv:1810.12281 (2018)
Comments
There are no comments yet.