Exemplar Normalization for Learning Deep Representation

03/19/2020 ∙ by Ruimao Zhang, et al. ∙ SenseTime Corporation The Chinese University of Hong Kong, Shenzhen 0

Normalization techniques are important in different advanced neural networks and different tasks. This work investigates a novel dynamic learning-to-normalize (L2N) problem by proposing Exemplar Normalization (EN), which is able to learn different normalization methods for different convolutional layers and image samples of a deep network. EN significantly improves flexibility of the recently proposed switchable normalization (SN), which solves a static L2N problem by linearly combining several normalizers in each normalization layer (the combination is the same for all samples). Instead of directly employing a multi-layer perceptron (MLP) to learn data-dependent parameters as conditional batch normalization (cBN) did, the internal architecture of EN is carefully designed to stabilize its optimization, leading to many appealing benefits. (1) EN enables different convolutional layers, image samples, categories, benchmarks, and tasks to use different normalization methods, shedding light on analyzing them in a holistic view. (2) EN is effective for various network architectures and tasks. (3) It could replace any normalization layers in a deep network and still produce stable model training. Extensive experiments demonstrate the effectiveness of EN in a wide spectrum of tasks including image recognition, noisy label learning, and semantic segmentation. For example, by replacing BN in the ordinary ResNet50, improvement produced by EN is 300 the noisy WebVision dataset.



There are no comments yet.


page 1

page 8

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

(a) The learning dynamic of EN ratios of four categories in three layers.
(b) Performance of EN and its counterparts on various CV tasks.
Figure 1: (a) The proposed Exemplar Normalization (EN) enables different categories to learn to select different normalizers in different layers. The four categories of ImageNet (i.e. flute, bald eagle, newfoundland and castle) in three layers (i.e

. bottom, middle and top) of ResNet50 are presented. (b) EN outperforms its counterparts on various computer vision tasks (

i.e. image classification, noisy-supervised classification and semantic image segmentation ) by using different network architectures. Zoom in three times for the best view.

Normalization techniques are one of the most essential components to improve performance and accelerate training of convolutional neural networks (CNNs). Recently, a family of normalization methods is proposed including batch normalization (BN) 

[14], instance normalization (IN) [36], layer normalization (LN) [1] and group normalization (GN) [39]. As these methods were designed for different tasks, they often normalize feature maps of CNNs from different dimensions.

To combine advantages of the above methods, switchable normalization (SN) [23] and its variant [33] were proposed to learn linear combination of normalizers for each convolutional layer in an end-to-end manner. We term this normalization setting as static ‘learning-to-normalize’. Despite the successes of these methods, once a CNN is optimized by using them, it employed the same combination ratios of the normalization methods for all image samples in a dataset, incapable to adapt to different instances and thus rendering suboptimal performance.

As shown in Fig. 1, this work studies a new learning problem, that is, dynamic ‘learning-to-normalize’, by proposing Exemplar Normalization (EN), which is able to learn arbitrary normalizer for different convolutional layers, image samples, categories, datasets, and tasks in an end-to-end way. Unlike previous conditional batch normalization (cBN) that used multi-layer perceptron (MLP) to learn data-dependent parameters in a normalization layer, suffering from over-fitting easily, the internal architecture of EN is carefully designed to learn data-dependent normalization with merely a few parameters, thus stabilizing training and improving generalization capacity of CNNs.

EN has several appealing benefits. (1) It can be treated as an explanation tool for CNNs. The exemplar-based important ratios in each EN layer provide information to analyze the properties of different samples, classes, and datasets in various tasks. As shown in Fig. 1(a), by training ResNet50 [9] on ImageNet [6], images from different categories would select different normalizers in the same EN layer, leading to superior performance compared to the ordinary network. (2) EN makes versatile design of the normalization layer possible, as EN is suitable for various benchmarks and tasks. Compared with state-of-the-art counterparts in Fig. 1(b), EN consistently outperforms them on many benchmarks such as ImageNet [6] for image classification, Webvision [18] for noisy label learning, ADE20K [42] and Cityscapes [5] for semantic segmentation. (3) EN is a plug and play module. It can be inserted into various CNN architectures such as ResNet [9], Inception v2 [35], and ShuffleNet v2 [26], to replace any normalization layer therein and boost their performance.

The contributions of this work are three-fold. (1) We present a novel normalization learning setting named dynamic ‘learning-to-normalize’, by proposing Exemplar Normalization (EN), which learns to select different normalizers in different normalization layers for different image samples. EN is able to normalize image sample in both training and testing stage. (2) EN provides a flexible way to analyze the selected normalizers in different layers, the relationship among distinct samples and their deep representations. (3) As a new building block, we apply EN to various tasks and network architectures. Extensive experiments show that EN outperforms its counterparts in wide spectrum of benchmarks and tasks. For example, by replacing BN in the ordinary ResNet50 [9], improvement produced by EN is more than that of SN on both ImageNet [6] and the noisy WebVision [18] dataset.

2 Related Work

Many normalization techniques are developed to normalize feature representations [14, 1, 36, 39, 23] or weights of filters [12, 32, 27] to accelerate training and boost generation ability of CNNs. Among them, Batch Normalization (BN) [14], Layer Normalization (LN) [1] and Instance Normalization (IN) [36] are most popular methods that compute statistics with respect to channel, layer, and minibatch respectively. The follow-up Position Normalization [17] normalizes the activations at each spatial position independently across the channels. Besides normalizing different dimensions of the feature maps, another branch of work improved the capability of BN to deal with small batch size, including Group Normalization (GN) [39], Batch Renormalization (BRN) [13], Batch Kalman Normalization (BKN) [37] and Stream Normalization (StN) [20].

In recent studies, using the hybrid of multiple normalizers in a single normalization layer has achieved much attention [29, 28, 24, 30, 25]. For example, Pan et al. introduced IBN-Net [29] to improve the generalization ability of CNNs by manually designing the mixture strategy of IN and BN. In [28], Nam et al. adopted the same scheme in style transfer, where they employed gated function to learn the important ratios of IN and BN. Luo et al. further proposed Switchable Normalization (SN) [23, 22] and its sparse version [33] to extend such a scheme to deal with arbitrary number of normalizers. More recently, Dynamic Normalization (DN) [25]

was introduced to estimate the computational pattern of statistics for the specific layer. Our work is motivated by this series of studies, but provides a more flexible way to learn normalization for each sample.

The adaptive normalization methods are also related to us. In [31], Conditional Batch Normalization (cBN) was introduced to learn parameters of BN (i.e. scale and offset) adaptively as a function of the input features. Attentive Normalization (AN) [19] learns sample-based coefficients to combine feature maps. In [21], Deecke et al. proposed Mode Normalization (MN) to detect modes of data on-the-fly and normalize them. However, these methods are incapable to learn various normalizers for different convolutional layers and images as EN did.

The proposed EN also has a connection with learning data-dependent [15] or dynamic weights [41] in convolution and pooling [16]. The subnet for computation of important ratios is also similar to SE-like [11, 2, 38] attention mechanism in form, but they are technically different. First, SE-like models encourage channels to contribute equally to the feature representation [34], while EN learns to select different normalizers in different layers. Second, SE is plugged into different networks by using different schemes. EN could directly replace other normalization layers.

3 Exemplar Normalization (EN)

3.1 Notation and Background


We introduce normalization in terms of a 4D tensor, which is the input data of a normalization layer in a mini-batch. Let

be the input 4D tensor, where indicate the number of images, number of channels, channel height and width respectively. Here and define the spatial size of a single feature map. Let matrix denote the feature maps of -th image, where . Different normalizers normalize

by removing its mean and standard deviation along different dimensions, performing a formulation


where is the feature maps after normalization. and

are the vectors of mean and standard deviation calculated by the

-th normalizer. Here we define BN, IN, LN, GN,…. The scale parameter and bias parameter are adopted to re-scale and re-shift the normalized feature maps. is a small constant to prevent dividing by zero, and both and are channel-wise operators.

Switchable Normalization (SN). Unlike previous methods that estimated statistics over different dimensions of the input tensor, SN [23, 24] learns a linear combination of statistics of existing normalizers,


where is a learnable parameter corresponding to the -th normalizer, and

. In practice, this important ratio is calculated by using the softmax function. The important ratios for mean and variance can be also different. Although SN 

[23] outperforms the individual normalizer in various tasks, it solves a static ‘learning-to-normalize’ problem by switching among several normalizers in each layer. Once SN is learned, its important ratios are fixed for the entire dataset. Thus the flexibility of SN is limited and it suffers from the bias between the training and the test set, leading to sub-optimal results.

In this paper, Exemplar Normalization (EN) is proposed to investigate a dynamic ‘learning-to-normalize’ problem, which learns different data-dependant normalizations for different image samples in each layer. EN extremely expands the flexibility of SN, while retaining SN’s advantages of differential learning, stability of model training, and capability in multiple tasks.

3.2 Formulation of EN

Given input feature maps , Exemplar Normalization (EN) is defined by


where indicates the important ratio of the -th normalizer for the -th sample. Similar with SN, we use softmax function to satisfy the summation constraint, . Compared with Eqn. (2) and Eqn. (3), the differences between SN and EN are three-fold. (1) The important ratios of mean and standard deviation in SN can be different, but such scheme is avoided in EN to ensure stability of training, because the learning capacity of EN already outperforms SN by learning different normalizers for different samples. (2) We use important ratios to combine the normalized feature maps instead of combining statistics of normalizers, reducing the bias in SN when combining the standard deviations. (3) Multiple and are adopted to re-scale and re-shift the normalized feature maps in EN.

To calculate the important ratios depended on the feature map of individual sample, we define


where , and is the total number of normalizers in EN. indicates a collection of statistics of different normalizers. We have . is a function (a small neural network) to calculate the instance-based important ratios, according to the input feature maps and statistics . denotes learnable parameters of function . We carefully design a lightweight module to implement the function in next subsection.

Figure 2: Illustration of the Exemplar Normalization (EN) layer, which is able to learn the sample-based important ratios to normalize the input feature maps by using multiple normalizers. Note that the scale parameter and shift parameter in Eqn. (3) are omitted to simplify the diagram.

3.3 An Exemplar Normalization Layer

Fig. 2 shows a diagram of the key operations in an EN layer, including important ratio calculation and feature map normalization. Given an input tensor , a set of statistics are estimated. We use to denote the -th statistics (mean and standard deviation). Then the EN layer uses and to calculate the important ratios as shown in the right branch of Fig. 2 in blue. As shown in the left branch of Fig. 2, multiple normalized tensors are also calculated.

In Fig. 2, there are three steps to calculate the important ratios for each sample. (1) The input tensor is firstly down-sampled in the spatial dimension by using average pooling. The output feature matrix is denoted as . Then we use every to pre-normalize by subtracting the means and dividing by the standard deviations. There are statistics and thus we have . After that, a 1-D convolutional operator is employed to reduce the channel dimension of from to , which is shown in the first blue block in Fig. 2. Here is a hyper-parameter that indicates the reduction rate. To further reduce the parameters in the above operation, we use group convolution with the group number to ensure the total number of convolutional parameters always equals to , irrelevant to the value of . The output in this step is denoted as .

(2) The second step is to compute the pairwise correlation of different normalizers for each sample, which is motivated by the high-order feature representation [7, 4]. For the -th sample, we use and its transposition to compute the pairwise correlations by . Then is reshaped to a vector to calculate the important ratios. Intuitively, the pairwise correlations capture the relationship between different normalizers for each sample, and allow the model to integrate more information to calculate the important ratios. In practice, we also find such operation could effectively stabilize the model training and make the model achieve higher performance.

(3) In the last step, the above vector is firstly fed into a fully-connected (FC) layer followed by a tanh unit. This is to raise its dimensions to , where is a hyper-parameter and the value of is usually small, e.g. . In practice, we set the value of as in experiments. After that, we perform another FC layer to reduce the dimension to . The output vector is regarded as the important ratios of the -th sample for normalizers, where each element is corresponding to an individual normalizer. Once we obtain the important ratio , the softmax function is applied to satisfy the summation constraint that the important ratios of different normalizers sum to .

Complexity Analysis. The numbers of parameters and computational complexity of different normalization methods are compared in Table 1. The additional parameters in EN are mainly from the convolutional and FC layers to calculate the data-dependant important ratios. In SN [23], such number is since it adopts the global important ratios for both mean and standard deviant. In EN, the total number of parameters that is applied to generate the data-dependant important ratios is , where equals to the input channel size of the convolutional layer (i.e. “Conv.” with parameters in Fig. 2). is a function of , which indicates the amount of parameters in the two FC layers (i.e. the top blue block in Fig. 2). In practice, since the number of is small (e.g. ), the value of is just about . In this paper, EN employ a pool of normalizers that is the same as SN, i.e. IN,LN,BN. Thus the computational complexities of both SN and EN for estimating the statistics are . We also compare FLOPs in Sec. 4, showing that the extra #parameters of EN is marginal compared to SN, but its relative improvement over the ordinary BN is 300% larger than SN.

Method params params computation
BN [14]
IN [36]
LN [1]
GN [39]
BKN [37]
SN [23]
Table 1: Comparisons of parameters and computational complexity of different normalizers. and indicate the scale and shift parameters in Eqn.(2), and is the parameters of “Conv.” and FC layer in proposed EN. denotes the number of normalizer and is a function of that determines the number of . are the learnable important ratios in SN [23].

4 Experiment

4.1 Image Classification with ImageNet dataset

Experiment Setting. We first examine the performance of proposed EN on ImageNet [6], a standard large-scale dataset for high-resolution image classification. Following [23], the and in all of the normalization methods are initialized as and respectively. In the training phase, the batch size is set as and the data augmentation scheme is employed same as  [9] for all of the methods. In inference, the single-crop validation accuracies based on center crop are reported.

We use ShuffleNet v2 x [26] and ResNet50 [9] as the backbone network to evaluate various normalization methods since the difference in their network architectures and the number of parameters. Same as [26], ShuffleNet v2 is trained by using Adam optimizer with the initial learning rate . For ResNet50, all of the methods are optimized by using stochastic gradient decent (SGD) with stepwise learning rate decay. The hyper-parameter in ShuffleNet v2 x and ResNet50 are set as and respectively since the smallest number of channels are different. The hyper-parameter is . For fair comparison, we replace compared normalizers with EN in all of the normalization layers in the backbone network.

Backbone Method GFLOPs Params. top-1 top-5
BN 0.046 1.37M 60.3 81.9
ShuffleNet SN 0.057 1.37M 61.2 82.9
v2 x0.5 SSN 0.052 1.37M 61.2 82.7
EN 0.063 1.59M 62.2 83.3
SENet 4.151 26.77M 77.6 93.7
AANet 4.167 25.80M 77.7 93.8
BN 4.136 25.56M 76.4 93.0
GN 4.155 25.56M 76.0 92.8
ResNet50 SN 4.225 25.56M 76.9 93.2
SSN 4.186 25.56M 77.2 93.1
EN 4.325 25.91M 78.1 93.6
Table 2: Comparisons of classification accuracies (), network parameters (Params.) and floating point operations per second (GFLOPs) of various methods on the validation set of ImageNet by using different network architectures.

Result Comparison. Table 2 reports the efficiency and accuracy of EN against its counterparts including BN [14], GN [39], SN [23] and SSN [33]. For both two backbone networks, EN offers a super-performance and a competitive computational cost compared with previous methods. For example, by considering the sample-based ratio selection, EN outperforms SN , and on top-1 accuracy by using ShuffleNet v2 x0.5 and ResNet50 with only a small amount of GFLOPs increment. The top-1 accuracy curves of ResNet50 by using BN, SN and EN on training and validation set of ImageNet are presented in Fig. 3. We also compare the performance with state-of-the-art attention-based methods, i.e. SENet [11] and AANet [2], without bells and whistles, the proposed EN still outperforms these methods.

(a) ResNet20 on CIFAR-10
(b) ResNet20 on CIFAR-10
(c) ResNet50 on ImageNet
(d) ResNet50 on ImageNet
(e) ResNet50 on Webvision
(f) ResNet50 on Webvision
Figure 3: Top-1 training and validation accuracy curves of different normalization methods on CIFAR-10, ImageNet and Webvision dataset. Zoom in three times for the best view.

4.2 Noisy Classification with Webvision dataset

Experiment Setting. We also evaluate the performance of EN on noisy image classification task with Webvision dataset [18]. We adopt Inception v2 [35] and ResNet50 [9] as the backbone network. Since the smallest number of channels in Inception v2 is

, the feature reduction rate

in the first “Conv.” is set as for such network architecture. In ResNet50 [9], we maintain the same reduction parameter as Imagenet. The center crop with the image size are adopted in inference. All of the models are optimized with SGD, where the learning rate is initialized as and decreases at the iterations of with a factor of . The batch size is set as and the data augmentation and data balance technologies are used by following [8]. In the training phase, we replace compared normalizers with EN in all of the normalization layers.

Result Comparison. Table 3 reports the top-1 and top-5 classification accuracies of various normalization methods. EN outperforms its counterparts by using both of two network architectures. Specially, by using ResNet50 as the backbone, EN significantly boost the top-1 accuracy from to compared with SN. It achieves about times relative improvement of EN against SN compared to the ordinary plain ResNet50. Such performance gain is consistent with the results on ImageNet. The training and validation curves are shown in Fig. 3.

The cross dataset test is also conducted to investigate the transfer ability of EN since the categories in ImageNet and Webvision are the same. The model trained on one dataset is used to do the test on another dataset’s validation set. The results are reported in Fig. 4 that EN still outperforms its counterparts.

Model Norm GFLOPs Params. top-1 top-5
Inception v2 BN 2.056 11.29M 70.7 88.0
SN 2.081 11.30M 71.3 88.5
EN 2.122 12.36M 71.6 88.6
ResNet50 BN 4.136 25.56M 72.5 89.1
SN 4.225 25.56M 72.8 89.2
EN 4.325 25.91M 73.5 89.4
Table 3: Comparison of classification accuracies (), network parameters and GFLOPs of various normalization methods on the validation set of Webvision by using different network architectures. The best results are bold.
training set val. set method top-1 top-5
ImageNet Webvision BN 67.9 85.8
SN 68.0 86.3
EN 68.4 86.8
Webvision ImageNet BN 64.4 84.3
SN 61.1 81.0
EN 64.7 84.6
Table 4: Top-1 and top-5 accuracy () of cross dataset results. The dataset before ’’ is adopted to train ResNet50 with various normalization methods. The validation set after ’’ is used for testing. The number of categories in two datasets are the same.

4.3 Tiny Image Classification with CIFAR dataset

Experiment Setting. We also conduct the experiment on CIFAR-10 and CIFAR-100 dataset. The training batch size is . All of the models are trained by using the single GPU. The training process contains epoches. The initial learning rate is set as and decayed at and epoch, respectively. We also adopt the warm up scheme [9, 10] for all of the models training, which increases the learning rate from to in the first epoch.

Result Comparison. The experiment results on CIFAR dataset are presented in Table 5. Compared with the previous methods, EN shows better performance than the other normalization methods over various depths of ResNet [9]. In particular, the top-1 accuracies of EN on CIFAR-100 are significantly improved by , and compared with SN with different network depths.

Dataset Backbone BN SN EN
CIFAR-10 ResNet20 91.54 91.81 92.41
ResNet56 93.15 93.41 93.73
ResNet110 93.88 94.01 94.22
CIFAR-100 ResNet20 67.87 67.74 68.78
ResNet56 70.83 70.70 72.01
ResNet110 72.41 72.53 73.32
Table 5: Top-1 accuracy () on CIFAR-10 and CIFAR-100 dataset by using various networks. The best results are bold.

4.4 Semantic Image Segmentation

Experiment Setting. We also evaluate the performance of EN on semantic segmentation task by using standard benchmarks, i.e. ADE20K [42] and Cityscapes [5] datasets, to demonstrate its generalization ability. Same as [23, 40], we use DeepLab [3] with ResNet50 as the backbone network and adopt the atrous convolution with the rate and in the last two blocks. The downsample rate of the backbone network is and the bilinear operation is employed to upsample the predicted semantic maps to the size of the input image. All of the models are trained with samples per GPU by using “ploy” learning rate decay. The initial learning rate on ADE20K and Cityscapes are set as and , respectively. Single-scale and multi-scale testing are used for evaluation. Note that the synchronization scheme is not used in SN and EN to estimate the batch mean and batch standard deviate across multi-GPU. To finetune the model on semantic segmentation, we use GPU with images per GPU to pre-train the EN-ResNet50 in ImageNet, thus we report the same configuration of SN (i.e. SN(8,32) [24]) for fair comparision.

Result Comparison. The mIoU scores on ADE20K validation set and Cityscapes test set are reported in Table 6. The performance improvement of EN is consistent with the results in classification. For example, the mIoUs on ADE20K and Cityscapes are improved from and to and by using multi-scale test.

Method ADE20K Cityscapes
mIoU mIoU mIoU mIoU
SyncBN 36.4 37.7 69.7 73.0
GN 35.7 36.6 68.4 73.1
SN 37.7 38.4 72.2 75.8
EN 38.2 38.9 72.6 76.1
Table 6: Semantic Segmentation results on ADE20K and Cityscapes datasets. The backbone is ResNet50 with dilated convolutions. The subscripts “ss” and “ms” indicate single-scale and multi-scale test respectively. The best results are bold.

4.5 Ablation Study

Hyper-parameter . We first investigate the effect of hyper-parameter in Sec. 3.3. The top-1 accuracy on ImageNet by using ResNet50 as the backbone network are reported in Table 7. All of the EN models outperform SN. With the number of increasing, the performance of classification growths steadily. The the gap between the lowest and highest is about excluding , which demonstrates the model is not sensitive to the hyper-parameter in most situations. To leverage the classification accuracy and computational efficiency, we set as in our model.

Method SN EN ( value of hyper-parameter )
1 10 20 50 100
top-1 76.9 77.1 77.5 77.8 78.1 78.0
vs. SN - 0.2 0.6 0.9 +1.2 1.1
Table 7: Top-1 accuracy () on ImageNet by using EN-ResNet50 with different ascending dimension hyper-parameter .
Method SN EN ( value of hyper-parameter )
2 4 16 32 64
top-1 76.9 77.7 77.9 77.9 78.1 77.7
vs. SN - +1.2
Table 8: Top-1 accuracy () on ImageNet by using EN-ResNet50 with different hyper-parameter in the ‘Conv.’ of Sec. 3.3. Note that the total number of parameters with different are the same.
Method top-1 / top5 top-1 / top5
vs. EN
EN-ResNet50 78.1 / 93.6 -
2-layer MLP 76.7 / 92.9 1.4 / 0.7

w/o Conv.
77.6 / 92.9 0.5 / 0.7

77.7 / 93.4 0.4 / 0.2

77.6 / 93.3 0.5 / 0.3
Table 9: Top-1 and Top-5 accuracy () on ImageNet by using EN-ResNet50 with different configurations.

Hyper-parameter . We also evaluate the different group division strategy in the first “Conv.” of Fig. 3.3 through controlling the hyper-parameter . Although the total numbers of parameters in “Conv.” layer are the same by using distinct , the reduced feature dimensions are different, leading to the different computational complexity, i.e. the larger , the smaller computation cost in the subsequent block. Table 8 shows the top-1 accuracy on ImageNet by using EN-ResNet50 with different group division in the first “Conv.” shown in Fig. 2. All of the configurations achieve higher performance than SN. With the value of growths, the performance of EN-ResNet50 increases stably expect , which equals to the smallest number of channels in ResNet50. These results indicate that feature dimension reduction benefits to the performance increment. However, such advantage may disappear if the reduction rate equals to the smallest number of channels.

Other Configurations. We replace the other components in the EN layer to verify their effectiveness. The configurations for comparison are as follows. () A 2-layer multi-layer perceptron (MLP) is used to replace the designed important ratio calculation module in Fig. 2. The MLP reduces the feature dimension to

in the first layer followed by an activation function, and then reduce the dimension to the number of important ratios in the second layer. (

) The “Conv.” operation in the Fig. 2 are omitted and pairwise correlations in Sec. 3.3 ‘step(2)’ are directly computed. () The Tanh activation function in the top blue block of Fig. 2 is replaced with ReLU. () Instead of multiple in Eqn. (3) (i.e. each is corresponding to one normalizer), single are adopted. Table 9 reports the comparisons of proposed EN with different internal configuration. According to the results, the current configuration of EN achieves the best performance compared with the other variants. It is worthy to note that we find the output of 2-layer MLP changing dramatically in the training phase (i.e. important ratios), making the distribution of feature maps at different iterations changed too much and leading to much poor accuracy.

(a) The average ratios on ImageNet validation set
(b) The average ratios on Webvision validation set
Figure 4: The averaged sample ratios in different layers of ResNet50 on ImageNet and Webvision validation set. The -axis denotes the important ratios of different normalizers after the softmax operation (i.e. sum to ). The -axis shows different residual blocks of ResNet50 and the image resolution in each block is represented as well. Different datasets learn distinct averaged ratios for different normalizers in different layers of the network.

4.6 Analysis of EN

Learning Dynamic of Ratios on Dataset. Since the parameters which are adopted to learn the important ratios in EN layer are initialized as , the important ratios of each sample in each layer have uniform values ( i.e. ) at the beginning of the model training. In the training phase, the values of changes between and . We first investigate the averaged sample ratios in different layers of ResNet50 on ImageNet and Webvision validation set. We use the optimized model to calculate the ratios of each sample in each layer, then the average ratios of each layer are calculated over all of the validation set. According to Fig. 4, once the training dataset is determined, the learned averaged ratios are usually distinct for different datasets.

To analysis the changes of ratios in the training process, Fig. 5 plots the leaning dynamic of ratios of epochs for normalization layers in ResNet50 . Each value of ratios are averaged over all of the samples in ImageNet validation set. From the perspective of the entire dataset, the changes of ratios in each layer of EN are similar to those in SN, whose values have smooth fluctuation in the training phase, implying that distinct layers may need their own preference of normalizers to optimize the model in different epochs.

Learning Dynamic of Ratios on Classes and Images. One advantage of EN compared with SN is able to learn important ratios to adaptive to different exemplars. To illustrate such benefit of EN, we further plot the averaged important ratios of different classes (i.e. w/ and w/o similar appearance) in different layers in Fig. 6, as well as the important ratios of various image samples in different layers in Fig. 7. We have the following observations.

(1) Different classes learn their own important ratios in different layers. However, once the neural network is optimized on a certain dataset (e.g. ImageNet), the trend of the ratio changes of are similar in different epochs. For example, in Fig. 6, since the Persian cat and Siamese cat have a similar appearance, their leaned ratio curves are very close and even coincident in some layers, e.g. Layer5 and Layer 10. While the ratio curves from the class of Cheeseburger are far away from the above two categories. But in most layers, the ratio changes of different normalizers are basically the same, only have the numerical nuances.

(2) For the images with the same class index but various appearances, their learned ratios could also be distinct in different layers. Such cases are shown in Fig. 7. All of the images are sampled from confectionery class but with various appearance, e.g. the exemplar of confectionery and shelves for selling candy. According to Fig. 7, different images from the same category also obtained different ratios in bottom, middle and top normalization layers.

Figure 5: The visualization of averaged sample ratios in normalization layers of EN-ResNet50 trained on ImageNet for 100 epoches. The y-axis of each sub-figure denotes the important ratios of different normalizers. The x-axis shows the different training epoches. Zoom in three times for the best view.
Figure 6: The visualization of the important ratios of 3 categories (i.e. Persian cat, Siamese cat and Cheeseburger ) in 6 different layers of ResNet50. Each column indicates one of the normalizers.
Figure 7: The visualization of the important ratios of 3 samples selected from Confectionery class in different layers of ResNet50.

5 Conclusion

In this paper, we propose Exemplar Normalization to learn the linear combination of different normalizers with a sample-based manner in a single layer. We show the effectiveness of EN on various computer vision tasks, such as classification, detection and segmentation, demonstrate its superior learning and generalization ability than static learning-to-normalize method such as SN. In addition, the interpretable visualization of learned important ratios reveals the properties of classes and datasets. The future work will explore EN in more intelligent tasks. In addition, the task-oriented constraint on the important ratios will also be a potential research direction.

Acknowledgement This work was partially supported by No. 2018YFB1800800, Open Research Fund from Shenzhen Research Institute of Big Data No. 2019ORF01005, 2018B030338001, 2017ZT07X152, ZDSYS201707251409055, HKU Seed Fund for Basic Research and Start-up Fund.