Difference in Euclidean Norm Can Cause Semantic Divergence in Batch Normalization

07/06/2022
by   Zhennan Wang, et al.
0

In this paper, we show that the difference in Euclidean norm of samples can make a contribution to the semantic divergence and even confusion, after the spatial translation and scaling transformation in batch normalization. To address this issue, we propose an intuitive but effective method to equalize the Euclidean norms of sample vectors. Concretely, we l_2-normalize each sample vector before batch normalization, and therefore the sample vectors are of the same magnitude. Since the proposed method combines the l_2 normalization and batch normalization, we name our method as L_2BN. The L_2BN can strengthen the compactness of intra-class features and enlarge the discrepancy of inter-class features. In addition, it can help the gradient converge to a stable scale. The L_2BN is easy to implement and can exert its effect without any additional parameters and hyper-parameters. Therefore, it can be used as a basic normalization method for neural networks. We evaluate the effectiveness of L_2BN through extensive experiments with various models on image classification and acoustic scene classification tasks. The experimental results demonstrate that the L_2BN is able to boost the generalization ability of various neural network models and achieve considerable performance improvements.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/09/2019

Four Things Everyone Should Know to Improve Batch Normalization

A key component of most neural network architectures is the use of norma...
research
01/01/2020

Attentive batch normalization for lstm-based acoustic modeling of speech recognition

Batch normalization (BN) is an effective method to accelerate model trai...
research
11/21/2019

Rethinking Normalization and Elimination Singularity in Neural Networks

In this paper, we study normalization methods for neural networks from t...
research
02/24/2023

On the Training Instability of Shuffling SGD with Batch Normalization

We uncover how SGD interacts with batch normalization and can exhibit un...
research
06/23/2020

Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction

Normalization has become one of the most fundamental components in many ...
research
05/29/2017

Feature Incay for Representation Regularization

Softmax loss is widely used in deep neural networks for multi-class clas...
research
12/08/2020

Efficient Semantic Image Synthesis via Class-Adaptive Normalization

Spatially-adaptive normalization (SPADE) is remarkably successful recent...

Please sign up or login with your details

Forgot password? Click here to reset