DeepAI AI Chat
Log In Sign Up

Positional Normalization

by   Boyi Li, et al.
cornell university

A widely deployed method for reducing the training time of deep neural networks is to normalize activations at each layer. Although various normalization schemes have been proposed, they all follow a common theme: normalize across spatial dimensions and discard the extracted statistics. In this paper, we propose a novel normalization method that noticeably departs from this convention. Our approach, which we refer to as Positional Normalization (PONO), normalizes exclusively across channels --- a naturally appealing dimension, which captures the first and second moments of features extracted at a particular image position. We argue that these moments convey structural information about the input image and the extracted features, which opens a new avenue along which a network can benefit from feature normalization: Instead of disregarding the PONO normalization constants, we propose to re-inject them into later layers to preserve or transfer structural information in generative networks.


page 1

page 7

page 16

page 18

page 19

page 20


Local Context Normalization: Revisiting Local Normalization

Normalization layers have been shown to improve convergence in deep neur...

Separating Controversy from Noise: Comparison and Normalization of Structural Polarization Measures

Quantifying the amount of polarization is crucial for understanding and ...

Convolutional Normalization

As the deep neural networks are being applied to complex tasks, the size...

Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Normalization techniques have only recently begun to be exploited in sup...

Pixel Normalization from Numeric Data as Input to Neural Networks

Text to image transformation for input to neural networks requires inter...

SubSpectral Normalization for Neural Audio Data Processing

Convolutional Neural Networks are widely used in various machine learnin...

On Feature Normalization and Data Augmentation

Modern neural network training relies heavily on data augmentation for i...

Code Repositories


Positional Normalization (PONO) and Moment Shortcut (MS)

view repo