To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding

03/28/2022
by   Edouard Yvinec, et al.
0

Batch-Normalization (BN) layers have become fundamental components in the evermore complex deep neural network architectures. Such models require acceleration processes for deployment on edge devices. However, BN layers add computation bottlenecks due to the sequential operation processing: thus, a key, yet often overlooked component of the acceleration process is BN layers folding. In this paper, we demonstrate that the current BN folding approaches are suboptimal in terms of how many layers can be removed. We therefore provide a necessary and sufficient condition for BN folding and a corresponding optimal algorithm. The proposed approach systematically outperforms existing baselines and allows to dramatically reduce the inference time of deep neural networks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/09/2019

Four Things Everyone Should Know to Improve Batch Normalization

A key component of most neural network architectures is the use of norma...
research
12/07/2018

On Batch Orthogonalization Layers

Batch normalization has become ubiquitous in many state-of-the-art nets....
research
04/06/2020

Evolving Normalization-Activation Layers

Normalization layers and activation functions are critical components in...
research
02/18/2018

Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Deep neural networks have led to a series of breakthroughs, dramatically...
research
06/07/2019

The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks

Normalization methods play an important role in enhancing the performanc...
research
06/08/2023

Normalization-Equivariant Neural Networks with Application to Image Denoising

In many information processing systems, it may be desirable to ensure th...
research
11/21/2015

GradNets: Dynamic Interpolation Between Neural Architectures

In machine learning, there is a fundamental trade-off between ease of op...

Please sign up or login with your details

Forgot password? Click here to reset