Mean Spectral Normalization of Deep Neural Networks for Embedded Automation

Deep Neural Networks (DNNs) have begun to thrive in the field of automation systems, owing to the recent advancements in standardising various aspects such as architecture, optimization techniques, and regularization. In this paper, we take a step towards a better understanding of Spectral Normalization (SN) and its potential for standardizing regularization of a wider range of Deep Learning models, following an empirical approach. We conduct several experiments to study their training dynamics, in comparison with the ubiquitous Batch Normalization (BN) and show that SN increases the gradient sparsity and controls the gradient variance. Furthermore, we show that SN suffers from a phenomenon, we call the mean-drift effect, which mitigates its performance. We, then, propose a weight reparameterization called as the Mean Spectral Normalization (MSN) to resolve the mean drift, thereby significantly improving the network's performance. Our model performs 16 practice, and has fewer trainable parameters. We also show the performance of our MSN for small, medium, and large CNNs - 3-layer CNN, VGG7 and DenseNet-BC, respectively - and unsupervised image generation tasks using Generative Adversarial Networks (GANs) to evaluate its applicability for a broad range of embedded automation tasks.

READ FULL TEXT

page 1

page 4

page 6

research
02/16/2018

Spectral Normalization for Generative Adversarial Networks

One of the challenges in the study of generative adversarial networks is...
research
08/19/2020

Regularization And Normalization For Generative Adversarial Networks: A Review

Generative adversarial networks(GANs) is a popular generative model. Wit...
research
09/06/2020

Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

Spectral normalization (SN) is a widely-used technique for improving the...
research
03/22/2023

An Empirical Analysis of the Shift and Scale Parameters in BatchNorm

Batch Normalization (BatchNorm) is a technique that improves the trainin...
research
01/30/2020

How Does BN Increase Collapsed Neural Network Filters?

Improving sparsity of deep neural networks (DNNs) is essential for netwo...
research
03/22/2021

Fast Approximate Spectral Normalization for Robust Deep Neural Networks

Deep neural networks (DNNs) play an important role in machine learning d...
research
04/02/2020

Controllable Orthogonalization in Training DNNs

Orthogonality is widely used for training deep neural networks (DNNs) du...

Please sign up or login with your details

Forgot password? Click here to reset