A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem

07/25/2019
by   Pei-Chang Guo, et al.
3

Convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacobian of a layer are bounded around 1 in the training process. We propose a new penalty function for a convolutional kernel to let the singular values of the corresponding transformation matrix are bounded around 1. We show how to carry out the gradient type methods. The penalty is about the structured transformation matrix corresponding to a convolutional kernel. This provides a new regularization method about the weights of convolutional layers.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/12/2019

On regularization for a convolutional kernel in neural networks

Convolutional neural network is a very important model of deep learning....
research
02/05/2021

Regularization for convolutional kernel tensors to avoid unstable gradient problem in convolutional neural networks

Convolutional neural networks are very popular nowadays. Training neural...
research
05/26/2018

The Singular Values of Convolutional Layers

We characterize the singular values of the linear transformation associa...
research
06/12/2020

Asymptotic Singular Value Distribution of Linear Convolutional Layers

In convolutional neural networks, the linear transformation of multi-cha...
research
08/12/2021

Existence, Stability And Scalability Of Orthogonal Convolutional Neural Networks

Imposing orthogonal transformations between layers of a neural network h...
research
07/25/2019

Filter Bank Regularization of Convolutional Neural Networks

Regularization techniques are widely used to improve the generality, rob...
research
11/21/2019

Approximated Orthonormal Normalisation in Training Neural Networks

Generalisation of a deep neural network (DNN) is one major concern when ...

Please sign up or login with your details

Forgot password? Click here to reset