How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks

10/12/2020
by   Gaojie Jin, et al.
1

This paper studies the novel concept of weight correlation in deep neural networks and discusses its impact on the networks' generalisation ability. For fully-connected layers, the weight correlation is defined as the average cosine similarity between weight vectors of neurons, and for convolutional layers, the weight correlation is defined as the cosine similarity between filter matrices. Theoretically, we show that, weight correlation can, and should, be incorporated into the PAC Bayesian framework for the generalisation of neural networks, and the resulting generalisation bound is monotonic with respect to the weight correlation. We formulate a new complexity measure, which lifts the PAC Bayes measure with weight correlation, and experimentally confirm that it is able to rank the generalisation errors of a set of networks more precisely than existing measures. More importantly, we develop a new regulariser for training, and provide extensive experiments that show that the generalisation error can be greatly reduced with our novel approach.

READ FULL TEXT

page 3

page 6

research
12/30/2017

PAC-Bayesian Margin Bounds for Convolutional Neural Networks - Technical Report

Recently the generalisation error of deep neural networks has been analy...
research
01/22/2022

Neuronal Correlation: a Central Concept in Neural Network

This paper proposes to study neural networks through neuronal correlatio...
research
02/20/2017

Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks

Traditionally, multi-layer neural networks use dot product between the o...
research
10/25/2022

Learning Ability of Interpolating Convolutional Neural Networks

It is frequently observed that overparameterized neural networks general...
research
03/01/2021

LocalDrop: A Hybrid Regularization for Deep Neural Networks

In neural networks, developing regularization algorithms to settle overf...
research
12/07/2021

Spectral Complexity-scaled Generalization Bound of Complex-valued Neural Networks

Complex-valued neural networks (CVNNs) have been widely applied to vario...
research
11/25/2022

Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessing

Over the last decade, deep neural networks have transformed our society,...

Please sign up or login with your details

Forgot password? Click here to reset