Boundary between noise and information applied to filtering neural network weight matrices

by   Max Staats, et al.

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.


page 1

page 2

page 3

page 4


Random matrix analysis of deep neural network weight matrices

Neural networks have been used successfully in a variety of fields, whic...

Koopman-Based Bound for Generalization: New Aspect of Neural Networks Regarding Nonlinear Noise Filtering

We propose a new bound for generalization of neural networks using Koopm...

Data-driven Regularization via Racecar Training for Generalizing Neural Networks

We propose a novel training approach for improving the generalization in...

On Computation and Generalization of GANs with Spectrum Control

Generative Adversarial Networks (GANs), though powerful, is hard to trai...

Improving training of deep neural networks via Singular Value Bounding

Deep learning methods achieve great success recently on many computer vi...

Orthogonal Deep Neural Networks

In this paper, we introduce the algorithms of Orthogonal Deep Neural Net...

The SVD of Convolutional Weights: A CNN Interpretability Framework

Deep neural networks used for image classification often use convolution...

Please sign up or login with your details

Forgot password? Click here to reset