Fiedler Regularization: Learning Neural Networks with Graph Sparsity

03/02/2020
by   Edric Tam, et al.
0

We introduce a novel regularization approach for deep learning that incorporates and respects the underlying graphical structure of the neural network. Existing regularization methods often focus on dropping/penalizing weights in a global manner that ignores the connectivity structure of the neural network. We propose to use the Fiedler value of the neural network's underlying graph as a tool for regularization. We provide theoretical support for this approach via spectral graph theory. We demonstrate the convexity of this penalty and provide an approximate, variational approach for fast computation in practical training of neural networks. We provide bounds on such approximations. We provide an alternative but equivalent formulation of this framework in the form of a structurally weighted L1 penalty, thus linking our approach to sparsity induction. We performed experiments on datasets that compare Fiedler regularization with traditional regularization methods such as dropout and weight decay. Results demonstrate the efficacy of Fiedler regularization.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/06/2023

Spectral Gap Regularization of Neural Networks

We introduce Fiedler regularization, a novel approach for regularizing n...
research
03/25/2020

Volumization as a Natural Generalization of Weight Decay

We propose a novel regularization method, called volumization, for neura...
research
01/23/2023

Deep Learning Meets Sparse Regularization: A Signal Processing Perspective

Deep learning has been wildly successful in practice and most state-of-t...
research
04/21/2018

Bridgeout: stochastic bridge regularization for deep neural networks

A major challenge in training deep neural networks is overfitting, i.e. ...
research
06/14/2021

The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization

Among the most successful methods for sparsifying deep (neural) networks...
research
12/29/2014

Improving approximate RPCA with a k-sparsity prior

A process centric view of robust PCA (RPCA) allows its fast approximate ...
research
12/05/2021

Training Structured Neural Networks Through Manifold Identification and Variance Reduction

This paper proposes an algorithm (RMDA) for training neural networks (NN...

Please sign up or login with your details

Forgot password? Click here to reset