Implicit Regularization in Deep Learning: A View from Function Space

by   Aristide Baratin, et al.

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a possible regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. By extrapolating a new analysis of Rademacher complexity bounds in linear models, we propose and study a new heuristic complexity measure for neural networks which captures this phenomenon, in terms of sequences of tangent kernel classes along in the learning trajectories.


page 7

page 8

page 9

page 21

page 23


Limitation of characterizing implicit regularization by data-independent functions

In recent years, understanding the implicit regularization of neural net...

The Equilibrium Hypothesis: Rethinking implicit regularization in Deep Neural Networks

Modern Deep Neural Networks (DNNs) exhibit impressive generalization pro...

Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

We theoretically analyze the Feedback Alignment (FA) algorithm, an effic...

On implicit regularization: Morse functions and applications to matrix factorization

In this paper, we revisit implicit regularization from the ground up usi...

Implicit Regularization with Polynomial Growth in Deep Tensor Factorization

We study the implicit regularization effects of deep learning in tensor ...

On the training dynamics of deep networks with L_2 regularization

We study the role of L_2 regularization in deep learning, and uncover si...

On Generalization and Regularization in Deep Learning

Why do large neural network generalize so well on complex tasks such as ...

Code Repositories