Implicit Regularization in Deep Learning May Not Be Explainable by Norms

05/13/2020
by   Noam Razin, et al.
5

Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimization of norms may apply, and a standard test-bed for studying this prospect is matrix factorization (matrix completion via linear neural networks). It is an open question whether norms can explain the implicit regularization in matrix factorization. The current paper resolves this open question in the negative, by proving that there exist natural matrix factorization problems on which the implicit regularization drives all norms (and quasi-norms) towards infinity. Our results suggest that, rather than perceiving the implicit regularization via norms, a potentially more useful interpretation is minimization of rank. We demonstrate empirically that this interpretation extends to a certain class of non-linear neural networks, and hypothesize that it may be key to explaining generalization in deep learning.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/31/2019

Implicit Regularization in Deep Matrix Factorization

Efforts to understand the generalization mystery in deep learning have l...
01/24/2021

A Generalization of QR Factorization To Non-Euclidean Norms

I propose a way to use non-Euclidean norms to formulate a QR-like factor...
09/13/2021

Implicit Regularization Effects of the Sobolev Norms in Image Processing

In this paper, we propose to use the general L^2-based Sobolev norms (i....
11/28/2018

Basis Pursuit Denoise with Nonsmooth Constraints

Level-set optimization formulations with data-driven constraints minimiz...
07/14/2019

Learning Neural Networks with Adaptive Regularization

Feed-forward neural networks can be understood as a combination of an in...
01/13/2020

On implicit regularization: Morse functions and applications to matrix factorization

In this paper, we revisit implicit regularization from the ground up usi...
02/19/2021

Implicit Regularization in Tensor Factorization

Implicit regularization in deep learning is perceived as a tendency of g...

Code Repositories

imp_reg_dl_not_norms

Code for Implicit Regularization in Deep Learning May Not Be Explainable by Norms


view repo