Implicit Regularization in Matrix Factorization

05/25/2017

∙

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix X with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.

READ FULL TEXT

Implicit Regularization in Matrix Factorization

Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning

Deep Matrix Factorization with Spectral Geometric Regularization

Implicit Regularization in Deep Tensor Factorization

Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization

Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors

On the Implicit Bias of Adam

Implicit Regularization in Matrix Factorization

Related Research

Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning

Deep Matrix Factorization with Spectral Geometric Regularization

Implicit Regularization in Deep Tensor Factorization

Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations

Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorization

Linear Convergence and Implicit Regularization of Generalized Mirror Descent with Time-Dependent Mirrors

On the Implicit Bias of Adam