-
Algorithmic Regularization in Over-parameterized Matrix Recovery
We study the problem of recovering a low-rank matrix X^ from linear meas...
read it
-
Implicit Regularization in Matrix Factorization
We study implicit regularization when optimizing an underdetermined quad...
read it
-
Learning One-hidden-layer neural networks via Provable Gradient Descent with Random Initialization
Although deep learning has shown its powerful performance in many applic...
read it
-
Implicit Regularization in ReLU Networks with the Square Loss
Understanding the implicit regularization (or implicit bias) of gradient...
read it
-
On the Convex Behavior of Deep Neural Networks in Relation to the Layers' Width
The Hessian of neural networks can be decomposed into a sum of two matri...
read it
-
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks
We analyze algorithms for approximating a function f(x) = Φ x mapping ^d...
read it
-
Model Repair: Robust Recovery of Over-Parameterized Statistical Models
A new type of robust estimation problem is introduced where the goal is ...
read it
Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations
We show that the (stochastic) gradient descent algorithm provides an implicit regularization effect in the learning of over-parameterized matrix factorization models and one-hidden-layer neural networks with quadratic activations. Concretely, we show that given Õ(dr^2) random linear measurements of a rank r positive semidefinite matrix X^, we can recover X^ by parameterizing it by UU^ with U∈R^d× d and minimizing the squared loss, even if r ≪ d. We prove that starting from a small initialization, gradient descent recovers X^ in Õ(√(r)) iterations approximately. The results solve the conjecture of Gunasekar et al.'17 under the restricted isometry property. The technique can be applied to analyzing neural networks with quadratic activations with some technical modifications.
READ FULL TEXT
Comments
There are no comments yet.