Algorithmic Regularization in Over-parameterized Matrix Recovery

12/26/2017

∙

by   Yuanzhi Li, et al.

∙

0

∙

share

We study the problem of recovering a low-rank matrix X^ from linear measurements using an over-parameterized model. We parameterize the rank-r matrix X^ by UU^ where U∈R^d× d is a square matrix, whereas the number of linear measurements is much less than d^2. We show that with Õ(dr^2) random linear measurements, the gradient descent on the squared loss, starting from a small initialization, recovers X^ approximately in Õ(√(r)) iterations. The results solve the conjecture of Gunasekar et al. under the restricted isometry property, and demonstrate that the training algorithm can provide an implicit regularization for non-linear matrix factorization models.