Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima

12/18/2017

∙

by   Yaodong Yu, et al.

∙

0

∙

share

We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs Õ(ϵ^-10/3) stochastic gradient evaluations to converge to an approximate local minimum x, which satisfies ∇ f(x)_2≤ϵ and λ_(∇^2 f(x))≥ -√(ϵ) in the general stochastic optimization setting, where Õ(·) hides logarithm polynomial terms and constants. This improves upon the Õ(ϵ^-7/2) gradient complexity achieved by the state-of-the-art stochastic local minima finding algorithms by a factor of Õ(ϵ^-1/6). For nonconvex finite-sum optimization, our algorithm also outperforms the best known algorithms in a certain regime.