Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima

12/18/2017 ∙ by Yaodong Yu, et al. ∙ 0

We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs Õ(ϵ^-10/3) stochastic gradient evaluations to converge to an approximate local minimum x, which satisfies ∇ f(x)_2≤ϵ and λ_(∇^2 f(x))≥ -√(ϵ) in the general stochastic optimization setting, where Õ(·) hides logarithm polynomial terms and constants. This improves upon the Õ(ϵ^-7/2) gradient complexity achieved by the state-of-the-art stochastic local minima finding algorithms by a factor of Õ(ϵ^-1/6). For nonconvex finite-sum optimization, our algorithm also outperforms the best known algorithms in a certain regime.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.