AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization

06/30/2021 ∙ by Feihu Huang, et al. ∙ 0

In the paper, we propose a class of faster adaptive gradient descent ascent methods for solving the nonconvex-strongly-concave minimax problems by using unified adaptive matrices used in the SUPER-ADAM <cit.>. Specifically, we propose a fast adaptive gradient decent ascent (AdaGDA) method based on the basic momentum technique, which reaches a low sample complexity of O(κ^4ϵ^-4) for finding an ϵ-stationary point without large batches, which improves the existing result of adaptive minimax optimization method by a factor of O(√(κ)). Moreover, we present an accelerated version of AdaGDA (VR-AdaGDA) method based on the momentum-based variance reduced technique, which achieves the best known sample complexity of O(κ^3ϵ^-3) for finding an ϵ-stationary point without large batches. Further assume the bounded Lipschitz parameter of objective function, we prove that our VR-AdaGDA method reaches a lower sample complexity of O(κ^2.5ϵ^-3) with the mini-batch size O(κ). In particular, we provide an effective convergence analysis framework for our adaptive methods based on unified adaptive matrices, which include almost existing adaptive learning rates.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.