AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax Optimization
share
In the paper, we propose a class of faster adaptive gradient descent ascent methods for solving the nonconvex-strongly-concave minimax problems by using unified adaptive matrices used in the SUPER-ADAM <cit.>. Specifically, we propose a fast adaptive gradient decent ascent (AdaGDA) method based on the basic momentum technique, which reaches a low sample complexity of O(κ^4ϵ^-4) for finding an ϵ-stationary point without large batches, which improves the existing result of adaptive minimax optimization method by a factor of O(√(κ)). Moreover, we present an accelerated version of AdaGDA (VR-AdaGDA) method based on the momentum-based variance reduced technique, which achieves the best known sample complexity of O(κ^3ϵ^-3) for finding an ϵ-stationary point without large batches. Further assume the bounded Lipschitz parameter of objective function, we prove that our VR-AdaGDA method reaches a lower sample complexity of O(κ^2.5ϵ^-3) with the mini-batch size O(κ). In particular, we provide an effective convergence analysis framework for our adaptive methods based on unified adaptive matrices, which include almost existing adaptive learning rates.
READ FULL TEXT