Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks

05/05/2020
by   Zhishuai Guo, et al.
0

In this paper, we study distributed algorithms for large-scale AUC maximization with a deep neural network as a predictive model. Although distributed learning techniques have been investigated extensively in deep learning, they are not directly applicable to stochastic AUC maximization with deep neural networks due to its striking differences from standard loss minimization problems (e.g., cross-entropy). Towards addressing this challenge, we propose and analyze a communication-efficient distributed optimization algorithm based on a non-convex concave reformulation of the AUC maximization, in which the communication of both the primal variable and the dual variable between each worker and the parameter server only occurs after multiple steps of gradient-based updates in each worker. Compared with the naive parallel version of an existing algorithm that computes stochastic gradients at individual machines and averages them for updating the model parameter, our algorithm requires a much less number of communication rounds and still achieves a linear speedup in theory. To the best of our knowledge, this is the first work that solves the non-convex concave min-max problem for AUC maximization with deep neural networks in a communication-efficient distributed manner while still maintaining the linear speedup property in theory. Our experiments on several benchmark datasets show the effectiveness of our algorithm and also confirm our theory.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/28/2019

Stochastic AUC Maximization with Deep Neural Networks

Stochastic AUC maximization has garnered an increasing interest due to b...
research
02/09/2021

Federated Deep AUC Maximization for Heterogeneous Data with a Constant Communication Complexity

eep UC (area under the ROC curve) aximization (DAM) has attracted much a...
research
06/12/2020

Fast Objective and Duality Gap Convergence for Non-convex Strongly-concave Min-max Problems

This paper focuses on stochastic methods for solving smooth non-convex s...
research
11/10/2020

Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems

While many distributed optimization algorithms have been proposed for so...
research
03/01/2022

When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee

In this paper, we propose systematic and efficient gradient-based method...
research
11/01/2021

Deep AUC Maximization for Medical Image Classification: Challenges and Opportunities

In this extended abstract, we will present and discuss opportunities and...
research
05/10/2022

A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks

In distributed training of deep neural networks or Federated Learning (F...

Please sign up or login with your details

Forgot password? Click here to reset