Non-convex Distributionally Robust Optimization: Non-asymptotic Analysis

10/24/2021
by   Jikai Jin, et al.
0

Distributionally robust optimization (DRO) is a widely-used approach to learn models that are robust against distribution shift. Compared with the standard optimization setting, the objective function in DRO is more difficult to optimize, and most of the existing theoretical results make strong assumptions on the loss function. In this work we bridge the gap by studying DRO algorithms for general smooth non-convex losses. By carefully exploiting the specific form of the DRO objective, we are able to provide non-asymptotic convergence guarantees even though the objective function is possibly non-convex, non-smooth and has unbounded gradient noise. In particular, we prove that a special algorithm called the mini-batch normalized gradient descent with momentum, can find an ϵ first-order stationary point within O( ϵ^-4 ) gradient complexity. We also discuss the conditional value-at-risk (CVaR) setting, where we propose a penalized DRO objective based on a smoothed version of the CVaR that allows us to obtain a similar convergence guarantee. We finally verify our theoretical results in a number of tasks and find that the proposed algorithm can consistently achieve prominent acceleration.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/30/2022

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

In this paper, we present a stochastic gradient algorithm for minimizing...
research
08/17/2016

Mollifying Networks

The optimization of deep neural networks can be more challenging than tr...
research
07/18/2018

Convergence guarantees for RMSProp and ADAM in non-convex optimization and their comparison to Nesterov acceleration on autoencoders

RMSProp and ADAM continue to be extremely popular algorithms for trainin...
research
10/12/2020

Large-Scale Methods for Distributionally Robust Optimization

We propose and analyze algorithms for distributionally robust optimizati...
research
09/29/2017

Optimisation of photometric stereo methods by non-convex variational minimisation

Estimating shape and appearance of a three dimensional object from a giv...
research
02/02/2019

Non-asymptotic Analysis of Biased Stochastic Approximation Scheme

Stochastic approximation (SA) is a key method used in statistical learni...
research
10/14/2020

Alternating Minimization Based First-Order Method for the Wireless Sensor Network Localization Problem

We propose an algorithm for the Wireless Sensor Network localization pro...

Please sign up or login with your details

Forgot password? Click here to reset