Better scalability under potentially heavy-tailed gradients

06/01/2020
by   Matthew J. Holland, et al.
0

We study a scalable alternative to robust gradient descent (RGD) techniques that can be used when the gradients can be heavy-tailed, though this will be unknown to the learner. The core technique is simple: instead of trying to robustly aggregate gradients at each step, which is costly and leads to sub-optimal dimension dependence in risk bounds, we choose a candidate which does not diverge too far from the majority of cheap stochastic sub-processes run for a single pass over partitioned data. In addition to formal guarantees, we also provide empirical analysis of robustness to perturbations to experimental conditions, under both sub-Gaussian and heavy-tailed data. The result is a procedure that is simple to implement, trivial to parallelize, which keeps the formal strength of RGD methods but scales much better to large learning problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/14/2020

Better scalability under potentially heavy-tailed feedback

We study scalable alternatives to robust gradient descent (RGD) techniqu...
research
06/02/2020

Improved scalability under heavy tails, without strong convexity

Real-world data is laden with outlying values. The challenge for machine...
research
06/03/2020

Learning with CVaR-based feedback under potentially heavy tails

We study learning algorithms that seek to minimize the conditional value...
research
06/01/2017

Efficient learning with robust gradient descent

Minimizing the empirical risk is a popular training strategy, but for le...
research
10/15/2018

Robust descent using smoothed multiplicative noise

To improve the off-sample generalization of classical procedures minimiz...
research
02/20/2021

On Proximal Policy Optimization's Heavy-tailed Gradients

Modern policy gradient algorithms, notably Proximal Policy Optimization ...
research
05/14/2018

Fork and Join Queueing Networks with Heavy Tails: Scaling Dimension and Throughput Limit

Parallel and distributed computing systems are foundational to the succe...

Please sign up or login with your details

Forgot password? Click here to reset