Robust Regression via Model Based Methods

06/20/2021
by   Armin Moharrer, et al.
0

The mean squared error loss is widely used in many applications, including auto-encoders, multi-target regression, and matrix factorization, to name a few. Despite computational advantages due to its differentiability, it is not robust to outliers. In contrast, l_p norms are known to be robust, but cannot be optimized via, e.g., stochastic gradient descent, as they are non-differentiable. We propose an algorithm inspired by so-called model-based optimization (MBO) [35, 36], which replaces a non-convex objective with a convex model function and alternates between optimizing the model function and updating the solution. We apply this to robust regression, proposing SADM, a stochastic variant of the Online Alternating Direction Method of Multipliers (OADM) [50] to solve the inner optimization in MBO. We show that SADM converges with the rate O(log T/T). Finally, we demonstrate experimentally (a) the robustness of l_p norms to outliers and (b) the efficiency of our proposed model-based algorithms in comparison with gradient methods on autoencoders and multi-target regression.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/01/2020

Universal Robust Regression via Maximum Mean Discrepancy

Many datasets are collected automatically, and are thus easily contamina...
research
03/07/2018

Sever: A Robust Meta-Algorithm for Stochastic Optimization

In high dimensions, most machine learning methods are brittle to even a ...
research
10/26/2017

Duality-free Methods for Stochastic Composition Optimization

We consider the composition optimization with two expected-value functio...
research
04/10/2016

Online Nonnegative Matrix Factorization with Outliers

We propose a unified and systematic framework for performing online nonn...
research
11/21/2019

Communication-Efficient and Byzantine-Robust Distributed Learning

We develop a communication-efficient distributed learning algorithm that...
research
07/01/2020

Online Robust Regression via SGD on the l1 loss

We consider the robust linear regression problem in the online setting w...
research
06/23/2019

Robust convex clustering: How does fusion penalty enhance robustness?

Convex clustering has gained popularity recently due to its desirable pe...

Please sign up or login with your details

Forgot password? Click here to reset