Algorithms for Heavy-Tailed Statistics: Regression, Covariance Estimation, and Beyond

12/23/2019
by   Yeshwanth Cherapanamjeri, et al.
0

We study efficient algorithms for linear regression and covariance estimation in the absence of Gaussian assumptions on the underlying distributions of samples, making assumptions instead about only finitely-many moments. We focus on how many samples are needed to do estimation and regression with high accuracy and exponentially-good success probability. For covariance estimation, linear regression, and several other problems, estimators have recently been constructed with sample complexities and rates of error matching what is possible when the underlying distribution is Gaussian, but algorithms for these estimators require exponential time. We narrow the gap between the Gaussian and heavy-tailed settings for polynomial-time estimators with: 1. A polynomial-time estimator which takes n samples from a random vector X ∈ R^d with covariance Σ and produces Σ̂ such that in spectral norm Σ̂ - Σ_2 ≤Õ(d^3/4/√(n)) w.p. 1-2^-d. The information-theoretically optimal error bound is Õ(√(d/n)); previous approaches to polynomial-time algorithms were stuck at Õ(d/√(n)). 2. A polynomial-time algorithm which takes n samples (X_i,Y_i) where Y_i = 〈 u,X_i 〉 + ε_i and produces û such that the loss u - û^2 ≤ O(d/n) w.p. 1-2^-d for any n ≥ d^3/2log(d)^O(1). This (information-theoretically optimal) error is achieved by inefficient algorithms for any n ≫ d; previous polynomial-time algorithms suffer loss Ω(d^2/n) and require n ≫ d^2. Our algorithms use degree-8 sum-of-squares semidefinite programs. We offer preliminary evidence that improving these rates of error in polynomial time is not possible in the median of means framework our algorithms employ.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/19/2018

Mean Estimation with Sub-Gaussian Rates in Polynomial Time

We study polynomial time algorithms for estimating the mean of a heavy-t...
research
07/12/2020

A spectral algorithm for robust regression with subgaussian rates

We study a new linear up to quadratic time algorithm for linear regressi...
research
09/19/2018

Sub-Gaussian Mean Estimation in Polynomial Time

We study polynomial time algorithms for estimating the mean of a random ...
research
02/06/2019

Fast Mean Estimation with Sub-Gaussian Rates

We propose an estimator for the mean of a random vector in R^d that can ...
research
03/19/2019

How Hard Is Robust Mean Estimation?

Robust mean estimation is the problem of estimating the mean μ∈R^d of a ...
research
08/13/2019

A Fast Spectral Algorithm for Mean Estimation with Sub-Gaussian Rates

We study the algorithmic problem of estimating the mean of heavy-tailed ...
research
09/10/2017

Complete Classification of Generalized Santha-Vazirani Sources

Let F be a finite alphabet and D be a finite set of distributions over F...

Please sign up or login with your details

Forgot password? Click here to reset