Ridge Regression with Frequent Directions: Statistical and Optimization Perspectives

11/06/2020
by   Charlie Dickens, et al.
0

Despite its impressive theory & practical performance, Frequent Directions (fd) has not been widely adopted for large-scale regression tasks. Prior work has shown randomized sketches (i) perform worse in estimating the covariance matrix of the data than fd; (ii) incur high error when estimating the bias and/or variance on sketched ridge regression. We give the first constant factor relative error bounds on the bias & variance for sketched ridge regression using fd. We complement these statistical results by showing that fd can be used in the optimization setting through an iterative scheme which yields high-accuracy solutions. This improves on randomized approaches which need to compromise the need for a new sketch every iteration with speed of convergence. In both settings, we also show using Robust Frequent Directions further enhances performance.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset