Streaming PCA: Matching Matrix Bernstein and Near-Optimal Finite Sample Guarantees for Oja's Algorithm

02/22/2016
by   Prateek Jain, et al.
0

This work provides improved guarantees for streaming principle component analysis (PCA). Given A_1, ..., A_n∈R^d× d sampled independently from distributions satisfying E[A_i] = Σ for Σ≽0, this work provides an O(d)-space linear-time single-pass streaming algorithm for estimating the top eigenvector of Σ. The algorithm nearly matches (and in certain cases improves upon) the accuracy obtained by the standard batch method that computes top eigenvector of the empirical covariance 1/n∑_i ∈ [n] A_i as analyzed by the matrix Bernstein inequality. Moreover, to achieve constant accuracy, our algorithm improves upon the best previous known sample complexities of streaming algorithms by either a multiplicative factor of O(d) or 1/gap where gap is the relative distance between the top two eigenvalues of Σ. These results are achieved through a novel analysis of the classic Oja's algorithm, one of the oldest and most popular algorithms for streaming PCA. In particular, this work shows that simply picking a random initial point w_0 and applying the update rule w_i + 1 = w_i + η_i A_i w_i suffices to accurately estimate the top eigenvector, with a suitable choice of η_i. We believe our result sheds light on how to efficiently perform streaming PCA both in theory and in practice and we hope that our analysis may serve as the basis for analyzing many variants and extensions of streaming PCA.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/04/2023

Nearly-Linear Time and Streaming Algorithms for Outlier-Robust PCA

We study principal component analysis (PCA), where given a dataset in ℝ^...
research
02/06/2021

Streaming k-PCA: Efficient guarantees for Oja's algorithm, beyond rank-one updates

We analyze Oja's algorithm for streaming k-PCA and prove that it achieve...
research
10/31/2015

Preconditioned Data Sparsification for Big Data with Applications to PCA and K-means

We analyze a compression scheme for large data sets that randomly keeps ...
research
09/19/2017

Finite Sample Guarantees for PCA in Non-Isotropic and Data-Dependent Noise

This work obtains novel finite sample guarantees for Principal Component...
research
11/04/2019

ODE-Inspired Analysis for the Biological Version of Oja's Rule in Solving Streaming PCA

Oja's rule [Oja, Journal of mathematical biology 1982] is a well-known b...
research
05/03/2023

Streaming PCA for Markovian Data

Since its inception in Erikki Oja's seminal paper in 1982, Oja's algorit...
research
02/23/2016

An Improved Gap-Dependency Analysis of the Noisy Power Method

We consider the noisy power method algorithm, which has wide application...

Please sign up or login with your details

Forgot password? Click here to reset