Diffusion Approximations for Online Principal Component Estimation and Global Convergence

08/29/2018
by   Chris Junchi Li, et al.
4

In this paper, we propose to adopt the diffusion approximation tools to study the dynamics of Oja's iteration which is an online stochastic gradient descent method for the principal component analysis. Oja's iteration maintains a running estimate of the true principal component from streaming data and enjoys less temporal and spatial complexities. We show that the Oja's iteration for the top eigenvector generates a continuous-state discrete-time Markov chain over the unit sphere. We characterize the Oja's iteration in three phases using diffusion approximation and weak convergence tools. Our three-phase analysis further provides a finite-sample error bound for the running estimate, which matches the minimax information lower bound for principal component analysis under the additional assumption of bounded samples.

READ FULL TEXT
research
03/16/2016

Near-Optimal Stochastic Approximation for Online Principal Component Estimation

Principal component analysis (PCA) has been a prominent tool for high-di...
research
01/03/2023

Diffusion approximations of Oja's online principal component analysis

Oja's algorithm of principal component analysis (PCA) has been one of th...
research
03/18/2022

Generative Principal Component Analysis

In this paper, we study the problem of principal component analysis with...
research
12/28/2020

Stochastic Approximation for Online Tensorial Independent Component Analysis

Independent component analysis (ICA) has been a popular dimension reduct...
research
05/15/2020

Non-Sparse PCA in High Dimensions via Cone Projected Power Iteration

In this paper, we propose a cone projected power iteration algorithm to ...
research
01/04/2020

Distributed Stochastic Algorithms for High-rate Streaming Principal Component Analysis

This paper considers the problem of estimating the principal eigenvector...
research
09/21/2014

Approximation errors of online sparsification criteria

Many machine learning frameworks, such as resource-allocating networks, ...

Please sign up or login with your details

Forgot password? Click here to reset