Overcomplete Independent Component Analysis via SDP

01/24/2019
by   Anastasia Podosinnikova, et al.
2

We present a novel algorithm for overcomplete independent components analysis (ICA), where the number of latent sources k exceeds the dimension p of observed variables. Previous algorithms either suffer from high computational complexity or make strong assumptions about the form of the mixing matrix. Our algorithm does not make any sparsity assumption yet enjoys favorable computational and theoretical properties. Our algorithm consists of two main steps: (a) estimation of the Hessians of the cumulant generating function (as opposed to the fourth and higher order cumulants used by most algorithms) and (b) a novel semi-definite programming (SDP) relaxation for recovering a mixing component. We show that this relaxation can be efficiently solved with a projected accelerated gradient descent method, which makes the whole algorithm computationally practical. Moreover, we conjecture that the proposed program recovers a mixing component at the rate k < p^2/4 and prove that a mixing component can be recovered with high probability when k < (2 - epsilon) p log p when the original components are sampled uniformly at random on the hyper sphere. Experiments are provided on synthetic data and the CIFAR-10 dataset of real images.

READ FULL TEXT

page 7

page 20

research
05/16/2019

The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA

We consider the problem of recovering a common latent source with indepe...
research
10/31/2019

Mixing of Stochastic Accelerated Gradient Descent

We study the mixing properties for stochastic accelerated gradient desce...
research
10/27/2021

Distributed Principal Component Analysis with Limited Communication

We study efficient distributed algorithms for the fundamental problem of...
research
07/20/2019

Cramér-Rao Bounds for Complex-Valued Independent Component Extraction: Determined and Piecewise Determined Mixing Models

This paper presents Cramér-Rao Lower Bound (CRLB) for the complex-valued...
research
09/04/2019

Likelihood-Free Overcomplete ICA and Applications in Causal Discovery

Causal discovery witnessed significant progress over the past decades. I...
research
07/13/2022

Probing the Robustness of Independent Mechanism Analysis for Representation Learning

One aim of representation learning is to recover the original latent cod...
research
01/24/2019

Visualizing Topographic Independent Component Analysis with Movies

Independent component analysis (ICA) has often been used as a tool to mo...

Please sign up or login with your details

Forgot password? Click here to reset