A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces

12/08/2022
by   Charline Le Lan, et al.
0

Many machine learning problems encode their data as a matrix with a possibly very large number of rows and columns. In several applications like neuroscience, image compression or deep reinforcement learning, the principal subspace of such a matrix provides a useful, low-dimensional representation of individual data. Here, we are interested in determining the d-dimensional principal subspace of a given matrix from sample entries, i.e. from small random submatrices. Although a number of sample-based methods exist for this problem (e.g. Oja's rule <cit.>), these assume access to full columns of the matrix or particular matrix structure such as symmetry and cannot be combined as-is with neural networks <cit.>. In this paper, we derive an algorithm that learns a principal subspace from sample entries, can be applied when the approximate subspace is represented by a neural network, and hence can be scaled to datasets with an effectively infinite number of rows and columns. Our method consists in defining a loss function whose minimizer is the desired principal subspace, and constructing a gradient estimate of this loss whose bias can be controlled. We complement our theoretical analysis with a series of experiments on synthetic matrices, the MNIST dataset <cit.> and the reinforcement learning domain PuddleWorld <cit.> demonstrating the usefulness of our approach.

READ FULL TEXT

page 7

page 15

research
05/23/2023

Constructions of Constant Dimension Subspace Codes

Subspace codes have important applications in random network coding. It ...
research
09/14/2017

Subspace Clustering using Ensembles of K-Subspaces

We present a novel approach to the subspace clustering problem that leve...
research
06/07/2022

Few-Shot Learning by Dimensionality Reduction in Gradient Space

We introduce SubGD, a novel few-shot learning method which is based on t...
research
04/03/2014

Subspace Learning from Extremely Compressed Measurements

We consider learning the principal subspace of a large set of vectors fr...
research
12/12/2014

Adaptive Stochastic Gradient Descent on the Grassmannian for Robust Low-Rank Subspace Recovery and Clustering

In this paper, we present GASG21 (Grassmannian Adaptive Stochastic Gradi...
research
11/17/2021

Evaluations of The Hierarchical Subspace Iteration Method

This document contains additional experiments concerned with the evaluat...
research
03/02/2023

Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distance

Differentially private stochastic gradient descent privatizes model trai...

Please sign up or login with your details

Forgot password? Click here to reset