Efficient Maximal Coding Rate Reduction by Variational Forms

03/31/2022
by   Christina Baek, et al.
0

The principle of Maximal Coding Rate Reduction (MCR^2) has recently been proposed as a training objective for learning discriminative low-dimensional structures intrinsic to high-dimensional data to allow for more robust training than standard approaches, such as cross-entropy minimization. However, despite the advantages that have been shown for MCR^2 training, MCR^2 suffers from a significant computational cost due to the need to evaluate and differentiate a significant number of log-determinant terms that grows linearly with the number of classes. By taking advantage of variational forms of spectral functions of a matrix, we reformulate the MCR^2 objective to a form that can scale significantly without compromising training accuracy. Experiments in image classification demonstrate that our proposed formulation results in a significant speed up over optimizing the original MCR^2 objective directly and often results in higher quality learned representations. Further, our approach may be of independent interest in other models that require computation of log-determinant forms, such as in system identification or normalizing flow models.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/15/2020

Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

To learn intrinsic low-dimensional structures from high-dimensional data...
research
10/01/2022

Federated Representation Learning via Maximal Coding Rate Reduction

We propose a federated methodology to learn low-dimensional representati...
research
04/12/2020

Variational Autoencoders with Normalizing Flow Decoders

Recently proposed normalizing flow models such as Glow have been shown t...
research
01/24/2016

Fast Binary Embedding via Circulant Downsampled Matrix -- A Data-Independent Approach

Binary embedding of high-dimensional data aims to produce low-dimensiona...
research
05/07/2022

Variational Sparse Coding with Learned Thresholding

Sparse coding strategies have been lauded for their parsimonious represe...
research
10/26/2020

A novel variational form of the Schatten-p quasi-norm

The Schatten-p quasi-norm with p∈(0,1) has recently gained considerable ...

Please sign up or login with your details

Forgot password? Click here to reset