An Information-Theoretic Framework for Non-linear Canonical Correlation Analysis

10/31/2018
by   Amichai Painsky, et al.
14

Canonical Correlation Analysis (CCA) is a linear representation learning method that seeks maximally correlated variables in multi-view data. Non-linear CCA extends this notion to a broader family of transformations, which are more powerful for many real-world applications. Given the joint probability, the Alternating Conditional Expectation (ACE) provides an optimal solution to the non-linear CCA problem. However, it suffers from limited performance and an increasing computational burden when only a finite number of observations is available. In this work we introduce an information-theoretic framework for the non-linear CCA problem (ITCCA), which extends the classical ACE approach. Our suggested framework seeks compressed representations of the data that allow a maximal level of correlation. This way we control the trade-off between the flexibility and the complexity of the representation. Our approach demonstrates favorable performance at a reduced computational burden, compared to non-linear alternatives, in a finite sample size regime. Further, ITCCA provides theoretical bounds and optimality conditions, as we establish fundamental connections to rate-distortion theory, the information bottleneck and remote source coding. In addition, it implies a "soft" dimensionality reduction, as the compression level is measured (and governed) by the mutual information between the original noisy data and the signals that we extract.

READ FULL TEXT

page 17

page 19

research
11/07/2017

Gaussian Lower Bound for the Information Bottleneck Limit

The Information Bottleneck (IB) is a conceptual method for extracting th...
research
11/07/2017

A Tutorial on Canonical Correlation Methods

Canonical correlation analysis is a family of multivariate statistical m...
research
10/12/2020

Deep Gated Canonical Correlation Analysis

Canonical Correlation Analysis (CCA) models can extract informative corr...
research
06/20/2022

Variational Distillation for Multi-View Learning

Information Bottleneck (IB) based multi-view learning provides an inform...
research
02/09/2022

Non-Linear Spectral Dimensionality Reduction Under Uncertainty

In this paper, we consider the problem of non-linear dimensionality redu...
research
05/04/2020

The Bussgang Decomposition of Non-Linear Systems: Basic Theory and MIMO Extensions

Many of the systems that appear in various signal processing application...
research
03/31/2023

Generalized Information Bottleneck for Gaussian Variables

The information bottleneck (IB) method offers an attractive framework fo...

Please sign up or login with your details

Forgot password? Click here to reset