Neural Methods for Point-wise Dependency Estimation

06/09/2020
by   Yao-Hung Hubert Tsai, et al.
13

Since its inception, the neural estimation of mutual information (MI) has demonstrated the empirical success of modeling expected dependency between high-dimensional random variables. However, MI is an aggregate statistic and cannot be used to measure point-wise dependency between different events. In this work, instead of estimating the expected dependency, we focus on estimating point-wise dependency (PD), which quantitatively measures how likely two outcomes co-occur. We show that we can naturally obtain PD when we are optimizing MI neural variational bounds. However, optimizing these bounds is challenging due to its large variance in practice. To address this issue, we develop two methods (free of optimizing MI variational bounds): Probabilistic Classifier and Density-Ratio Fitting. We demonstrate the effectiveness of our approaches in 1) MI estimation, 2) self-supervised representation learning, and 3) cross-modal retrieval task.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/16/2019

On Variational Bounds of Mutual Information

Estimating and optimizing Mutual Information (MI) is core to many proble...
research
02/15/2018

"Dependency Bottleneck" in Auto-encoding Architectures: an Empirical Study

Recent works investigated the generalization properties in deep neural n...
research
11/09/2020

Estimating Total Correlation with Mutual Information Bounds

Total correlation (TC) is a fundamental concept in information theory to...
research
04/07/2021

Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification

The Information Bottleneck (IB) provides an information theoretic princi...
research
11/17/2020

Reducing the Variance of Variational Estimates of Mutual Information by Limiting the Critic's Hypothesis Space to RKHS

Mutual information (MI) is an information-theoretic measure of dependenc...
research
07/19/2022

Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution

Self-supervised cross-modal super-resolution (SR) can overcome the diffi...
research
07/04/2022

Representation Learning with Information Theory for COVID-19 Detection

Successful data representation is a fundamental factor in machine learni...

Please sign up or login with your details

Forgot password? Click here to reset