The Compound Information Bottleneck Outlook

by   Michael Dikshtein, et al.

We formulate and analyze the compound information bottleneck programming. In this problem, a Markov chain 𝖷→𝖸→𝖹 is assumed with fixed marginal distributions 𝖯_𝖷 and 𝖯_𝖸, and the mutual information between 𝖷 and 𝖹 is sought to be maximized over the choice of conditional probability of 𝖹 given 𝖸 from a given class, under the worst choice of the joint probability of the pair (𝖷,𝖸) from a different class. We consider several classes based on extremes of: mutual information; minimal correlation; total variation; and the relative entropy class. We provide values, bounds, and various characterizations for specific instances of this problem: the binary symmetric case, the scalar Gaussian case, the vector Gaussian case and the symmetric modulo-additive case. Finally, for the general case, we propose a Blahut-Arimoto type of alternating iterations algorithm to find a consistent solution to this problem.


page 1

page 2

page 3

page 4

βˆ™ 10/03/2021

A Class of Nonbinary Symmetric Information Bottleneck Problems

We study two dual settings of information processing. Let 𝖸→𝖷→𝖢 be a Mar...
βˆ™ 02/24/2017

Nonparanormal Information Estimation

We study the problem of using i.i.d. samples from an unknown multivariat...
βˆ™ 03/31/2023

Generalized Information Bottleneck for Gaussian Variables

The information bottleneck (IB) method offers an attractive framework fo...
βˆ™ 12/31/2019

On the Difference Between the Information Bottleneck and the Deep Information Bottleneck

Combining the Information Bottleneck model with deep learning by replaci...
βˆ™ 11/07/2015

Information Extraction Under Privacy Constraints

A privacy-constrained information extraction problem is considered where...
βˆ™ 07/10/2023

Information decomposition to identify relevant variation in complex systems with machine learning

One of the fundamental steps toward understanding a complex system is id...
βˆ™ 01/29/2021

An Information Bottleneck Problem with RΓ©nyi's Entropy

This paper considers an information bottleneck problem with the objectiv...

Please sign up or login with your details

Forgot password? Click here to reset