Critical Slowing Down Near Topological Transitions in Rate-Distortion Problems

03/03/2021
by   Shlomi Agmon, et al.
0

In Rate Distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the Rate Distortion and Information Bottleneck settings. We argue that they suffer from Critical Slowing Down – a diverging number of iterations for convergence – near the critical points. This phenomenon can have theoretical and practical implications for both Machine Learning and Data Compression problems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/22/2022

The Lack of Convexity of the Relevance-Compression Function

In this paper we investigate the convexity of the relevance-compression ...
research
02/09/2022

Minimax Rate-Distortion

We show the existence of universal, variable-rate rate-distortion codes ...
research
05/08/2020

Lossy Compression with Distortion Constrained Optimization

When training end-to-end learned models for lossy compression, one has t...
research
01/11/2022

Rate Distortion Theory for Descriptive Statistics

Rate distortion theory was developed for optimizing lossy compression of...
research
08/23/2021

Rate distortion comparison of a few gradient quantizers

This article is in the context of gradient compression. Gradient compres...
research
02/27/2020

A Free-Energy Principle for Representation Learning

This paper employs a formal connection of machine learning with thermody...
research
05/08/2023

Computation of Rate-Distortion-Perception Function under f-Divergence Perception Constraints

In this paper, we study the computation of the rate-distortion-perceptio...

Please sign up or login with your details

Forgot password? Click here to reset