Analyzing Training Using Phase Transitions in Entropy—Part I: General Theory

12/02/2020
by   Kang Gao, et al.
0

We analyze phase transitions in the conditional entropy of a sequence caused by a change in the conditional variables. Such transitions happen, for example, when training to learn the parameters of a system, since the transition from the training phase to the data phase causes a discontinuous jump in the conditional entropy of the measured system response. For large-scale systems, we present a method of computing a bound on the mutual information obtained with one-shot training, and show that this bound can be calculated using the difference between two derivatives of a conditional entropy. The system model does not require Gaussianity or linearity in the parameters, and does not require worst-case noise approximations or explicit estimation of any unknown parameters. The model applies to a broad range of algorithms and methods in communication, signal processing, and machine learning that employ training as part of their operation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/30/2021

A Training-Based Mutual Information Lower Bound for Large-Scale Systems

We provide a mutual information lower bound that can be used to analyze ...
research
12/02/2020

Analyzing Training Using Phase Transitions in Entropy—Part II: Application to Quantization and Classification

We show that a quantized large-scale system with unknown parameters and ...
research
04/02/2019

Unveiling phase transitions with machine learning

The classification of phase transitions is a central and challenging tas...
research
09/12/2021

Neural network based order parameter for phase transitions and its applications in high-entropy alloys

Phase transition is one of the most important phenomena in nature and pl...
research
01/15/2020

Improvement of an Approximated Self-Improving Sorter and Error Analysis of its Estimated Entropy

The self-improving sorter proposed by Ailon et al. consists of two phase...
research
06/10/2019

HTDet: A Clustering Method using Information Entropy for Hardware Trojan Detection

Hardware Trojans (HTs) have drawn more and more attention in both academ...
research
07/03/2019

Understanding Phase Transitions via Mutual Information and MMSE

The ability to understand and solve high-dimensional inference problems ...

Please sign up or login with your details

Forgot password? Click here to reset