DeepAI AI Chat
Log In Sign Up

Understanding VAEs in Fisher-Shannon Plane

by   Huangjie Zheng, et al.

In information theory, Fisher information and Shannon information (entropy) are respectively used to measure the ability in parameter estimation and the uncertainty among variables. The uncertainty principle asserts a fundamental relationship between Fisher information and Shannon information, i.e., the more Fisher information we get, the less Shannon information we gain, and vice versa. This enlightens us about the essence of the encoding/decoding procedure in variational auto-encoders (VAEs) and motivates us to investigate VAEs in the Fisher-Shannon plane. Our studies show that the performance of the latent representation learning and the log-likelihood estimation are intrinsically influenced by the trade-off between Fisher information and Shannon information. To flexibly adjust the trade-off, we further propose a variant of VAEs that can explicitly control Fisher information in encoding/decoding mechanism, termed as Fisher auto-encoder (FAE). Through qualitative and quantitative experiments, we show the complementary properties of Fisher information and Shannon information, and give a guide for Fisher information conditioning to achieve high resolution reconstruction, disentangle feature learning, over-fitting/over-regularization resistance, etc.


page 12

page 13

page 14

page 18


Fisher Auto-Encoders

It has been conjectured that the Fisher divergence is more robust to mod...

The Relation Between Bayesian Fisher Information and Shannon Information for Detecting a Change in a Parameter

We derive a connection between performance of estimators the performance...

Properties of discrete Fisher information: Cramer-Rao-type and log-Sobolev-type inequalities

The Fisher information have connections with the standard deviation and ...

Information Theoretic Limits of Cardinality Estimation: Fisher Meets Shannon

In this paper we study the intrinsic tradeoff between the space complexi...

Uncertainty Estimation by Fisher Information-based Evidential Deep Learning

Uncertainty estimation is a key factor that makes deep learning reliable...

Flowing the Information from Shannon to Fisher: Towards the Fundamental Tradeoff in ISAC

Integrated Sensing and Communication (ISAC) is recognized as a promising...