Understanding VAEs in Fisher-Shannon Plane

07/10/2018 ∙ by Huangjie Zheng, et al. ∙ 0

In information theory, Fisher information and Shannon information (entropy) are respectively used to measure the ability in parameter estimation and the uncertainty among variables. The uncertainty principle asserts a fundamental relationship between Fisher information and Shannon information, i.e., the more Fisher information we get, the less Shannon information we gain, and vice versa. This enlightens us about the essence of the encoding/decoding procedure in variational auto-encoders (VAEs) and motivates us to investigate VAEs in the Fisher-Shannon plane. Our studies show that the performance of the latent representation learning and the log-likelihood estimation are intrinsically influenced by the trade-off between Fisher information and Shannon information. To flexibly adjust the trade-off, we further propose a variant of VAEs that can explicitly control Fisher information in encoding/decoding mechanism, termed as Fisher auto-encoder (FAE). Through qualitative and quantitative experiments, we show the complementary properties of Fisher information and Shannon information, and give a guide for Fisher information conditioning to achieve high resolution reconstruction, disentangle feature learning, over-fitting/over-regularization resistance, etc.



There are no comments yet.


page 12

page 13

page 14

page 18

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The common latent variable models fit in order to model the data with a latent variable

as representation. Variational Autoencoders (VAEs)

[14], recently as one of the most popular latent variable models, maximize the evidence lower bound (ELBO) in an encoding/decoding mechanism.

where and

are encoder and decoder implemented with neural networks parameterized by


denotes the Kullback-Leibler divergence. The learning targets of VAEs may be interpreted in the following two perspectives. In the perspective of variational optimization

[8], VAEs aim to learn a proper model by maximum likelihood with . In the perspective of representation learning [3], VAEs target to learn latent codes that are sufficiently encoded with information about the input by . Several variants of VAEs [29, 30, 7, 32] have been developed based on above two perspectives in recent years.

In recent studies, it has been reported that VAEs are difficult to balance between the representation learning and likelihood maximizing. For instance, the latent variable generated by the encoder is approximately ignored when the decoder is too expressive [4]. As discussed in [1], the evidence lower bound (ELBO) lacks a measure in the quality of the representation, since the KL divergence only controls the way VAE encodes a representation. Several studies in VAEs [7, 32] have attempted to improve the balance between the learned representation quality and ELBO maximization on the basis of information theory. A Shannon entropy-based constraint is introduced to assess the quality of representation learning when optimizing the ELBO, so as to guarantee that sufficient information of the observation flows into the latent code. For example, [21] optimized ELBO plus a mutual information regularizer to explicitly control the information stored in latent codes. Although it is useful to consider mutual information (a member of Shannon family) in VAE encoding for representation learning, how these information quantities affect VAEs has not yet been theoretically analyzed so far. Besides, previous works mainly leverage the Shannon information, which usually suffers from the intractability in computing, yielding an approximation surrogate [21].

In information theory, the uncertainty property [9, 31] has revealed a trade-off between Fisher information and Shannon information, which quantify the uncertainty associated with distribution modeling and the entropy in the predicted values of variables respectively [22]. Fisher-Shannon (FS) information plane [31] is proposed to analyze the complementarity between Fisher information and Shannon information. In this paper, based on the uncertainty property, we attempt to investigate VAEs in FS information plane. We first perform a theoretical analysis with VAEs and show that representation learning with latent variable models via Maximum likelihood estimation is intrinsically related to the trade-off between Fisher information and Shannon entropy. Based on the above findings, we propose a family of VAEs regularized by Fisher information, named Fisher auto-encoder (FAE), to control the information quality during encoding/decoding. Finally, we perform a range of experiments to analyze a variety of VAE models in the FS information plane to empirically validate our findings. In addition, regularized with the Fisher information, FAE is shown to not only provide a novel insight in the information trade-off, but can also improve the reconstruction accuracy and the representation learning.

2 Related work

Information uncertainty: Fisher information and Shannon information are considered important tools to describe the informational behavior in information systems respectively in the distribution modeling view and in the variable view [5]. The generalization of Heisenberg Uncertainty Principle [24] into information system demonstrates that Fisher information and Shannon information are intrinsically linked, with the uncertainty property, where higher Fisher information will result in lower Shannon information and vice versa [9]. With this property, Fisher information and Shannon information are considered complementary aspects and be widely used in solving dual problem when one aspect is intractable [19]. To better take advantage of this property, [31] construct the Fisher-Shannon information plane for signal analysis in joint view of Fisher-Shannon information.

VAEs in information perspective: Variational autoencoders [14], with a auto-encoding form, can be regarded as an information system serves two goals. On one hand, we expect a proper distribution estimation to maximize the marginal likelihood; on the other hand, we hope the latent code can provide sufficient information of data point so as to serve downstream tasks. To improve the log-likelihood estimation, several works, such as PixelCNN and PixelRNN [29, 30] model the dependencies among pixels with autoregressive structure to achieve an expressive density estimator. As for the latent code, plenty of works address it in the perspective of Shannon information [7]. Mutual information, an member of Shannon family, is applied to measure the mutual dependence between datapoint and latent code [32, 1]. The leverage of mutual information is achieved with Maximum-Mean Discrepancy [32] and Wasserstein distance [27]. More generally, [21] regularize the mutual information in VAE’s objective to control the information in latent code.

Effects of Fisher information and Shannon information: Fisher information and Shannon information (typically we call entropy), as complementary aspects, possess their properties. In [22], the entropy is explained as a measure of “global character” that is invariant to strong changes in the distribution, while the Fisher information is interpreted as a measure of the ability to model a distribution, which corresponds to the “local" characteristics. The characteristics of these two sides have been taken into advantages of several existing works, e.g., entropy has been introduced in improving deep neural networks on tasks like classification [25, 23]; FI has been introduced to to evaluate neural network’s parameter estimation [28, 11]. To better understand how these two perspectives affect the mechanism of VAEs, we study VAEs in the Fisher-Shannon information plane to provide a complete understanding of VAEs in a joint view of Fisher information and Shannon information.

3 Fisher-Shannon Information Plane

In this section, we first present the information uncertainty property to link the Fisher information and the Shannon information [9]

of the random variable. After that, a Fisher-Shannon information plane 

[31] is constructed to jointly analyze the characteristics of the random variable on its distribution. This provides the simple basics to understand VAEs with the Fisher-Shannon information plane.

3.1 Fisher-Shannon Information Uncertainty property

In information theory, considering a random variable

, whose probability density function is denoted as

, the Fisher information111Note that, we follow the non-parametric Fisher information definition that differentiates on random variables, which can be transformed with a translation of parameter from parametric version [26]. for and its Shannon entropy can be formulated as:


Above two information quantities are respectively related to the precision that the model fits in observed data and the uncertainty in the outcome of the random variable. For convenience to use in deduction, Shannon entropy is usually transformed to the following quantity which is called the entropy power [26]:


The measure of and verifies a set of resembling inequalities in information theory [26]. Specifically, one of the inequalities connecting the two quantities and being tightly related to the phenomena studied in VAEs, is the uncertainty inequality [9], which is formulated as:


where the equality holds when the random variable

is a Gaussian variable. Note that this inequality possesses several versions; in the case of a random vector

, the corresponding Fisher information turns into a dimensional Fisher information matrix and we need to compute the trace of this matrix (see [10]) and we have .

When the distribution of given variable is fixed, the product of the Fisher information and the entropy power is a constant that is greater or equal to , which depends on the distribution form, the dimension of the random vector, etc. We can further formulate this property as follow:


where is a constant number and . Eq. (3) and (4) indicate the measure of Fisher information and Shannon information exists a trade-off between these two quantities.

3.2 Fisher-Shannon Information Plane

To facilitate the analysis of above two information quantities together, an information plane based on the Fisher information and the Shannon entropy power is proposed in [31] and we generalize it as follows,


where denotes a region , which is limited by the Gaussian case. This plane consists of several Fisher-Shannon (FS) curves , which characterizes the random variable with different distributions.

As discussed in [1], the quality of latent variable is hard to measure in maximizing ELBO, and various VAEs, like [7, 12] have been proposed to balance the trade-off between representation learning and optimization. In the FS plane, different VAEs can be analyzed jointly with Fisher information and Shannon information. By observing their location in FS plane, we can identify the characteristic of this VAE model.

In addition, from the uncertainty property between Fisher information and Shannon information, these two quantities are shown tightly connected. As shown in Eq. (4), when the distribution of given random variable is fixed, the Fisher information and Shannon entropy power’s product is a constant, where the trade-off exists. We can take advantage of this trade-off to avoid the intractability in Shannon information computing. In this paper, we propose a family of VAEs that control the Fisher information, named Fisher Auto-Encoder (FAE), which allows a more accurate description in situations where the Shannon information shows limited dynamics [19] in VAEs. The details of FAE will be discussed in the next section.

4 The Fisher Auto-Encoder

As shown in the previous section, one can apply either Fisher information or Shannon entropy power to control the trade-off between the likelihood estimation and the dependence between data and the latent code . In this section, we come up with a family of VAEs that takes advantage of the Fisher information, named Fisher Autoencoder (FAE), and analyze its characteristics in this section.

4.1 Fisher Information Control in VAE

The Fisher AutoEncoder aims to to control the Fisher information quantity in the objective. Thus, the objective becomes to maximize the evidence lower bound (ELBO) with constraint of Fisher information and we reformulate the VAE’s objective as follows:


where denotes the Kullback-Leibler divergence [13]; and are positive constant that denote the desired Fisher information value. A large value of (resp. ) implies we favor a precise distribution estimation in the model parameterized by (resp. ); while a low value of (resp. ) indicates we weaken the distribution modeling to increase the Shannon entropy power.

To solve the scenario described in Eq. (6), we transfer this optimization problem into a Lagrangian objective, formulated as:


Now the objective consists of three parts, the ELBO to maximize, and two Fisher information regularizers in encoder and decoder, where and are positive constant that control the regularizers. With this objective, we can control the Fisher information in encoder/decoder with an expected desired value . In the most cases, the calculation of Fisher information is not difficult. We can estimate the Fisher information directly by its definition.

4.2 Characteristic of FAE: an example of FI regularization in Gaussian encoder

Here we give a FAE exemplar that only controls the Fisher information in encoder, which means we set in Eq. (7) as zero. In this model, we assume that all random variables are of dimension (i.e. in the scalar case) for simplicity in presentation. The FAE objective is formulated as:


This objective consists of a reconstruction term and a generalized regularizor that considering the Fisher information other than KL divergence. Same to the VAE [14], both prior distribution and posterior approximation are Gaussian, thus the KL-divergence can be analytically computed as:


where and

respectively correspond to the mean and standard derivation of a Gaussian distribution. The Fisher information can be easily computed by definition:


Finally, putting Eq. (9) and Eq. (10) together, we have the following regularizer ,


Considering the KL-divergence term in the original VAEs [14], the optimal is reached at , which aligns the posterior

to a normal distribution

. However, in Eq. (11

), we can observe that the variance is also penalized by the desired Fisher information value

, which will push the variance to approach zero when is large or make the variance larger than when is picked as a small value.

In the above discussion, we analyze the characteristics of FAE in variance control. This property corresponds to the inequality of Cramer-Rao, from which the uncertainty principle shown in Eq. (3) can be derive [9]. Given a stochastic variable of mean and variance , the Fisher information is the lower bound of the variance in a non-biased estimation:


the equality holds if and only if is Gaussian. This inequality gives us the first impression of the characteristic of Fisher information: When FI is in a low value, the variance of the estimation is forced to be high, causing larger uncertainty of the model estimation. Thus, we need to enlarge the FI to make the variance more controllable.

4.3 Connection to the Mutual Auto-Encoder

In this section, we demonstrate the connection between the Fisher Auto-Encoder and the Mutual Auto-Encoder (MAE) [21], which is representative in the family of VAEs that leverage the Shannon information.

As discussed in the previous section, the product of Fisher information and entropy power is a constant when the distribution of variable is fixed, as shown in Eq. (4). We can derive:


where the FI regularizor in FAE is equivalent to a regularizor of the conditional entropy .

Looking back into the MAE proposed in [21], this model controls the mutual information between latent variable and data as follows:

where and are positive constants that respectively control the information regularization and the desired mutual information quantity. Since the mutual information is difficult to compute directly, the mutual information is inferred using Gibbs inequality [2]:


where is a parametric distribution that can be modeled by a network. The objective is to maximizing in Eq. (14) with the constraint . Thus, MAE intrinsically controls the mutual information by controlling the conditional entropy:


with of large value, the conditional entropy can minimized more to obtain a larger mutual information, and vice versa. FAE can also set constraint to control the conditional entropy (or ). As Eq. (13) shows, using Fisher information, the FI regularizers are equivalent to the regularizers of the conditional entropy; thus the mutual information between and can also be assessed without derive approximative upper or lower bounds. FAE can thus implicitly control the mutual information by setting proper Fisher information constraint .

5 Experiments

In this section, we perform a range of experiments to investigate the Fisher-Shannon impacts in VAEs. Meanwhile, we expose how the Fisher Auto-Encoder can improve VAEs in encoding/decoding with the Fisher information constraint.

5.1 Experiment Goals and Experimental Settings

As discussed, the entropy power and Fisher Information corresponds to different characteristics. Thus, we aim to explore these characteristics and give corresponding analysis in order to give a better understanding of existing VAE variants. Some specific goals are summarized as:

  • Explore several variants of VAEs models’ characteristics in the FS plane.

  • Explore the characteristics of latent code w.r.t. the Fisher Information and entropy power.

  • Discuss the effect of different FI constraint in FAE.

The experiments are conducted on the MNIST dataset [17] and the SVHN dataset [20]. The first dataset consists of ten categories of 28

28 hand-written digits and is binarized as in 

[16]. We follow the original partition to split the data as 50,000/10,000/10,000 for the training, validation and test. For SVHN dataset , it is MNIST-like but consists of 3232 color images. We apply 73257 digits for training, 26032 digits for testing, and 20000 samples from the additional set [20] for validation. Moreover, we also construct a toy dataset using MNIST data to better illustrate the characteristics of FI and entropy power. The dataset consists of 5800 samples of label “0" and each 100 samples of other labels in training set; the validation and test set remains same as MNIST.

In FAE and its baselines, all random variables are supposed to be Gaussian variables. Here we only concern the hyper-parameters to adjust the penalty of Fisher information in encoding. In practice, we observe this value can be effective when set from 0.01 to 10 (depends on dataset). For the architecture of inference network and generative network, we both deploy a 5-layers network. Since the impacts of fully-connected and convolution architecture do not differ much in the experiments, we here present results using the architecture as 5 full-connected layers of dimension 300. The latent code is of dimension 40.

5.2 Quantitative Results

5.2.1 Fisher-Shannon Plane Analysis

In this part, we conduct a series of experiments on different models to evaluate them in Fisher-Shannon plane to present different characteristics of using Fisher information and entropy power.

Figure 1: (Best view in color) Visualization of variants of VAE models in the Fisher-Shannon Plane (left) and visualization of FAE’s training process in the plane. (right

) The color bar indicates the training epoch.

We first evaluate the test log-likelihood. To compute the test marginal negative log-likelihood (NLL), we apply the Importance Sampling [6] with 5,000 data points for the previously mentioned models. We select the most representative average results from extensive hyper-parameter configuration and expose them in Table 1: when the Fisher information constraint of in FAE () (or the mutual information constraint between data and latent variable in MAE ) is large, the models can achieve a competitive results of state-of-the-arts like pixelVAE [30] and Inverse Autoregressive Flow(IAF) [15]. When set the information constraint or to zero, we can observe that the results are comparable to the plain VAE, but less competitive than the former models.

VAE[14] pixelVAE[29] IAF[15]
NLL 85.56 79.21 79.85
MAE () MAE () FAE () FAE ()
NLL 80.86 81.58 79.30 83.24
Table 1: Test negative log-likelihood (NLL) estimates for different models on MNIST

We put the former models in the FS information plane and draw the “NJ curve” for the Gaussian variable (where ) in the left subfigure of Figure 1. According to the illustration, we can observe the trade-off between the Fisher information and entropy power in VAE. When the Fisher information elevates, the corresponding entropy power abases and vice versa. When the dependence between data and latent code is higher, where we set larger information constraint or in , the corresponding models appear in the bottom-right corner in the FS plane. In the contrary, the models that contains less information in latent code appear in the upper-left corner, for instance, pixelVAE, which was reported to ignore the latent code [7] appears nearby FAE and MAE with . It is also interesting to notice that the inverse autoregressive flow VAE [15] is beyond the curve. This is due to the IAF transforms the posterior into a more complex distribution from the simple Gaussian distribution. This phenomenon gives us the inspiration to improve the Fisher information and entropy power at the same time. That is to apply a more complex and a more proper distribution assumption for the modeling.

From this plane, it is not hard to learn that we can vary the Fisher information constraint and “move" on this curve. The upper-left corner indicates a less informative code, while the bottom-right indicates a more informative code. In the right sub-figure of Figure 1, the training process of a FAE is also visualized in the FS plane. We plot the location of different epochs in FS plane. It is obvious that in FS plane the training process is intrinsically moving along the “NJ curve" from upper-left side to the bottom-right side. In fact, for most models, the goal is to move further in the bottom-right corner, thus we get better knowledge about data. Setting constraint of the Fisher information means to tell the model how much information we can transfer from data to the latent code, which can affect how far we can move to the right side along this curve.

5.2.2 Effects of Fisher Information and Entropy Power

The former part discusses the characteristics of different models and the corresponding performance. We are still curious about the effects of Fisher information and entropy power in encoding: when we should keep larger Fisher information and when we should keep larger entropy power? In this part, we show different effects when varying Fisher information and entropy power.

Figure 2: From left to right: Visualization of latent code embedding (upper) and distribution of (lower) as entropy power increases (FI decreases).

We set the latent code size to 10, and gradually increase the value of in FAE (from 0 to 20). The embedding of latent variable is visualized with T-SNE [18]; the distribution of is visualized by sampling from and count the norm of normalized , i.e. . The results are presented in Figure 2.

In Figure 2, from left to right, the entropy power increases while the Fisher information decreases. As the entropy power increases, the latent variable embedding becomes more and more expanded in the latent space, where we can observe the clusters become more and more identifiable; while as the Fisher information increases, the embedding becomes more constrained to a smaller space. When observing the distribution , it is obvious that the distribution is more centered when Fisher information is larger, while the distribution swells with larger variance when the Fisher information is abased.

As discussed, the Fisher information will control the variance of the encoding distribution. We can easily find out in Figure 2, the variance of the distribution is getting smaller as FI increases. Intuitively, when VAEs encode the data, if we assign a large Fisher information constraint, the encoding variance is compressed to be smaller, thus the hashing cost is smaller and facilitates the model in distribution fitting. In the contrary, we can set a larger entropy power (or a smaller Fisher information) leaves more uncertainty to the encoding space, thus the latent code grabs the most common information from data points. This helps assemble data points in tasks like classification.

In brief, we conclude the characteristics of large Fisher information and entropy power:

  • Large Fisher information provides with a more refined encoding mechanism, which ensures the latent code contains more detailed information.

  • Large entropy power provides with a more coarse encoding mechanism, which helps in global information extraction.

Larger Fisher information is thus helpful in learning of detailed features, high quality reconstruction, etc.; while larger entropy power is helpful in classification, generalization, etc.

5.3 Qualitative Evaluation

In this section, we present some qualitative results to provide an intuitive visualization. This will help us better understand the characteristics of Fisher information.

We present some reconstruction samples of FAE with large and small Fisher information constraint in Figure 7. As shown, the samples reconstructed with large provide with more pixel details and are more similar to the real images. This is especially more obvious in the case of SVHN, where we can observe more clear texture compared to the one reconstructed with larger constraint . In the contrary, we can find some blurry samples reconstructed with small on MNIST. The blur is more obvious among reconstructed samples from SVHN.

Real images
Reconstructions (=20)
Reconstructions (=0)
Figure 3: From left to right: Real images from test sets of MNIST (upper) and SVHN (lower, best viewed in color); images reconstructed by FAE with large () and small () Fisher information.

We also present some qualitative results on disentangled feature learning in the latent variable in Figure 4. We respectively train two FAE model with and ; then we reconstruct samples by traversing over the latent variable and visualize the corresponding results. The traversal on 10 dimension of the latent variable is over the [-10, 10] range. As shown, the FAE with large learns a better disentangled representation of the generative factors. The latent variable is encoded in a more refined mechanism, where we can distinguish the character that each dimension controls. For example, the first line presents the variance of the width of the digit zero; the second line represents the variance of the inclination direction of the digits, etc. However, when we set , which indicates a large entropy power in the latent variable, the disentangled representation is not obvious as the former. This refers to the characteristic that we discussed in the previous sections: larger entropy power helps the model absorb similar attributes to make a high-level summary of data.

Figure 4: FAE adjust Fisher information value to learn disentangle features in latent variable. (left: ; right: )
FAE () FAE () FAE () FAE ()
Train NLL 94.48 95.12 95.89 96.67
Test NLL 164.56 134.94 109.19 108.37
Table 2: Negative log-likelihood (NLL) estimates on reversed MNIST

5.4 Generalization of using FAE

In this section, we conduct a series of experiments on the reversed MNIST dataset (which mostly consists of data with label “0", as described in section 5.1) to describe a scenario, where we should constrain the Fisher information to obtain larger entropy power.

We trained FAE on this dataset with different . The results of test log-likelihood are presented in Table 2. From the table, the train NLL is shown to decrease as increases, while the test NLL increases. This phenomenon indicates the model suffers from over-fitting on the dataset.

In Figure 5, the reconstructed samples are presented. When Fisher information is large, the model tends to well fit data. However, since the most part of training data are digit zero, the model mainly captures the attributes of digit “0". In test cases, the model will reconstruct samples in a way that contains some attributes of zero. As we constrain the Fisher information, the model has better capacity of generalization with larger entropy power to overcome the over-fitting.

Figure 5: The reconstructed samples by FAE with increasing entropy power.

5.5 Discussion: How to Benefit FS Information

From the previous parts, we can learn that larger Fisher information helps in precise distribution modeling, disentangled feature learning, and achieve a good reconstruction accuracy; while larger entropy power leaves more uncertainty in the model for generalization and enables to handle the over-fitting.

In real world, when we train a model on a dataset, it is important to estimate the uncertainty of the given dataset; for example, whether the data distribution is skewed,

etc.. When the dataset is believed to be complete, or to be representative for the future incoming data, larger Fisher information can be preferred in the model to refine the learning. Otherwise, we should warn the model with uncertainty in the future, so it can get a better generalization capacity to handle the “surprisal" in the future.

6 Conclusion

Based on the uncertainty property between Fisher information and Shannon information, in this paper, we apply the Fisher-Shannon plane to study VAEs in a joint view of these two quantities. In our study of VAEs in Fisher-Shannon plane, these information quantities are demonstrated related to the representation learning and likelihood maximization; the trade-off between Fisher information and Shannon information is shown to result in different characteristics of VAEs. We further propose the Fisher Auto-Encoder for the information control by different Fisher information constraints. In our experiments, we demonstrate the complementary characteristics of Fisher information and Shannon information and provide with a novel understanding of VAEs; we also justify the effectiveness of FAE in information control, high-accuracy reconstruction and non-informative latent code resistance.


  • Alemi et al. [2018] Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. 2018.
  • Barber and Agakov [2003] David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, pages 201–208. MIT Press, 2003.
  • Bengio et al. [2013] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, Aug 2013. ISSN 0162-8828. doi: 10.1109/TPAMI.2013.50.
  • Bowman et al. [2015] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.
  • Brunel and Nadal [1998] Nicolas Brunel and Jean-Pierre Nadal. Mutual information, fisher information, and population coding. Neural computation, 1998.
  • Burda et al. [2015] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519, 2015.
  • Chen et al. [2017] Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. International Conference on Learning Representation, 2017.
  • Cremer et al. [2018] Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. International Conference on Learning Representation, 2018.
  • Dembo et al. [1991] A. Dembo, T. M. Cover, and J. A. Thomas. Information theoretic inequalities. IEEE Transactions on Information Theory, 37(6):1501–1518, Nov 1991. ISSN 0018-9448. doi: 10.1109/18.104312.
  • Dembo [1990] Armir Dembo. Information Inequalities and Uncertainty Principles. Technical Report No. 75, 1990.
  • Desjardins et al. [2015] Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and Koray Kavukcuoglu. Natural neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 2071–2079, 2015.
  • Higgins et al. [2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with a constrained variational framework. 2016.
  • Joyce [2011] James M Joyce. Kullback-leibler divergence. In International encyclopedia of statistical science, pages 720–722. Springer, 2011.
  • Kingma and Welling [2014] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
  • Kingma et al. [2016] Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4743–4751, 2016.
  • Larochelle and Murray [2011] Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 29–37, 2011.
  • Lecun et al. [1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, Nov 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
  • Maaten and Hinton [2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE.

    Journal of Machine Learning Research

    , 9(Nov):2579–2605, 2008.
  • Martin et al. [1999] MT Martin, F Pennini, and A Plastino. Fisher’s information and the analysis of complex signals. Physics Letters A, 256(2-3):173–180, 1999.
  • Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In

    NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011

    , 2011.
    URL http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf.
  • Phuong et al. [2018] Mary Phuong, Max Welling, Nate Kushman, Ryota Tomioka, and Sebastian Nowozin. The mutual autoencoder: Controlling information in latent code representations, 2018.
  • Rosso et al. [2015] Osvaldo Anibal Rosso, Felipe Olivares, and Angelo Plastino. Noise versus chaos in a causal fisher-shannon plane. Papers in physics, 7:070006, 2015.
  • Saxe et al. [2018] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018.
  • Schrödinger [1930] E Schrödinger. About heisenberg uncertainty relation. Proc. Prussian Acad. Sci. Phys. Math. XIX, 293, 1930.
  • Silva et al. [2005] Luis M Silva, Joaquim Marques de Sá, and Luis A Alexandre. Neural network classification using shannon’s entropy. In ESANN, pages 217–222, 2005.
  • Stam [1959] A.J. Stam. Some inequalities satisfied by the quantities of information of fisher and shannon. Information and Control, 2(2):101 – 112, 1959. ISSN 0019-9958. doi: https://doi.org/10.1016/S0019-9958(59)90348-1. URL http://www.sciencedirect.com/science/article/pii/S0019995859903481.
  • Tolstikhin et al. [2018] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HkL7n1-0b.
  • Tu et al. [2016] M. Tu, V. Berisha, M. Woolf, J. Seo, and Y. Cao. Ranking the parameters of deep neural networks using the fisher information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2647–2651, March 2016. doi: 10.1109/ICASSP.2016.7472157.
  • van den Oord et al. [2016a] Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. 2016a. URL http://arxiv.org/abs/1601.06759.
  • van den Oord et al. [2016b] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. 2016b. URL http://arxiv.org/abs/1606.05328.
  • Vignat and Bercher [2003] C. Vignat and J.-F. Bercher. Analysis of signals in the fisher-shannon information plane. Physics Letters A, 312(1):27 – 33, 2003. ISSN 0375-9601. doi: https://doi.org/10.1016/S0375-9601(03)00570-X.
  • Zhao et al. [2017] Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing variational autoencoders. arXiv:1706.02262, 2017.
  • Zheng et al. [2018] H. Zheng, J. Yao, Y. Zhang, and I. W. Tsang. Degeneration in VAE: in the Light of Fisher Information Loss. ArXiv e-prints, February 2018.

Appendix A Non-parametric Fisher information

The Fisher information we formulated in section 3.1 is a non-parametric version. In the original definition of Fisher information, it is formulated as [26]:


where can be multi-dimensional. In [26], if only depends on , for example, the gaussian distribution w.r.t. mean , then can be dropped from to become a non-parametric version.


In reality, we can regard the distribution parameter as a variable and formulate the parametric distribution as a posterior:


Then we can manipulate the prior of to find a distribution that satisfies:


Finally, we transform a distribution from parametric version into non-parametric version. This transformation needs to satisfy three conditions according to [26]:

  1. is strictly positive

  2. is differentiable

  3. The integral (17) exists

Appendix B Minor Characteristic: Stability in Learning

In terms of optimization for VAEs, the stability of parameter estimation is also a concern [15, 32], which also reflects a minor characteristic of using Fisher information. In [33], the instability of parameter will lead to degeneration when VAE’s architecture is deep and is linked to the Fisher information loss. Here, we present the resistance of degeneration with the advantage of Fisher information regularizer in VAEs.

Specifically, we extend the encoder and the decoder to 20, 30, 40 layers (marked as 20L, 30L, 40L) respectively to observe the performance of FAE and MAE in the context of degeneration. The performance is illustrated in Figure 6. As can be seen, the latent space and the reconstruction is similar to those in normal situation with the network going deep. Fisher information captures the variability of the gradient and can alleviate the degeneration in very deep networks. Instead, MAE, although it can also control information in encoding mechanism, it is risky in gradient vanishing. It thus causes inaccurate parameter estimation and results in degeneration in the latent space and reconstruction.

Figure 6: FAE and MAE analysis in different model complexity (best view in color). Visualization of learned latent space and sample reconstructed by FAEs (upper) and MAEs. (lower)

Appendix C Supplementary experiment results on SVHN

Figure 7: Samples reconstructed by FAE.