    # Reinterpreting Importance-Weighted Autoencoders

The standard interpretation of importance-weighted autoencoders is that they maximize a tighter lower bound on the marginal likelihood than the standard evidence lower bound. We give an alternate interpretation of this procedure: that it optimizes the standard variational lower bound, but using a more complex distribution. We formally derive this result, present a tighter lower bound, and visualize the implicit importance-weighted distribution.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Background

The importance-weighted autoencoder (IWAE; Burda et al. (2016)) is a variational inference strategy capable of producing arbitrarily tight evidence lower bounds. IWAE maximizes the following multi-sample evidence lower bound (ELBO):

 logp(x) ≥Ez1...zk∼q(z|x)[log(1kk∑i=1p(x,zi)q(zi|x))]=LIWAE[q] (IWAE ELBO)

which is a tighter lower bound than the ELBO maximized by the variational autoencoder (VAE; Kingma & Welling (2014)):

 logp(x) ≥Ez∼q(z|x)[log(p(x,z)q(z|x))]=LVAE[q]. (VAE ELBO)

## 2 Defining the implicit distribution ~qIW

In this section, we derive the implicit distribution that arises from importance sampling from a distribution using as a proposal distribution. Given a batch of samples from , the following is the unnormalized importance-weighted distribution:

 ~qIW(z|x,z2:k)=p(x,z)q(z|x)1k∑kj=1p(x,zj)q(zj|x)q(z|x)=p(x,z)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x)) (1)

Here are some properties of the approximate IWAE posterior:

• When , equals .

• When , the form of depends on the true posterior .

• As , approaches the true posterior pointwise.

See the appendix for details. Importantly, is dependent on the batch of samples . See Fig. 3 in the appendix for a visualization of with different batches of .

### 2.1 Recovering the IWAE bound from the VAE bound

Here we show that the IWAE ELBO is equivalent to the VAE ELBO in expectation, but with a more flexible, unnormalized distribution, implicitly defined by importance reweighting. If we replace with and take an expectation over , then we recover the IWAE ELBO:

 Ez2…zk∼q(z|x)[LVAE[~qIW(z|z2:k)]] =Ez2…zk∼q(z|x)[∫z~qIW(z|z2:k)log(p(x,z)~qIW(z|x,z2:k))dz] =Ez2…zk∼q(z|x)[∫z~qIW(z|z2:k)log(1kk∑i=1p(x,zi)q(zi|x))dz] =Ez1…zk∼q(z|x)[log(1kk∑i=1p(x,zi)q(zi|x))]=LIWAE[q]

For a more detailed derivation, see the appendix. Note that we are abusing the VAE lower bound notation because this implies an expectation over an unnormalized distribution. Consequently, we replace the expectation with an equivalent integral.

### 2.2 Expected importance weighted distribution qEW

We can achieve a tighter lower bound than by taking the expectation over of . The expected importance-weighted distribution is a distribution given by:

 qEW(z|x)=Ez2...zk∼q(z|x)[~qIW(z|x,z2:k)]=Ez2...zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢⎣p(x,z)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x))⎤⎥ ⎥ ⎥ ⎥⎦ (2)

See section 5.2 for a proof that

is a normalized distribution. Using

in the VAE ELBO, , results in an upper bound of . See section 5.3 for the proof, which is a special case of the proof in Naesseth et al. (2017). The procedure to sample from is shown in Algorithm 1. It is equivalent to sampling-importance-resampling (SIR).

### 2.3 Visualizing the nonparameteric approximate posterior

The IWAE approximating distribution is nonparametric in the sense that, as the true posterior grows more complex, so does the shape of and

. This makes plotting these distributions challenging. A kernel-density-estimation approach could be used, but requires many samples. Thankfully, equations (

1) and (2) give us a simple and fast way to approximately plot and without introducing artifacts due to kernel density smoothing.

Figure 1 visualizes on a 2D distribution approximation problem using Algorithm 2. The base distribution is a Gaussian. As we increase the number of samples and keep the base distribution fixed, we see that the approximation approaches the true distribution. See section 5.6 for 1D visualizations of and with .

## 3 Resampling for prediction

During training, we sample the distribution and implicitly weight them with the IWAE ELBO. After training, we need to explicitly reweight samples from . Figure 2: Reconstructions of MNIST samples from q(z|x) and qEW. The model was trained by maximizing the IWAE ELBO with K=50 and 2 latent dimensions. The reconstructions from q(z|x) are greatly improved with the sampling-resampling step of qEW.

In figure 2, we demonstrate the need to sample from rather than for reconstructing MNIST digits. We trained the model to maximize the IWAE ELBO with K=50 and 2 latent dimensions, similar to Appendix C in Burda et al. (2016). When we sample from and reconstruct the samples, we see a number of anomalies. However, if we perform the sampling-resampling step (Alg. 1), then the reconstructions are much more accurate. The intuition here is that we trained the model with with then sampled from ( with ), which are very different distributions, as seen in Fig. 1.

## 4 Discussion

Bachman & Precup (2015) also showed that the IWAE objective is equivalent to stochastic variational inference with a proposal distribution corrected towards the true posterior via normalized importance sampling. We build on this idea by further examining and by providing visualizations to help better grasp the interpretation. To summarize our observations, the following is the ordering of lower bounds given specific proposal distributions,

 logp(x)≥LVAE[qEW]≥Ez2…zk∼q(z|x)[LVAE[~qIW(z|z2:k)]]=LIWAE[q]≥LVAE[q]

In light of this, IWAE can be seen as increasing the complexity of the approximate distribution , similar to other methods that increase the complexity of , such as Normalizing Flows (Jimenez Rezende & Mohamed, 2015), Variational Boosting (Miller et al., 2016) or Hamiltonian variational inference (Salimans et al., 2015). With this interpretation in mind, we can possibly generalize to be applicable to other divergence measures. An interesting avenue of future work is the comparison of IW-based variational families with alpha-divergences or operator variational objectives.

#### Acknowledgments

We’d like to thank an anonymous ICLR reviewer for providing insightful future directions for this work. We’d like to thank Yuri Burda, Christian Naesseth, and Scott Linderman for bringing our attention to oversights in the paper. We’d also like to thank Christian Naesseth for the derivation in section 5.3 and for providing many helpful comments.

## References

• Bachman & Precup (2015) Philip Bachman and Doina Precup. Training Deep Generative Models: Variations on a Theme. NIPS Approximate Inference Workshop, 2015.
• Burda et al. (2016) Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR, 2016.
• Jimenez Rezende & Mohamed (2015) Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In ICML, 2015.
• Kingma & Welling (2014) Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.
• Miller et al. (2016) Andrew C. Miller, Nicholas Foti, and Ryan P. Adams. Variational Boosting: Iteratively Refining Posterior Approximations.

Advances in Approximate Bayesian Inference, NIPS Workshop

, 2016.
• Naesseth et al. (2017) C. A. Naesseth, S. W. Linderman, R. Ranganath, and D. M. Blei. Variational Sequential Monte Carlo. ArXiv preprint, 2017.
• Salimans et al. (2015) Tim Salimans, Diederik P. Kingma, and Max Welling. Markov chain monte carlo and variational inference: Bridging the gap. In ICML, 2015.

## 5 Appendix

### 5.1 Detailed derivation of the equivalence of VAE and IWAE bound

Here we show that the expectation over of the VAE lower bound with the unnomalized importance-weighted distribution , , is equivalent to the IWAE bound with the original distribution, .

 Ez2…zk∼q(z|x)[LVAE[~qIW(z|z2:k)]] =Ez2…zk∼q(z|x)[∫z~qIW(z|x,z2:k)log(p(x,z)~qIW(z|x,z2:k))dz] (3) =Ez2…zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢ ⎢⎣∫z~qIW(z|x,z2:k)log⎛⎜ ⎜ ⎜ ⎜ ⎜⎝p(x,z)p(x,z)1k∑ki=1p(x,zi)q(zi|x)⎞⎟ ⎟ ⎟ ⎟ ⎟⎠dz⎤⎥ ⎥ ⎥ ⎥ ⎥⎦ (4) =Ez2…zk∼q(z|x)[∫z~qIW(z|x,z2:k)log(1kk∑i=1p(x,zi)q(zi|x))dz] (5) =Ez2…zk∼q(z|x)[∫z~qIW(z|x,z2:k)log(1kk∑i=1p(x,zi)q(zi|x))dz] (6) =Ez2…zk∼q(z|x)⎡⎢ ⎢ ⎢⎣∫zkp(x,z)q(z|x)∑kj=1p(x,zj)q(zj|x)q(z|x)log(1kk∑i=1p(x,zi)q(zi|x))dz⎤⎥ ⎥ ⎥⎦ (7) =Ez2…zk∼q(z|x)⎡⎢ ⎢ ⎢⎣∫z1kp(x,z1)q(z1|x)∑kj=1p(x,zj)q(zj|x)q(z|x)log(1kk∑i=1p(x,zi)q(zi|x))dz⎤⎥ ⎥ ⎥⎦ (8) =Ez1…zk∼q(z|x)⎡⎢ ⎢ ⎢⎣kp(x,z1)q(z1|x)∑kj=1p(x,zj)q(zj|x)log(1kk∑i=1p(x,zi)q(zi|x))⎤⎥ ⎥ ⎥⎦ (9) =Ez1…zk∼q(z|x)⎡⎢ ⎢ ⎢⎣∑kj=1p(x,zj)q(zj|x)∑kj=1p(x,zj)q(zj|x)log(1kk∑i=1p(x,zi)q(zi|x))⎤⎥ ⎥ ⎥⎦ (10) =Ez1…zk∼q(z|x)[log(1kk∑i=1p(x,zi)q(zi|x))] (11) =LIWAE[q] (12)

(8): Change of notation .
(10): has the same expectation as so we can replace with the sum of terms.

### 5.2 Proof that qEW is a normalized distribution

 ∫zqEW(z|x)dz =∫zEz2...zk∼q(z|x)[~qIW(z|x,z2:k)]dz (13) =∫zEz2...zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢⎣p(x,z)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x))⎤⎥ ⎥ ⎥ ⎥⎦dz (14) =∫zq(z|x)q(z|x)Ez2...zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢⎣p(x,z)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x))⎤⎥ ⎥ ⎥ ⎥⎦dz (15) =Ez∼q(z|x)Ez2...zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢⎣p(x,z)q(z|x)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x))⎤⎥ ⎥ ⎥ ⎥⎦ (16) =Ez1...zk∼q(z|x)⎡⎢ ⎢ ⎢ ⎢⎣p(x,z1)q(z1|x)1k(∑kj=1p(x,zj)q(zj|x))⎤⎥ ⎥ ⎥ ⎥⎦ (17) =k∗Ez1...zk∼q(z|x)⎡⎢ ⎢ ⎢⎣p(x,z1)q(z1|x)∑kj=1p(x,zj)q(zj|x)⎤⎥ ⎥ ⎥⎦ (18) =k∑i=1Ez1...zk∼q(z|x)⎡⎢ ⎢ ⎢⎣p(x,zi)q(zi|x)∑kj=1p(x,zj)q(zj|x)⎤⎥ ⎥ ⎥⎦ (19) =Ez1...zk∼q(z|x)⎡⎢ ⎢ ⎢⎣∑ki=1p(x,zi)q(zi|x)∑kj=1p(x,zj)q(zj|x)⎤⎥ ⎥ ⎥⎦ (20) =Ez1...zk∼q(z|x) (21) =1 (22)

(17): Change of notation .
(19): has the same expectation as so we can replace with the sum of terms.
(20): Linearity of expectation.

### 5.3 Proof that LVAE[qEW] is an upper bound of LIWAE[q]

Proof provided by Christian Naesseth.

Let

 LVAE[qEW] =Ez∼qEW[log(p(x,z)qEW(z|x))] (23) =Ez∼qEW⎡⎢ ⎢⎣log⎛⎜ ⎜⎝p(x,z)Eq(z2:k|x)[p(x,z)^p(x|z1:k)]⎞⎟ ⎟⎠⎤⎥ ⎥⎦ (24) =Ez∼qEW⎡⎢ ⎢⎣log⎛⎜ ⎜⎝1Eq(z2:k|x)[1^p(x|z1:k)]⎞⎟ ⎟⎠⎤⎥ ⎥⎦ (25) =Ez∼qEW[−log(Eq(z2:k|x)[^p(x|z1:k)−1])] (26) =−∫zp(x,z)Eq(z2:k|x)[^p(x|z1:k)−1]log(Eq(z2:k|x)[^p(x|z1:k)−1])dz (27) ≥−∫zp(x,z)Eq(z2:k|x)[^p(x|z1:k)−1log(^p(x|z1:k)−1)]dz (28) =−∫zp(x,z)∫z2:kq(z2:k|x)^p(x|z1:k)−1log(^p(x|z1:k)−1)dz (29) =−∫z1:kq(z1|x)q(z1|x)p(x,z1)q(z2:k|x)^p(x|z1:k)−1log(^p(x|z1:k)−1)dz (30) =−∫z1:kp(x,z1)q(z1|x)q(z1:k|x)^p(x|z1:k)−1log(^p(x|z1:k)−1)dz (31) =∫z1:kp(x,z1)q(z1|x)^p(x|z1:k)q(z1:k|x)log(^p(x|z1:k))dz (32) =k∫z1:kp(x,z1)q(z1|x)∑kj=1p(x,zj)q(zj|x)q(z1:k|x)log(1kk∑j=1p(x,zj)q(zj|x))dz (33) =k∑i=1∫z1:kp(x,zi)q(zi|x)k∑j=1p(x,zj)q(zj|x)q(z1:k|x)log(1kk∑j=1p(x,zj)q(zj|x))dz (34) =∫z1:k∑ki=1p(x,zi)q(zi|x)k∑j=1p(x,zj)q(zj|x)q(z1:k|x)log(1kk∑j=1p(x,zj)q(zj|x))dz (35) =∫z1:kq(z1:k|x)log(1kk∑j=1p(x,zj)q(zj|x))dz (36) =Eq(z1:k)[log(1kk∑j=1p(x,zj)q(zj|x))] (37) =LIWAE[q] (38)

(28): Given that is concave for , and , then .
(30): Change of notation .
(34): has the same expectation as so we can replace with the sum of terms.

### 5.4 Proof that qEW is closer to the true posterior than q

The previous section showed that . That is, the IWAE ELBO with the base is a lower bound to the VAE ELBO with the importance weighted . Due to Jensen's inequality and as shown in Burda et al. (2016), we know that the IWAE ELBO is an upper bound of the VAE ELBO: . Furthermore, the log marginal likelihood can be factorized into: , and rearranged to: .

Following the observations above and substituting for :

 KL(qEW||p) =log(p(x))−LVAE[qEW] (39) ≤log(p(x))−LIWAE[q] (40) ≤log(p(x))−LVAE[q]=KL(q||p) (41)

Thus, , meaning is closer to the true posterior than in terms of KL divergence.

### 5.5 In the limit of the number of samples

Another perspective is in the limit of . Recall that the marginal likelihood can be approximated by importance sampling:

 p(x) =Eq(z|x)[p(x,z)q(z|x)]≈1kk∑ip(x,zi)q(zi|x) (42)

where is sampled from . We see that the denominator of is approximating . If

is bounded, then it follows from the strong law of large numbers that, as

approaches infinity, converges to the true posterior almost surely. This interpretation becomes clearer when we factor out the true posterior from :

 ~qIW(z|x,z2:k)=p(x)1k(p(x,z)q(z|x)+∑kj=2p(x,zj)q(zj|x))p(z|x) (43)

We see that the closer the denominator becomes to , the closer is to the true posterior.

### 5.6 Visualizing ~qIW and qEW in 1D Figure 3: Visualization of 1D ~qIW and qEW distributions. The blue p(z) distribution and the green q(z) distribution are both normalized. The three instances of ~qIW(z|z2) (k=2) have different z2 samples from q(z) and we can see that they are unnormalized. qEW(z) is normalized and is the expectation over 30 ~qIW(z|z2) distributions. The ~qIW distributions were plotted using Algorithm 2 with S=1.