Deep Probabilistic Programming

01/13/2017 ∙ by Dustin Tran, et al. ∙ 0

We propose Edward, a Turing-complete probabilistic programming language. Edward defines two compositional representations---random variables and inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. For flexibility, Edward makes it easy to fit the same model using a variety of composable inference methods, ranging from point estimation to variational inference to MCMC. In addition, Edward can reuse the modeling representation as part of inference, facilitating the design of rich variational models and generative adversarial networks. For efficiency, Edward is integrated into TensorFlow, providing significant speedups over existing probabilistic systems. For example, we show on a benchmark logistic regression task that Edward is at least 35x faster than Stan and 6x faster than PyMC3. Further, Edward incurs no runtime overhead: it is as fast as handwritten TensorFlow.



There are no comments yet.


page 1

page 2

page 3

page 4

Code Repositories


Reference MEMO of LifeScience AI

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The nature of deep neural networks is compositional. Users can connect layers in creative ways, without having to worry about how to perform testing (forward propagation) or inference (gradient-based optimization, with back propagation and automatic differentiation).

In this paper, we design compositional representations for probabilistic programming. Probabilistic programming lets users specify generative probabilistic models as programs and then “compile” those models down into inference procedures. Probabilistic models are also compositional in nature, and much work has enabled rich probabilistic programs via compositions of random variables (Goodman et al., 2012; Ghahramani, 2015; Lake et al., 2016).
Less work, however, has considered an analogous compositionality for inference. Rather, many existing PPLPPL treat the inference engine as a black box, abstracted away from the model. These cannot capture probabilistic inferences that reuse the model’s representation—a key idea in recent advances in variational inference (Kingma & Welling, 2014; Rezende & Mohamed, 2015; Tran et al., 2016b), GANGAN (Goodfellow et al., 2014), and also in more classic inferences (Dayan et al., 1995; Gutmann & Hyvärinen, 2010).
We propose Edward111See Tran et al. (2016a) for details of the API. A companion webpage for this paper is available at It contains more complete examples with runnable code., a Turing-complete PPL which builds on two compositional representations—one for random variables and one for inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. For flexibility, we show how Edward makes it easy to fit the same model using a variety of composable inference methods, ranging from point estimation to variational inference to MCMC. For efficiency, we show how to integrate Edward into existing computational graph frameworks such as TensorFlow (Abadi et al., 2016)

. Frameworks like TensorFlow provide computational benefits like distributed training, parallelism, vectorization, and GPUGPU support “for free.” For example, we show on a benchmark task that Edward’s HMC is many times faster than existing software. Further, Edward incurs no runtime overhead: it is as fast as handwritten TensorFlow.

2 Related Work

PPL typically trade off the expressiveness of the language with the computational efficiency of inference. On one side, there are languages which emphasize expressiveness (Pfeffer, 2001; Milch et al., 2005; Pfeffer, 2009; Goodman et al., 2012), representing a rich class beyond graphical models. Each employs a generic inference engine, but scales poorly with respect to model and data size. On the other side, there are languages which emphasize efficiency (Spiegelhalter et al., 1995; Murphy, 2001; Plummer, 2003; Salvatier et al., 2015; Carpenter et al., 2016). The PPL is restricted to a specific class of models, and inference algorithms are optimized to be efficient for this class. For example, Infer.NET enables fast message passing for graphical models (Minka et al., 2014)

, and Augur enables data parallelism with GPU for Gibbs sampling in Bayesian networks

(Tristan et al., 2014)

. Edward bridges this gap. It is Turing complete—it supports any computable probability distribution—and it supports efficient algorithms, such as those that leverage model structure and those that scale to massive data.

There has been some prior research on efficient algorithms in Turing-complete languages. Venture and Anglican design inference as a collection of local inference problems, defined over program fragments (Mansinghka et al., 2014; Wood et al., 2014). This produces fast program-specific inference code, which we build on. Neither system supports inference methods such as programmable posterior approximations, inference models, or data subsampling. Concurrent with our work, WebPPL features amortized inference (Ritchie et al., 2016). Unlike Edward, WebPPL does not reuse the model’s representation; rather, it annotates the original program and leverages helper functions, which is a less flexible strategy. Finally, inference is designed as program transformations in Kiselyov & Shan (2009); Ścibior et al. (2015); Zinkov & Shan (2016). This enables the flexibility of composing inference inside other probabilistic programs. Edward builds on this idea to compose not only inference within modeling but also modeling within inference (e.g., variational models).

3 Compositional Representations for Probabilistic Models

We first develop compositional representations for probabilistic models. We desire two criteria: (a) integration with computational graphs, an efficient framework where nodes represent operations on data and edges represent data communicated between them (Culler, 1986); and (b) invariance of the representation under the graph, that is, the representation can be reused during inference.

Edward defines random variables as the key compositional representation. They are class objects with methods, for example, to compute the log density and to sample. Further, each random variable

is associated to a tensor (multi-dimensional array)

, which represents a single sample . This association embeds the random variable onto a computational graph on tensors.

The design’s simplicity makes it easy to develop probabilistic programs in a computational graph framework. Importantly, all computation is represented on the graph. This enables one to compose random variables with complex deterministic structure such as deep neural networks, a diverse set of math operations, and third party libraries that build on the same framework. The design also enables compositions of random variables to capture complex stochastic structure.

As an illustration, we use a Beta-Bernoulli model, , where

is a latent probability shared across the 50 data points

. The random variable x is 50-dimensional, parameterized by the random tensor . Fetching the object x runs the graph: it simulates from the generative process and outputs a binary vector of elements.

1theta = Beta(a=1.0, b=1.0)
2x = Bernoulli(p=tf.ones(50) * theta)


Figure 1: Beta-Bernoulli program (left) alongside its computational graph (right). Fetching from the graph generates a binary vector of elements.

All computation is registered symbolically on random variables and not over their execution. Symbolic representations do not require reifying the full model, which leads to unreasonable memory consumption for large models (Tristan et al., 2014). Moreover, it enables us to simplify both deterministic and stochastic operations in the graph, before executing any code (Ścibior et al., 2015; Zinkov & Shan, 2016).

With computational graphs, it is also natural to build mutable states within the probabilistic program. As a typical use of computational graphs, such states can define model parameters; in TensorFlow, this is given by a tf.Variable. Another use case is for building discriminative models , where are features that are input as training or test data. The program can be written independent of the data, using a mutable state (tf.placeholder) for in its graph. During training and testing, we feed the placeholder the appropriate values.

In Appendix A, we provide examples of a Bayesian neural network for classification (A.1), latent Dirichlet allocation (A.2), and Gaussian matrix factorization (A.3). We present others below.

3.1 Example: Variational Auto-encoder

1# Probabilistic model
2z = Normal(mu=tf.zeros([N, d]), sigma=tf.ones([N, d]))
3h = Dense(256, activation=


4x = Bernoulli(logits=Dense(28 * 28, activation=None)(h))
6# Variational model
7qx = tf.placeholder(tf.float32, [N, 28 * 28])
8qh = Dense(256, activation=’relu’)(qx)
9qz = Normal(mu=Dense(d, activation=None)(qh),
10            sigma=Dense(d, activation=’softplus’)(qh))
Figure 2: VAE for a data set of pixel images: (left) graphical model, with dotted lines for the inference model; (right) probabilistic program, with 2-layer neural networks.

Figure 2 implements a VAE (Kingma & Welling, 2014; Rezende et al., 2014) in Edward. It comprises a probabilistic model over data and a variational model designed to approximate the former’s posterior. Here we use random variables to construct both the probabilistic model and the variational model; they are fit during inference (more details in Section 4).

There are data points each with latent variables,

. The program uses Keras

(Chollet, 2015) to define neural networks. The probabilistic model is parameterized by a 2-layer neural network, with 256 hidden units (and ReLU activation), and generates pixel images. The variational model is parameterized by a 2-layer inference network, with 256 hidden units and outputs parameters of a normal posterior approximation.

The probabilistic program is concise. Core elements of the VAE—such as its distributional assumptions and neural net architectures—are all extensible. With model compositionality, we can embed it into more complicated models (Gregor et al., 2015; Rezende et al., 2016) and for other learning tasks (Kingma et al., 2014). With inference compositionality (which we discuss in Section 4), we can embed it into more complicated algorithms, such as with expressive variational approximations (Rezende & Mohamed, 2015; Tran et al., 2016b; Kingma et al., 2016) and alternative objectives (Ranganath et al., 2016a; Li & Turner, 2016; Dieng et al., 2016).

3.2 Example: Bayesian Recurrent Neural Network with Variable Length

Random variables can also be composed with control flow operations. As an example, Figure 3 implements a Bayesian RNNRNN with variable length. The data is a sequence of inputs and outputs of length with and per time step. For , a RNN applies the update

where the previous hidden state is . We feed each hidden state into the output’s likelihood, , and we place a standard normal prior over all parameters

. Our implementation is dynamic: it differs from a RNN with fixed length, which pads and unrolls the computation.

1def rnn_cell(hprev, xt):
2  return tf.tanh(, Wh) +, Wx) + bh)
4Wh = Normal(mu=tf.zeros([H, H]), sigma=tf.ones([H, H]))
5Wx = Normal(mu=tf.zeros([D, H]), sigma=tf.ones([D, H]))
6Wy = Normal(mu=tf.zeros([H, 1]), sigma=tf.ones([H, 1]))
7bh = Normal(mu=tf.zeros(H), sigma=tf.ones(H))
8by = Normal(mu=tf.zeros(1), sigma=tf.ones(1))
10x = tf.placeholder(tf.float32, [None, D])
11h = tf.scan(rnn_cell, x, initializer=tf.zeros(H))
12y = Normal(mu=tf.matmul(h, Wy) + by, sigma=1.0)
Figure 3: Bayesian RNNRNN: (left) graphical model; (right) probabilistic program. The program has an unspecified number of time steps; it uses a symbolic for loop (tf.scan).

3.3 Stochastic Control Flow and Model Parallelism


Figure 4: Computational graph for a probabilistic program with stochastic control flow.

Random variables can also be placed in the control flow itself, enabling probabilistic programs with stochastic control flow. Stochastic control flow defines dynamic conditional dependencies, known in the literature as contingent or existential dependencies (Mansinghka et al., 2014; Wu et al., 2016). See Figure 4, where may or may not depend on for a given execution. In Section A.4, we use stochastic control flow to implement a Dirichlet process mixture model. Tensors with stochastic shape are also possible: for example, tf.zeros(Poisson(lam=5.0)) defines a vector of zeros with length given by a Poisson draw with rate .

Stochastic control flow produces difficulties for algorithms that use the graph structure because the relationship of conditional dependencies changes across execution traces. The computational graph, however, provides an elegant way of teasing out static conditional dependence structure () from dynamic dependence structure (. We can perform model parallelism (parallel computation across components of the model) over the static structure with GPU and batch training. We can use more generic computations to handle the dynamic structure.

4 Compositional Representations for Inference

We described random variables as a representation for building rich probabilistic programs over computational graphs. We now describe a compositional representation for inference. We desire two criteria: (a) support for many classes of inference, where the form of the inferred posterior depends on the algorithm; and (b) invariance of inference under the computational graph, that is, the posterior can be further composed as part of another model.

To explain our approach, we will use a simple hierarchical model as a running example. Figure 5

displays a joint distribution

of data , local variables , and global variables . The ideas here extend to more expressive programs.

1N = 10000  # number of data points
2D = 2  # data dimension
3K = 5  # number of clusters
5beta = Normal(mu=tf.zeros([K, D]), sigma=tf.ones([K, D]))
6z = Categorical(logits=tf.zeros([N, K]))
7x = Normal(mu=tf.gather(beta, z), sigma=tf.ones([N, D]))
Figure 5: Hierarchical model: (left) graphical model; (right) probabilistic program. It is a mixture of Gaussians over -dimensional data . There are latent cluster means .

4.1 Inference as Stochastic Graph Optimization

The goal of inference is to calculate the posterior distribution given data , where are any model parameters that we will compute point estimates for.222For example, we could replace x’s sigma argument with tf.exp(tf.Variable(0.0))*tf.ones([N, D]). This defines a model parameter initialized at 0 and positive-constrained. We formalize this as the following optimization problem:


where is an approximation to the posterior , and

is a loss function with respect to

and .

The choice of approximation , loss , and rules to update parameters are specified by an inference algorithm. (Note can be nonparametric, such as a point or a collection of samples.)

In Edward, we write this problem as follows:

1inference = ed.Inference({beta: qbeta, z: qz}, data={x: x_train})

Inference is an abstract class which takes two inputs. The first is a collection of latent random variables beta and z, associated to their “posterior variables” qbeta and qz respectively. The second is a collection of observed random variables x, which is associated to their realizations x_train.

The idea is that Inference defines and solves the optimization in Equation 1. It adjusts parameters of the distribution of qbeta and qz (and any model parameters) to be close to the posterior.

Class methods are available to finely control the inference. Calling inference.initialize() builds a computational graph to update . Calling inference.update() runs this computation once to update ; we call the method in a loop until convergence. Importantly, no efficiency is lost in Edward’s language: the computational graph is the same as if it were handwritten for a specific model. This means the runtime is the same; also see our experiments in Section 5.2.

A key concept in Edward is that there is no distinct “model” or “inference” block. A model is simply a collection of random variables, and inference is a way of modifying parameters in that collection subject to another. This reductionism offers significant flexibility. For example, we can infer only parts of a model (e.g., layer-wise training (Hinton et al., 2006)), infer parts used in multiple models (e.g., multi-task learning), or plug in a posterior into a new model (e.g., Bayesian updating).

4.2 Classes of Inference

The design of Inference is very general. We describe subclasses to represent many algorithms below: variational inference, Monte Carlo, and GAN.

Variational inference posits a family of approximating distributions and finds the closest member in the family to the posterior (Jordan et al., 1999). In Edward, we build the variational family in the graph; see Figure 6 (left). For our running example, the family has mutable variables as parameters , where and .

1qbeta = Normal(
2  mu=tf.Variable(tf.zeros([K, D])),
3  sigma=tf.exp(tf.Variable(tf.zeros([K, D]))))
4qz = Categorical(
5  logits=tf.Variable(tf.zeros([N, K])))
7inference = ed.VariationalInference(
8  {beta: qbeta, z: qz}, data={x: x_train})
1T = 10000  # number of samples
2qbeta = Empirical(
3  params=tf.Variable(tf.zeros([T, K, D])))
4qz = Empirical(
5  params=tf.Variable(tf.zeros([T, N])))
7inference = ed.MonteCarlo(
8  {beta: qbeta, z: qz}, data={x: x_train})
Figure 6: (left) Variational inference. (right) Monte Carlo.

Specific variational algorithms inherit from the VariationalInference class. Each defines its own methods, such as a loss function and gradient. For example, we represent MAP estimation with an approximating family (qbeta and qz) of PointMass random variables, i.e., with all probability mass concentrated at a point. MAP inherits from VariationalInference and defines the negative log joint density as the loss function; it uses existing optimizers inside TensorFlow. In Section 5.1, we experiment with multiple gradient estimators for black box variational inference (Ranganath et al., 2014). Each estimator implements the same loss (an objective proportional to the divergence ) and a different update rule (stochastic gradient).

Monte Carlo approximates the posterior using samples (Robert & Casella, 1999). Monte Carlo is an inference where the approximating family is an empirical distribution, and . The parameters are . See Figure 6 (right). Monte Carlo algorithms proceed by updating one sample at a time in the empirical approximation. Specific MCMC samplers determine the update rules: they can use gradients such as in Hamiltonian Monte Carlo (Neal, 2011) and graph structure such as in sequential Monte Carlo (Doucet et al., 2001).

1def generative_network(eps):
2  h = Dense(256, activation=’relu’)(eps)
3  return Dense(28 * 28, activation=None)(h)
5def discriminative_network(x):
6  h = Dense(28 * 28, activation=’relu’)(x)
7  return Dense(h, activation=None)(1)
9# Probabilistic model
10eps = Normal(mu=tf.zeros([N, d]), sigma=tf.ones([N, d]))
11x = generative_network(eps)
13inference = ed.GANInference(data={x: x_train},
14    discriminator=discriminative_network)
Figure 7: GAN: (left) graphical model; (right) probabilistic program. The model (generator) uses a parameterized function (discriminator) for training.

Edward also supports non-Bayesian methods such as GAN (Goodfellow et al., 2014). See Figure 7. The model posits random noise eps over data points, each with dimensions; this random noise feeds into a generative_network function, a neural network that outputs real-valued data x. In addition, there is a discriminative_network which takes data as input and outputs the probability that the data is real (in logit parameterization). We build GANInference; running it optimizes parameters inside the two neural network functions. This approach extends to many advances in GAN (e.g., Denton et al. (2015); Li et al. (2015)).

Finally, one can design algorithms that would otherwise require tedious algebraic manipulation. With symbolic algebra on nodes of the computational graph, we can uncover conjugacy relationships between random variables. Users can then integrate out variables to automatically derive classical Gibbs (Gelfand & Smith, 1990), mean-field updates (Bishop, 2006), and exact inference. These algorithms are being currently developed in Edward.

4.3 Composing Inferences

Core to Edward’s design is that inference can be written as a collection of separate inference programs. Below we demonstrate variational EM, with an (approximate) E-step over local variables and an M-step over global variables. We instantiate two algorithms, each of which conditions on inferences from the other, and we alternate with one update of each (Neal & Hinton, 1993),

1qbeta = PointMass(params=tf.Variable(tf.zeros([K, D])))
2qz = Categorical(logits=tf.Variable(tf.zeros([N, K])))
4inference_e = ed.VariationalInference({z: qz}, data={x: x_train, beta: qbeta})
5inference_m = ed.MAP({beta: qbeta}, data={x: x_train, z: qz})
7for _ in range(10000):
8  inference_e.update()
9  inference_m.update()

This extends to many other cases such as exact EM for exponential families, contrastive divergence

(Hinton, 2002), pseudo-marginal methods (Andrieu & Roberts, 2009), and Gibbs sampling within variational inference (Wang & Blei, 2012; Hoffman & Blei, 2015). We can also write message passing algorithms, which solve a collection of local inference problems (Koller & Friedman, 2009)

. For example, classical message passing uses exact local inference and expectation propagation locally minimizes the Kullback-Leibler divergence,

(Minka, 2001).

4.4 Data Subsampling

Stochastic optimization (Bottou, 2010) scales inference to massive data and is key to algorithms such as stochastic gradient Langevin dynamics (Welling & Teh, 2011) and stochastic variational inference (Hoffman et al., 2013). The idea is to cheaply estimate the model’s log joint density in an unbiased way. At each step, one subsamples a data set of size and then scales densities with respect to local variables,

To support stochastic optimization, we represent only a subgraph of the full model. This prevents reifying the full model, which can lead to unreasonable memory consumption (Tristan et al., 2014). During initialization, we pass in a dictionary to properly scale the arguments. See Figure 8.

1beta = Normal(mu=tf.zeros([K, D]), sigma=tf.ones([K, D]))
2z = Categorical(logits=tf.zeros([M, K]))
3x = Normal(mu=tf.gather(beta, z), sigma=tf.ones([M, D]))
5qbeta = Normal(mu=tf.Variable(tf.zeros([K, D])),
6               sigma=tf.nn.softplus(tf.Variable(tf.zeros([K, D]))))
7qz = Categorical(logits=tf.Variable(tf.zeros([M, D])))
9inference = ed.VariationalInference({beta: qbeta, z: qz}, data={x: x_batch})
10inference.initialize(scale={x: float(N)/M, z: float(N)/M})
Figure 8: Data subsampling with a hierarchical model. We define a subgraph of the full model, forming a plate of size rather than . We then scale all local random variables by .

Conceptually, the scale argument represents scaling for each random variable’s plate, as if we had seen that random variable as many times. As an example, Appendix B shows how to implement stochastic variational inference in Edward. The approach extends naturally to streaming data (Doucet et al., 2000; Broderick et al., 2013; McInerney et al., 2015), dynamic batch sizes, and data structures in which working on a subgraph does not immediately apply (Binder et al., 1997; Johnson & Willsky, 2014; Foti et al., 2014).

5 Experiments

In this section, we illustrate two main benefits of Edward: flexibility and efficiency. For the former, we show how it is easy to compare different inference algorithms on the same model. For the latter, we show how it is easy to get significant speedups by exploiting computational graphs.

5.1 Recent Methods in Variational Inference

Inference method Negative log-likelihood
VAE (Kingma & Welling, 2014) 88.2
VAE without analytic KL 89.4
VAE with analytic entropy 88.1
VAE with score function gradient 87.9
Normalizing flows (Rezende & Mohamed, 2015) 85.8
Hierarchical variational model (Ranganath et al., 2016b) 85.4
Importance-weighted auto-encoders () (Burda et al., 2016) 86.3
HVM with IWAE objective () 85.2
Rényi divergence () (Li & Turner, 2016) 140.5
Table 1:

Inference methods for a probabilistic decoder on binarized MNIST. The Edward PPL is a convenient research platform, making it easy to both develop and experiment with many algorithms.

We demonstrate Edward’s flexibility for experimenting with complex inference algorithms. We consider the VAE setup from Figure 2 and the binarized MNIST data set (Salakhutdinov & Murray, 2008). We use latent variables per data point and optimize using ADAM. We study different components of the VAE setup using different methods; Section C.1 is a complete script. After training we evaluate held-out log likelihoods, which are lower bounds on the true value.

Table 1 shows the results. The first method uses the VAE from Figure 2. The next three methods use the same VAE but apply different gradient estimators: reparameterization gradient without an analytic KL; reparameterization gradient with an analytic entropy; and the score function gradient (Paisley et al., 2012; Ranganath et al., 2014). This typically leads to the same optima but at different convergence rates. The score function gradient was slowest. Gradients with an analytic entropy produced difficulties around convergence: we switched to stochastic estimates of the entropy as it approached an optima. We also use HVM (Ranganath et al., 2016b) with a normalizing flow prior; it produced similar results as a normalizing flow on the latent variable space (Rezende & Mohamed, 2015), and better than IWAE (Burda et al., 2016).

We also study novel combinations, such as HVM with the IWAE objective, GAN-based optimization on the decoder (with pixel intensity-valued data), and Rényi divergence on the decoder. GAN-based optimization does not enable calculation of the log-likelihood; Rényi divergence does not directly optimize for log-likelihood so it does not perform well. The key point is that Edward is a convenient research platform: they are all easy modifications of a given script.

5.2 GPU-accelerated Hamiltonian Monte Carlo

1  # Model
2  x = tf.Variable(x_data, trainable=False)
3  beta = Normal(mu=tf.zeros(D), sigma=tf.ones(D))
4  y = Bernoulli(, beta))
6  # Inference
7  qbeta = Empirical(params=tf.Variable(tf.zeros([T, D])))
8  inference = ed.HMC({beta: qbeta}, data={y: y_data})
9 / N, n_steps=10)
Figure 9: Edward program for Bayesian logistic regression with HMC.
Probabilistic programming system Runtime (s)
Handwritten NumPy (1 CPU) 534
Stan (1 CPU) (Carpenter et al., 2016) 171
PyMC3 (12 CPU) (Salvatier et al., 2015) 30.0
Edward (12 CPU) 8.2
Handwritten TensorFlow (GPU) 5.0
Edward (GPU) 4.9
Table 2: HMC benchmark for large-scale logistic regression. Edward (GPU) is significantly faster than other systems. In addition, Edward has no overhead: it is as fast as handwritten TensorFlow.

We benchmark runtimes for a fixed number of Hamiltonian Monte Carlo (HMC; Neal, 2011) iterations on modern hardware: a 12-core Intel i7-5930K CPU at 3.50GHz and an NVIDIA Titan X (Maxwell) GPU. We apply logistic regression on the Covertype dataset (, ; responses were binarized) using Edward, Stan (with PyStan) (Carpenter et al., 2016), and PyMC3 (Salvatier et al., 2015). We ran 100 HMC iterations, with 10 leapfrog updates per iteration, a step size of , and single precision. Figure 9 illustrates the program in Edward.

Table 2 displays the runtimes.333In a previous version of this paper, we reported PyMC3 took 361s. This was caused by a bug preventing PyMC3 from correctly handling single-precision floating point. (PyMC3 with double precision is roughly 14x slower than Edward (GPU).) This has been fixed after discussion with Thomas Wiecki. The reported numbers also exclude compilation time, which is significant for Stan. Edward (GPU) features a dramatic 35x speedup over Stan (1 CPU) and 6x speedup over PyMC3 (12 CPU). This showcases the value of building a PPL on top of computational graphs. The speedup stems from fast matrix multiplication when calculating the model’s log-likelihood; GPUs can efficiently parallelize this computation. We expect similar speedups for models whose bottleneck is also matrix multiplication, such as deep neural networks.

There are various reasons for the speedup. Stan only used 1 CPU as it leverages multiple cores by running HMC chains in parallel. Stan also used double-precision floating point as it does not allow single-precision. For PyMC3, we note Edward’s speedup is not a result of PyMC3’s Theano backend compared to Edward’s TensorFlow. Rather, PyMC3 does not use Theano for all its computation, so it experiences communication overhead with NumPy. (PyMC3 was actually slower when using the GPU.) We predict that porting Edward’s design to Theano would feature similar speedups.

In addition to these speedups, we highlight that Edward has no runtime overhead: it is as fast as handwritten TensorFlow. Following Section 4.1, this is because the computational graphs for inference are in fact the same for Edward and the handwritten code.

5.3 Probability Zoo

In addition to Edward, we also release the Probability Zoo, a community repository for pre-trained probability models and their posteriors.444The Probability Zoo is available at It includes model parameters and inferred posterior factors, such as local and global variables during training and any inference networks.

It is inspired by the model zoo in Caffe

(Jia et al., 2014), which provides many pre-trained discriminative neural networks, and which has been key to making large-scale deep learning more transparent and accessible. It is also inspired by Forest (Stuhlmüller, 2012), which provides examples of probabilistic programs.

6 Discussion: Challenges & Extensions

We described Edward, a Turing-complete PPL with compositional representations for probabilistic models and inference. Edward expands the scope of probabilistic programming to be as flexible and computationally efficient as traditional deep learning. For flexibility, we showed how Edward can use a variety of composable inference methods, capture recent advances in variational inference and GAN, and finely control the inference algorithms. For efficiency, we showed how Edward leverages computational graphs to achieve fast, parallelizable computation, scales to massive data, and incurs no runtime overhead over handwritten code.

In present work, we are applying Edward as a research platform for developing new probabilistic models (Rudolph et al., 2016; Tran et al., 2017) and new inference algorithms (Dieng et al., 2016). As with any language design, Edward makes tradeoffs in pursuit of its flexibility and speed for research. For example, an open challenge in Edward is to better facilitate programs with complex control flow and recursion. While possible to represent, it is unknown how to enable their flexible inference strategies. In addition, it is open how to expand Edward’s design to dynamic computational graph frameworks—which provide more flexibility in their programming paradigm—but may sacrifice performance. A crucial next step for probabilistic programming is to leverage dynamic computational graphs while maintaining the flexibility and efficiency that Edward offers.


We thank the probabilistic programming community—for sharing our enthusiasm and motivating further work—including developers of Church, Venture, Gamalon, Hakaru, and WebPPL. We also thank Stan developers for providing extensive feedback as we developed the language, as well as Thomas Wiecki for experimental details. We thank the Google BayesFlow team—Joshua Dillon, Ian Langmore, Ryan Sepassi, and Srinivas Vasudevan—as well as Amr Ahmed, Matthew Johnson, Hung Bui, Rajesh Ranganath, Maja Rudolph, and Francisco Ruiz for their helpful feedback. This work is supported by NSF IIS-1247664, ONR N00014-11-1-0651, DARPA FA8750-14-2-0009, DARPA N66001-15-C-4032, Adobe, Google, NSERC PGS-D, and the Sloan Foundation.


  • Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zhang. TensorFlow: A system for large-scale machine learning. arXiv preprint arXiv:1605.08695, 2016.
  • Andrieu & Roberts (2009) Christophe Andrieu and Gareth O Roberts. The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics, pp. 697–725, 2009.
  • Binder et al. (1997) John Binder, Kevin Murphy, and Stuart Russell. Space-efficient inference in dynamic probabilistic networks. In

    International Joint Conference on Artificial Intelligence

    , 1997.
  • Bishop (2006) Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
  • Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. Latent Dirichlet Allocation. Journal of Machine Learning Research, 3:993–1022, 2003.
  • Bottou (2010) Léon Bottou.

    Large-scale machine learning with stochastic gradient descent.

    In Proceedings of COMPSTAT’2010, pp. 177–186. Springer, 2010.
  • Broderick et al. (2013) Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. Streaming Variational Bayes. In Neural Information Processing Systems, 2013.
  • Burda et al. (2016) Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov.

    Importance weighted autoencoders.

    In International Conference on Learning Representations, 2016.
  • Carpenter et al. (2016) Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of Statistical Software, 2016.
  • Chollet (2015) François Chollet. Keras., 2015.
  • Culler (1986) David E Culler. Dataflow architectures. Technical report, DTIC Document, 1986.
  • Dayan et al. (1995) Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The Helmholtz machine. Neural computation, 7(5):889–904, 1995.
  • Denton et al. (2015) Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a Laplacian pyramid of adversarial networks. In Neural Information Processing Systems, 2015.
  • Dieng et al. (2016) Adji B. Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David M. Blei. -divergence for approximate inference. In arXiv preprint arXiv:1611.00328, 2016.
  • Doucet et al. (2000) Arnaud Doucet, Simon Godsill, and Christophe Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing, 10(3):197–208, 2000.
  • Doucet et al. (2001) Arnaud Doucet, Nando De Freitas, and Neil Gordon. An introduction to sequential monte carlo methods. In Sequential Monte Carlo methods in practice, pp. 3–14. Springer, 2001.
  • Foti et al. (2014) Nicholas Foti, Jason Xu, Dillon Laird, and Emily Fox.

    Stochastic variational inference for hidden Markov models.

    In Neural Information Processing Systems, 2014.
  • Gelfand & Smith (1990) Alan E Gelfand and Adrian FM Smith. Sampling-based approaches to calculating marginal densities. Journal of the American statistical association, 85(410):398–409, 1990.
  • Ghahramani (2015) Zoubin Ghahramani. Probabilistic machine learning and artificial intelligence. Nature, 521(7553):452–459, 2015.
  • Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, M Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014.
  • Goodman et al. (2012) Noah Goodman, Vikash Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. Church: A language for generative models. In Uncertainty in Artificial Intelligence, 2012.
  • Gregor et al. (2015) Karol Gregor, Ivo Danihelka, Alex Graves, Danilo J Rezende, and Daan Wierstra.

    DRAW: A recurrent neural network for image generation.

    In International Conference on Machine Learning, 2015.
  • Gutmann & Hyvärinen (2010) M Gutmann and A Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Artificial Intelligence and Statistics, 2010.
  • Hinton (2002) Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002.
  • Hinton et al. (2006) Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527–1554, 2006.
  • Hoffman & Blei (2015) Matthew Hoffman and David M. Blei. Structured stochastic variational inference. In Artificial Intelligence and Statistics, 2015.
  • Hoffman et al. (2013) Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347, 2013.
  • Jia et al. (2014) Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
  • Johnson & Willsky (2014) Matthew Johnson and Alan S Willsky. Stochastic variational inference for Bayesian time series models. In International Conference on Machine Learning, 2014.
  • Jordan et al. (1999) M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183–233, 1999.
  • Kingma & Welling (2014) Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
  • Kingma et al. (2014) Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Neural Information Processing Systems, 2014.
  • Kingma et al. (2016) Diederik P Kingma, Tim Salimans, and Max Welling. Improving variational inference with inverse autoregressive flow. In Neural Information Processing Systems, 2016.
  • Kiselyov & Shan (2009) Oleg Kiselyov and Chung-Chieh Shan. Embedded probabilistic programming. In Domain-Specific Languages, pp. 360–384. Springer, 2009.
  • Koller & Friedman (2009) Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. MIT press, 2009.
  • Lake et al. (2016) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. arXiv preprint arXiv:1604.00289, 2016.
  • Li & Turner (2016) Yingzhen Li and Richard E Turner. Variational inference with Rényi divergence. In Neural Information Processing Systems, 2016.
  • Li et al. (2015) Yujia Li, Kevin Swersky, and Richard Zemel.

    Generative moment matching networks.

    In International Conference on Machine Learning, 2015.
  • Mansinghka et al. (2014) V Mansinghka, D Selsam, and Y Perov. Venture: A higher-order probabilistic programming platform with programmable inference., 2014.
  • McInerney et al. (2015) James McInerney, Rajesh Ranganath, and David M Blei.

    The Population Posterior and Bayesian Inference on Streams.

    In Neural Information Processing Systems, 2015.
  • Milch et al. (2005) Brian Milch, Bhaskara Marthi, Stuart Russell, David Sontag, Daniel L Ong, and Andrey Kolobov. Blog. In International Joint Conference on Artificial Intelligence, 2005.
  • Minka et al. (2014) T. Minka, J.M. Winn, J.P. Guiver, S. Webster, Y. Zaykov, B. Yangel, A. Spengler, and J. Bronskill. Infer.NET 2.6, 2014. Microsoft Research Cambridge.
  • Minka (2001) Thomas P Minka. Expectation propagation for approximate Bayesian inference. In Uncertainty in Artificial Intelligence, 2001.
  • Murphy (2001) Kevin Murphy. The Bayes net toolbox for Matlab. Computing Science and Statistics, 33(2):1024–1034, 2001.
  • Neal (2011) Radford M Neal. MCMC using Hamiltonian dynamics.

    Handbook of Markov Chain Monte Carlo

    , 2011.
  • Neal & Hinton (1993) Radford M. Neal and Geoffrey E. Hinton. A new view of the EM algorithm that justifies incremental and other variants. In Learning in Graphical Models, pp. 355–368. Kluwer Academic Publishers, 1993.
  • Paisley et al. (2012) John Paisley, David M. Blei, and Michael Jordan. Variational Bayesian inference with stochastic search. In International Conference on Machine Learning, 2012.
  • Pfeffer (2001) Avi Pfeffer. IBAL: A probabilistic rational programming language. In International Joint Conference on Artificial Intelligence

    , pp. 733–740. Citeseer, 2001.

  • Pfeffer (2009) Avi Pfeffer. Figaro: An object-oriented probabilistic programming language. Charles River Analytics Technical Report, 137, 2009.
  • Plummer (2003) Martyn Plummer. JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In International Workshop on Distributed Statistical Computing, 2003.
  • Ranganath et al. (2014) Rajesh Ranganath, Sean Gerrish, and David M. Blei. Black box variational inference. In Artificial Intelligence and Statistics, 2014.
  • Ranganath et al. (2016a) Rajesh Ranganath, Jaan Altosaar, Dustin Tran, and David M. Blei. Operator variational inference. In Neural Information Processing Systems, 2016a.
  • Ranganath et al. (2016b) Rajesh Ranganath, Dustin Tran, and David M Blei. Hierarchical variational models. In International Conference on Machine Learning, 2016b.
  • Rezende & Mohamed (2015) Danilo J Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, 2015.
  • Rezende et al. (2014) Danilo J Rezende, Shakir Mohamed, and Daan Wierstra.

    Stochastic backpropagation and approximate inference in deep generative models.

    In International Conference on Machine Learning, 2014.
  • Rezende et al. (2016) Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. One-shot generalization in deep generative models. In International Conference on Machine Learning, 2016.
  • Ritchie et al. (2016) Daniel Ritchie, Paul Horsfall, and Noah D Goodman. Deep amortized inference for probabilistic programs. arXiv preprint arXiv:1610.05735, 2016.
  • Robert & Casella (1999) Christian P Robert and George Casella. Monte Carlo Statistical Methods. Springer, 1999.
  • Rudolph et al. (2016) Maja R Rudolph, Francisco J R Ruiz, Stephan Mandt, and David M Blei. Exponential family embeddings. In Neural Information Processing Systems, 2016.
  • Salakhutdinov & Murray (2008) Ruslan Salakhutdinov and Iain Murray.

    On the quantitative analysis of deep belief networks.

    In International Conference on Machine Learning, 2008.
  • Salvatier et al. (2015) John Salvatier, Thomas Wiecki, and Christopher Fonnesbeck. Probabilistic Programming in Python using PyMC. arXiv preprint arXiv:1507.08050, 2015.
  • Ścibior et al. (2015) Adam Ścibior, Zoubin Ghahramani, and Andrew D Gordon. Practical probabilistic programming with monads. In the 8th ACM SIGPLAN Symposium, pp. 165–176, New York, New York, USA, 2015. ACM Press.
  • Spiegelhalter et al. (1995) David J Spiegelhalter, Andrew Thomas, Nicky G Best, and Wally R Gilks. BUGS: Bayesian inference using Gibbs sampling, version 0.50. MRC Biostatistics Unit, Cambridge, 1995.
  • Stuhlmüller (2012) Andreas Stuhlmüller. Forest: A repository for generative models, 2012. URL
  • Tran et al. (2016a) Dustin Tran, Alp Kucukelbir, Adji B. Dieng, Maja Rudolph, Dawen Liang, and David M. Blei. Edward: A library for probabilistic modeling, inference, and criticism. arXiv preprint arXiv:1610.09787, 2016a.
  • Tran et al. (2016b) Dustin Tran, Rajesh Ranganath, and David M. Blei. The variational Gaussian process. In International Conference on Learning Representations, 2016b.
  • Tran et al. (2017) Dustin Tran, Rajesh Ranganath, and David M Blei. Deep and hierarchical implicit models. arXiv preprint arXiv:1702.08896, 2017.
  • Tristan et al. (2014) Jean-Baptiste Tristan, Daniel Huang, Joseph Tassarotti, Adam C Pocock, Stephen Green, and Guy L Steele. Augur: Data-parallel probabilistic modeling. In Neural Information Processing Systems, 2014.
  • Wang & Blei (2012) Chong Wang and David M Blei. Truncation-free online variational inference for Bayesian nonparametric models. In Neural Information Processing Systems, pp. 413–421, 2012.
  • Welling & Teh (2011) Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, 2011.
  • Wood et al. (2014) Frank Wood, Jan Willem van de Meent, and Vikash Mansinghka. A new approach to probabilistic programming inference. In Artificial Intelligence and Statistics, 2014.
  • Wu et al. (2016) Yi Wu, Lei Li, Stuart Russell, and Rastislav Bodik. Swift: Compiled inference for probabilistic programming languages. arXiv preprint arXiv:1606.09242, 2016.
  • Zinkov & Shan (2016) Robert Zinkov and Chung-chieh Shan. Composing inference algorithms as program transformations. arXiv preprint arXiv:1603.01882, 2016.

Appendix A Model Examples

There are many examples available at, including models, inference methods, and complete scripts. Below we describe several model examples; Appendix B describes an inference example (stochastic variational inference); Appendix C describes complete scripts. All examples in this paper are comprehensive, only leaving out import statements and fixed values. See the companion webpage for this paper ( for examples in a machine-readable format with runnable code.

a.1 Bayesian Neural Network for Classification

A Bayesian neural network is a neural network with a prior distribution on its weights.

Define the likelihood of an observation with binary label as

where is a 2-layer neural network whose weights and biases form the latent variables . Define the prior on the weights and biases to be the standard normal. See Figure 10. There are data points, features, and hidden units.

1W_0 = Normal(mu=tf.zeros([D, H]), sigma=tf.ones([D, H]))
2W_1 = Normal(mu=tf.zeros([H, 1]), sigma=tf.ones([H, 1]))
3b_0 = Normal(mu=tf.zeros(H), sigma=tf.ones(H))
4b_1 = Normal(mu=tf.zeros(1), sigma=tf.ones(1))
6x = tf.placeholder(tf.float32, [N, D])
7y = Bernoulli(logits=tf.matmul(tf.nn.tanh(tf.matmul(x, W_0) + b_0), W_1) + b_1)
Figure 10: Bayesian neural network for classification.

a.2 Latent Dirichlet Allocation

See Figure 11. Note that the program is written for illustration. We recommend vectorization in practice: instead of storing scalar random variables in lists of lists, one should prefer to represent few random variables, each which have many dimensions.

1D = 4  # number of documents
2N = [11502, 213, 1523, 1351]  # words per doc
3K = 10  # number of topics
4V = 100000  # vocabulary size
6theta = Dirichlet(alpha=tf.zeros([D, K]) + 0.1)
7phi = Dirichlet(alpha=tf.zeros([K, V]) + 0.05)
8z = [[0] * N] * D
9w = [[0] * N] * D
10for d in range(D):
11  for n in range(N[d]):
12    z[d][n] = Categorical(pi=theta[d, :])
13    w[d][n] = Categorical(pi=phi[z[d][n], :])
Figure 11: Latent Dirichlet allocation (Blei et al., 2003).

a.3 Gaussian Matrix Factorizationn

See Figure 12.

1N = 10
2M = 10
3K = 5  # latent dimension
5U = Normal(mu=tf.zeros([M, K]), sigma=tf.ones([M, K]))
6V = Normal(mu=tf.zeros([N, K]), sigma=tf.ones([N, K]))
7Y = Normal(mu=tf.matmul(U, V, transpose_b=True), sigma=tf.ones([N, M]))
Figure 12: Gaussian matrix factorization.

a.4 Dirichlet Process Mixture Model

See Figure 13.

A Dirichlet process mixture model is written as follows:

1mu = DirichletProcess(alpha=0.1, base_cls=Normal, mu=tf.zeros(D), sigma=tf.ones(D), sample_n=N)
2x = Normal(mu=mu, sigma=tf.ones([N, D]))

where mu has shape (N, D). The DirichletProcess random variable returns sample_n=N draws, each with shape given by the base distribution Normal(mu, sigma). The essential component defining the DirichletProcess random variable is a stochastic while loop. We define it below. See Edward’s code base for a more involved version with a base distribution.

1def dirichlet_process(alpha):
2  def cond(k, beta_k):
3    flip = Bernoulli(p=beta_k)
4    return tf.equal(flip, tf.constant(1))
6  def body(k, beta_k):
7    beta_k = beta_k * Beta(a=1.0, b=alpha)
8    return k + 1, beta_k
10  k = tf.constant(0)
11  beta_k = Beta(a=1.0, b=alpha)
12  stick_num, stick_beta = tf.while_loop(cond, body, loop_vars=[k, beta_k])
13  return stick_num
Figure 13: Dirichlet process mixture model.

Appendix B Inference Example: Stochastic Variational Inference

In the subgraph setting, we do data subsampling while working with a subgraph of the full model. This setting is necessary when the data and model do not fit in memory. It is scalable in that both the algorithm’s computational complexity (per iteration) and memory complexity are independent of the data set size.

For the code, we use the running example, a mixture model described in Figure 5.

1N = 10000000  # data set size
2D = 2  # data dimension
3K = 5  # number of clusters

The model is

To avoid memory issues, we work on only a subgraph of the model,

1M = 128  # mini-batch size
3beta = Normal(mu=tf.zeros([K, D]), sigma=tf.ones([K, D]))
4z = Categorical(logits=tf.zeros([M, K]))
5x = Normal(mu=tf.gather(beta, z), sigma=tf.ones([M, D]))

Assume the variational model is

parameterized by . Again, we work on only a subgraph of the model,

parameterized by . Importantly, only parameters are stored in memory for rather than .

1qbeta = Normal(mu=tf.Variable(tf.zeros([K, D])),
2               sigma=tf.nn.softplus(tf.Variable(tf.zeros[K, D])))
3qz_variables = tf.Variable(tf.zeros([M, K]))
4qz = Categorical(logits=qz_variables)

We use KLqp, a variational method that minimizes the divergence measure (Jordan et al., 1999). We instantiate two algorithms: a global inference over given the subset of and a local inference over the subset of given . We also pass in a TensorFlow placeholder x_ph for the data, so we can change the data at each step.

1x_ph = tf.placeholder(tf.float32, [M])
2inference_global = ed.KLqp({beta: qbeta}, data={x: x_ph, z: qz})
3inference_local = ed.KLqp({z: qz}, data={x: x_ph, beta: qbeta})

We initialize the algorithms with the scale argument, so that computation on z and x

will be scaled appropriately. This enables unbiased estimates for stochastic gradients.

1inference_global.initialize(scale={x: float(N) / M, z: float(N) / M})
2inference_local.initialize(scale={x: float(N) / M, z: float(N) / M})

We now run the algorithm, assuming there is a next_batch function which provides the next batch of data.

1qz_init = tf.initialize_variables([qz_variables])
2for _ in range(1000):
3  x_batch = next_batch(size=M)
4  for _ in range(10):  # make local inferences
5    inference_local.update(feed_dict={x_ph: x_batch})
7  # update global parameters
8  inference_global.update(feed_dict={x_ph: x_batch})
9  # reinitialize the local factors

After each iteration, we also reinitialize the parameters for ; this is because we do inference on a new set of local variational factors for each batch. This demo readily applies to other inference algorithms such as SGLD (stochastic gradient Langevin dynamics): simply replace qbeta and qz with Empirical random variables; then call ed.SGLD instead of ed.KLqp.

Note that if the data and model fit in memory but you’d still like to perform data subsampling for fast inference, we recommend not defining subgraphs. You can reify the full model, and simply index the local variables with a placeholder. The placeholder is fed at runtime to determine which of the local variables to update at a time. (For more details, see the website’s API.)

Appendix C Complete Examples

c.1 Variational Auto-encoder

See Figure 14.

1import edward as ed
2import tensorflow as tf
4from edward.models import Bernoulli, Normal
5from scipy.misc import imsave
6from tensorflow.contrib import slim
7from tensorflow.examples.tutorials.mnist import input_data
9M = 100  # batch size during training
10d = 2  # latent variable dimension
12# Probability model (subgraph)
13z = Normal(mu=tf.zeros([M, d]), sigma=tf.ones([M, d]))
14h = Dense(256, activation=’relu’)(z)
15x = Bernoulli(logits=Dense(28 * 28, activation=None)(h))
17# Variational model (subgraph)
18x_ph = tf.placeholder(tf.float32, [M, 28 * 28])
19qh = Dense(256, activation=’relu’)(x_ph)
20qz = Normal(mu=Dense(d, activation=None)(qh),
21            sigma=Dense(d, activation=’softplus’)(qh))
23# Bind p(x, z) and q(z | x) to the same TensorFlow placeholder for x.
24mnist = input_data.read_data_sets("data/mnist", one_hot=True)
25data = {x: x_ph}
27inference = ed.KLqp({z: qz}, data)
28optimizer = tf.train.RMSPropOptimizer(0.01, epsilon=1.0)
33n_epoch = 100
34n_iter_per_epoch = 1000
35for _ in range(n_epoch):
36  for _ in range(n_iter_per_epoch):
37    x_train, _ = mnist.train.next_batch(M)
38    info_dict = inference.update(feed_dict={x_ph: x_train})
40  # Generate images.
41  imgs = x.value().eval()
42  for m in range(M):
43    imsave("img/%d.png" % m, imgs[m].reshape(28, 28))
Figure 14: Complete script for a VAE (Kingma & Welling, 2014) with batch training. It generates MNIST digits after every 1000 updates.

c.2 Probabilistic Model for Word Embeddings

1import edward as ed
2import tensorflow as tf
4from edward.models import Bernoulli, Normal, PointMass
6N = 581238  # number of total words
7M = 128  # batch size during training
8K = 100  # number of factors
9ns = 3  # number of negative samples
10cs = 4  # context size
11L = 50000  # vocabulary size
13# Prior over embedding vectors
14p_rho = Normal(mu=tf.zeros([M, K]),
15               sigma=tf.sqrt(N) * tf.ones([M, K]))
16n_rho = Normal(mu=tf.zeros([M, ns, K]),
17               sigma=tf.sqrt(N) * tf.ones([M, ns, K]))
19# Prior over context vectors
20ctx_alphas = Normal(mu=tf.zeros([M, cs, K]),
21                    sigma=tf.sqrt(N)*tf.ones([M, cs, K]))
23# Conditional likelihoods
24ctx_sum = tf.reduce_sum(ctx_alphas, [1])
25p_eta = tf.expand_dims(tf.reduce_sum(p_rho * ctx_sum, -1),1)
26n_eta = tf.reduce_sum(n_rho * tf.tile(tf.expand_dims(ctx_sum, 1), [1, ns, 1]), -1)
27y_pos = Bernoulli(logits=p_eta)
28y_neg = Bernoulli(logits=n_eta)
30# placeholders for batch training
31p_idx = tf.placeholder(tf.int32, [M, 1])
32n_idx = tf.placeholder(tf.int32, [M, ns])
33ctx_idx = tf.placeholder(tf.int32, [M, cs])
35# Variational parameters (embedding vectors)
36rho_params = tf.Variable(tf.random_normal([L, K]))
37alpha_params = tf.Variable(tf.random_normal([L, K]))
39# Variational distribution on embedding vectors
40q_p_rho = PointMass(params=tf.squeeze(tf.gather(rho_params, p_idx)))
41q_n_rho = PointMass(params=tf.gather(rho_params, n_idx))
42q_alpha = PointMass(params=tf.gather(alpha_params, ctx_idx))
44inference = ed.MAP(
45  {p_rho: q_p_rho, n_rho: q_n_rho, ctx_alphas: q_alpha},
46  data={y_pos: tf.ones((M, 1)), y_neg: tf.zeros((M, ns))})
51for _ in range(inference.n_iter):
52  targets, windows, negatives = next_batch(M)  # a function to generate data
53  info_dict = inference.update(feed_dict={p_idx: targets, ctx_idx: windows, n_idx: negatives})
54  inference.print_progress(info_dict)
Figure 15: Exponential family embedding for binary data (Rudolph et al., 2016). Here, MAP is used to maximize the total sum of conditional log-likelihoods and log-priors.

See Figure 15. This example uses data subsampling (Section 4.4). The priors and conditional likelihoods are defined only for a minibatch of data. Similarly the variational model only models the embeddings used in a given minibatch. TensorFlow variables contain the embedding vectors for the entire vocabulary. TensorFlow placeholders ensure that the correct embedding vectors are used as variational parameters for a given minibatch.

The Bernoulli variables y_pos and y_neg are fixed to be ’s and ’s respectively. They model whether a word is indeed the target word for a given context window or has been drawn as a negative sample. Without regularization (via priors), the objective we optimize is identical to negative sampling.