Partition Functions from Rao-Blackwellized Tempered Sampling

03/07/2016
by   David Carlson, et al.
0

Partition functions of probability distributions are important quantities for model evaluation and comparisons. We present a new method to compute partition functions of complex and multimodal distributions. Such distributions are often sampled using simulated tempering, which augments the target space with an auxiliary inverse temperature variable. Our method exploits the multinomial probability law of the inverse temperatures, and provides estimates of the partition function in terms of a simple quotient of Rao-Blackwellized marginal inverse temperature probability estimates, which are updated while sampling. We show that the method has interesting connections with several alternative popular methods, and offers some significant advantages. In particular, we empirically find that the new method provides more accurate estimates than Annealed Importance Sampling when calculating partition functions of large Restricted Boltzmann Machines (RBM); moreover, the method is sufficiently accurate to track training and validation log-likelihoods during learning of RBMs, at minimal computational cost.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/23/2020

Efficient Evaluation of the Partition Function of RBMs with Annealed Importance Sampling

Probabilistic models based on Restricted Boltzmann Machines (RBMs) imply...
research
10/15/2016

Markov Chain Truncation for Doubly-Intractable Inference

Computing partition functions, the normalizing constants of probability ...
research
05/31/2019

Parallel Tempering via Simulated Tempering Without Normalizing Constants

In this paper we develop a new general Bayesian methodology that simulta...
research
11/11/2016

Annealing Gaussian into ReLU: a New Sampling Strategy for Leaky-ReLU RBM

Restricted Boltzmann Machine (RBM) is a bipartite graphical model that i...
research
05/19/2022

Continuously-Tempered PDMP Samplers

New sampling algorithms based on simulating continuous-time stochastic p...
research
03/15/2017

A New Unbiased and Efficient Class of LSH-Based Samplers and Estimators for Partition Function Computation in Log-Linear Models

Log-linear models are arguably the most successful class of graphical mo...
research
07/12/2017

Computing Entropies With Nested Sampling

The Shannon entropy, and related quantities such as mutual information, ...

Please sign up or login with your details

Forgot password? Click here to reset