Asymptotically Optimal Exact Minibatch Metropolis-Hastings

06/20/2020
by   Ruqi Zhang, et al.
1

Metropolis-Hastings (MH) is a commonly-used MCMC algorithm, but it can be intractable on large datasets due to requiring computations over the whole dataset. In this paper, we study minibatch MH methods, which instead use subsamples to enable scaling. We observe that most existing minibatch MH methods are inexact (i.e. they may change the target distribution), and show that this inexactness can cause arbitrarily large errors in inference. We propose a new exact minibatch MH method, , which exposes a tunable trade-off between its batch size and its theoretically guaranteed convergence rate. We prove a lower bound on the batch size that any minibatch MH method must use to retain exactness while guaranteeing fast convergence—the first such bound for minibatch MH—and show is asymptotically optimal in terms of the batch size. Empirically, we show outperforms other exact minibatch MH methods on robust linear regression, truncated Gaussian mixtures, and logistic regression.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

09/05/2013

Semistochastic Quadratic Bound Methods

Partition functions arise in a variety of settings, including conditiona...
08/07/2018

Fast Variance Reduction Method with Stochastic Batch Size

In this paper we study a family of variance reduction methods with rando...
11/19/2021

Convergence rates for Metropolis-Hastings algorithms in the Wasserstein distance

We develop necessary conditions for geometrically fast convergence in th...
11/05/2015

Stop Wasting My Gradients: Practical SVRG

We present and analyze several strategies for improving the performance ...
12/09/2017

Cost-Sensitive Approach to Batch Size Adaptation for Gradient Descent

In this paper, we propose a novel approach to automatically determine th...
05/20/2016

Coresets for Scalable Bayesian Logistic Regression

The use of Bayesian methods in large-scale data settings is attractive b...
02/22/2022

Adaptive Cholesky Gaussian Processes

We present a method to fit exact Gaussian process models to large datase...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.