Bayesian Allocation Model: Inference by Sequential Monte Carlo for Nonnegative Tensor Factorizations and Topic Models using Polya Urns

by   Ali Taylan Cemgil, et al.

We introduce a dynamic generative model, Bayesian allocation model (BAM), which establishes explicit connections between nonnegative tensor factorization (NTF), graphical models of discrete probability distributions and their Bayesian extensions, and the topic models such as the latent Dirichlet allocation. BAM is based on a Poisson process, whose events are marked by using a Bayesian network, where the conditional probability tables of this network are then integrated out analytically. We show that the resulting marginal process turns out to be a Polya urn, an integer valued self-reinforcing process. This urn processes, which we name a Polya-Bayes process, obey certain conditional independence properties that provide further insight about the nature of NTF. These insights also let us develop space efficient simulation algorithms that respect the potential sparsity of data: we propose a class of sequential importance sampling algorithms for computing NTF and approximating their marginal likelihood, which would be useful for model selection. The resulting methods can also be viewed as a model scoring method for topic models and discrete Bayesian networks with hidden variables. The new algorithms have favourable properties in the sparse data regime when contrasted with variational algorithms that become more accurate when the total sum of the elements of the observed tensor goes to infinity. We illustrate the performance on several examples and numerically study the behaviour of the algorithms for various data regimes.


An Empirical-Bayes Score for Discrete Bayesian Networks

Bayesian network structure learning is often performed in a Bayesian set...

Inference Networks for Sequential Monte Carlo in Graphical Models

We introduce a new approach for amortizing inference in directed graphic...

Learning high-dimensional probability distributions using tree tensor networks

We consider the problem of the estimation of a high-dimensional probabil...

Fast Bayesian inference in large Gaussian graphical models

Despite major methodological developments, Bayesian inference for Gaussi...

Posterior distributions for Hierarchical Spike and Slab Indian Buffet processes

Bayesian nonparametric hierarchical priors are highly effective in provi...

Scalable Bayesian Non-Negative Tensor Factorization for Massive Count Data

We present a Bayesian non-negative tensor factorization model for count-...

Prior and Likelihood Choices for Bayesian Matrix Factorisation on Small Datasets

In this paper, we study the effects of different prior and likelihood ch...

Please sign up or login with your details

Forgot password? Click here to reset