DeepAI AI Chat
Log In Sign Up

Augment and Reduce: Stochastic Inference for Large Categorical Distributions

02/12/2018
by   Francisco J. R. Ruiz, et al.
University of Cambridge
Columbia University
0

Categorical distributions are ubiquitous in machine learning, e.g., in classification, language models, and recommendation systems. They are also at the core of discrete choice models. However, when the number of possible outcomes is very large, using categorical distributions becomes computationally expensive, as the complexity scales linearly with the number of outcomes. To address this problem, we propose augment and reduce (A&R), a method to alleviate the computational complexity. A&R uses two ideas: latent variable augmentation and stochastic variational inference. It maximizes a lower bound on the marginal likelihood of the data. Unlike existing methods which are specific to softmax, A&R is more general and is amenable to other categorical models, such as multinomial probit. On several large-scale classification problems, we show that A&R provides a tighter bound on the marginal likelihood and has better predictive performance than existing approaches.

READ FULL TEXT

page 1

page 2

page 3

page 4

09/23/2016

One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities

The softmax representation of probabilities for categorical variables pl...
06/28/2019

The Thermodynamic Variational Objective

We introduce the thermodynamic variational objective (TVO) for learning ...
04/07/2022

Categorical Distributions of Maximum Entropy under Marginal Constraints

The estimation of categorical distributions under marginal constraints s...
03/29/2021

Variational Rejection Particle Filtering

We present a variational inference (VI) framework that unifies and lever...
11/12/2018

Modeling Text Complexity using a Multi-Scale Probit

We present a novel model for text complexity analysis which can be fitte...
05/09/2022

ReCAB-VAE: Gumbel-Softmax Variational Inference Based on Analytic Divergence

The Gumbel-softmax distribution, or Concrete distribution, is often used...
12/01/2020

Field-wise Learning for Multi-field Categorical Data

We propose a new method for learning with multi-field categorical data. ...