Fast Generating A Large Number of Gumbel-Max Variables

02/02/2020
by   Yiyan Qi, et al.
0

The well-known Gumbel-Max Trick for sampling elements from a categorical distribution (or more generally a nonnegative vector) and its variants have been widely used in areas such as machine learning and information retrieval. To sample a random element i (or a Gumbel-Max variable i) in proportion to its positive weight v_i, the Gumbel-Max Trick first computes a Gumbel random variable g_i for each positive weight element i, and then samples the element i with the largest value of g_i+ln v_i. Recently, applications including similarity estimation and graph embedding require to generate k independent Gumbel-Max variables from high dimensional vectors. However, it is computationally expensive for a large k (e.g., hundreds or even thousands) when using the traditional Gumbel-Max Trick. To solve this problem, we propose a novel algorithm, FastGM, that reduces the time complexity from O(kn^+) to O(k ln k + n^+), where n^+ is the number of positive elements in the vector of interest. Instead of computing k independent Gumbel random variables directly, we find that there exists a technique to generate these variables in descending order. Using this technique, our method FastGM computes variables g_i+ln v_i for all positive elements i in descending order. As a result, FastGM significantly reduces the computation time because we can stop the procedure of Gumbel random variables computing for many elements especially for those with small weights. Experiments on a variety of real-world datasets show that FastGM is orders of magnitude faster than state-of-the-art methods without sacrificing accuracy and incurring additional expenses.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/10/2023

Fast Gumbel-Max Sketch and its Applications

The well-known Gumbel-Max Trick for sampling elements from a categorical...
research
01/12/2015

A fast numerical method for max-convolution and the application to efficient max-product inference in Bayesian networks

Observations depending on sums of random variables are common throughout...
research
02/10/2016

High Dimensional Inference with Random Maximum A-Posteriori Perturbations

This paper presents a new approach, called perturb-max, for high-dimensi...
research
04/24/2020

Differential Network Learning Beyond Data Samples

Learning the change of statistical dependencies between random variables...
research
06/11/2023

Examining Collinearities

The cos-max method is a little-known method of identifying collinearitie...
research
05/31/2018

Simulation of Random Variables under Rényi Divergence Measures of All Orders

The random variable simulation problem consists in using a k-dimensional...
research
04/01/2021

Fast Jacobian-Vector Product for Deep Networks

Jacobian-vector products (JVPs) form the backbone of many recent develop...

Please sign up or login with your details

Forgot password? Click here to reset