C-MinHash: Rigorously Reducing K Permutations to Two

09/07/2021
by   Xiaoyun Li, et al.
14

Minwise hashing (MinHash) is an important and practical algorithm for generating random hashes to approximate the Jaccard (resemblance) similarity in massive binary (0/1) data. The basic theory of MinHash requires applying hundreds or even thousands of independent random permutations to each data vector in the dataset, in order to obtain reliable results for (e.g.,) building large-scale learning models or approximate near neighbor search in massive data. In this paper, we propose Circulant MinHash (C-MinHash) and provide the surprising theoretical results that we just need two independent random permutations. For C-MinHash, we first conduct an initial permutation on the data vector, then we use a second permutation to generate hash values. Basically, the second permutation is re-used K times via circulant shifting to produce K hashes. Unlike classical MinHash, these K hashes are obviously correlated, but we are able to provide rigorous proofs that we still obtain an unbiased estimate of the Jaccard similarity and the theoretical variance is uniformly smaller than that of the classical MinHash with K independent permutations. The theoretical proofs of C-MinHash require some non-trivial efforts. Numerical experiments are conducted to justify the theory and demonstrate the effectiveness of C-MinHash.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/10/2021

C-MinHash: Practically Reducing Two Permutations to Just One

Traditional minwise hashing (MinHash) requires applying K independent pe...
research
11/18/2021

C-OPH: Improving the Accuracy of One Permutation Hashing (OPH) with Circulant Permutations

Minwise hashing (MinHash) is a classical method for efficiently estimati...
research
06/13/2023

Differentially Private One Permutation Hashing and Bin-wise Consistent Weighted Sampling

Minwise hashing (MinHash) is a standard algorithm widely used in the ind...
research
08/06/2012

One Permutation Hashing for Efficient Search and Learning

Recently, the method of b-bit minwise hashing has been applied to large-...
research
05/23/2020

DartMinHash: Fast Sketching for Weighted Sets

Weighted minwise hashing is a standard dimensionality reduction techniqu...
research
02/21/2016

2-Bit Random Projections, NonLinear Estimators, and Approximate Near Neighbor Search

The method of random projections has become a standard tool for machine ...

Please sign up or login with your details

Forgot password? Click here to reset