C-MinHash: Practically Reducing Two Permutations to Just One

by   Xiaoyun Li, et al.

Traditional minwise hashing (MinHash) requires applying K independent permutations to estimate the Jaccard similarity in massive binary (0/1) data, where K can be (e.g.,) 1024 or even larger, depending on applications. The recent work on C-MinHash (Li and Li, 2021) has shown, with rigorous proofs, that only two permutations are needed. An initial permutation is applied to break whatever structures which might exist in the data, and a second permutation is re-used K times to produce K hashes, via a circulant shifting fashion. (Li and Li, 2021) has proved that, perhaps surprisingly, even though the K hashes are correlated, the estimation variance is strictly smaller than the variance of the traditional MinHash. It has been demonstrated in (Li and Li, 2021) that the initial permutation in C-MinHash is indeed necessary. For the ease of theoretical analysis, they have used two independent permutations. In this paper, we show that one can actually simply use one permutation. That is, one single permutation is used for both the initial pre-processing step to break the structures in the data and the circulant hashing step to generate K hashes. Although the theoretical analysis becomes very complicated, we are able to explicitly write down the expression for the expectation of the estimator. The new estimator is no longer unbiased but the bias is extremely small and has essentially no impact on the estimation accuracy (mean square errors). An extensive set of experiments are provided to verify our claim for using just one permutation.


page 1

page 2

page 3

page 4


C-OPH: Improving the Accuracy of One Permutation Hashing (OPH) with Circulant Permutations

Minwise hashing (MinHash) is a classical method for efficiently estimati...

C-MinHash: Rigorously Reducing K Permutations to Two

Minwise hashing (MinHash) is an important and practical algorithm for ge...

One Permutation Hashing for Efficient Search and Learning

Recently, the method of b-bit minwise hashing has been applied to large-...

OPORP: One Permutation + One Random Projection

Consider two D-dimensional data vectors (e.g., embeddings): u, v. In man...

Hierarchical One Permutation Hashing: Efficient Multimedia Near Duplicate Detection

With advances in multimedia technologies and the proliferation of smart ...

Six Permutation Patterns Force Quasirandomness

A sequence π_1,π_2,… of permutations is said to be "quasirandom" if the ...

A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness

The aggregation and denoising of crowd labeled data is a task that has g...

Please sign up or login with your details

Forgot password? Click here to reset