C-OPH: Improving the Accuracy of One Permutation Hashing (OPH) with Circulant Permutations

by   Xiaoyun Li, et al.

Minwise hashing (MinHash) is a classical method for efficiently estimating the Jaccrad similarity in massive binary (0/1) data. To generate K hash values for each data vector, the standard theory of MinHash requires K independent permutations. Interestingly, the recent work on "circulant MinHash" (C-MinHash) has shown that merely two permutations are needed. The first permutation breaks the structure of the data and the second permutation is re-used K time in a circulant manner. Surprisingly, the estimation accuracy of C-MinHash is proved to be strictly smaller than that of the original MinHash. The more recent work further demonstrates that practically only one permutation is needed. Note that C-MinHash is different from the well-known work on "One Permutation Hashing (OPH)" published in NIPS'12. OPH and its variants using different "densification" schemes are popular alternatives to the standard MinHash. The densification step is necessary in order to deal with empty bins which exist in One Permutation Hashing. In this paper, we propose to incorporate the essential ideas of C-MinHash to improve the accuracy of One Permutation Hashing. Basically, we develop a new densification method for OPH, which achieves the smallest estimation variance compared to all existing densification schemes for OPH. Our proposed method is named C-OPH (Circulant OPH). After the initial permutation (which breaks the existing structure of the data), C-OPH only needs a "shorter" permutation of length D/K (instead of D), where D is the original data dimension and K is the total number of bins in OPH. This short permutation is re-used in K bins in a circulant shifting manner. It can be shown that the estimation variance of the Jaccard similarity is strictly smaller than that of the existing (densified) OPH methods.


page 1

page 2

page 3

page 4


C-MinHash: Practically Reducing Two Permutations to Just One

Traditional minwise hashing (MinHash) requires applying K independent pe...

One Permutation Hashing for Efficient Search and Learning

Recently, the method of b-bit minwise hashing has been applied to large-...

C-MinHash: Rigorously Reducing K Permutations to Two

Minwise hashing (MinHash) is an important and practical algorithm for ge...

Hierarchical One Permutation Hashing: Efficient Multimedia Near Duplicate Detection

With advances in multimedia technologies and the proliferation of smart ...

Differentially Private One Permutation Hashing and Bin-wise Consistent Weighted Sampling

Minwise hashing (MinHash) is a standard algorithm widely used in the ind...

Robust Data Valuation via Variance Reduced Data Shapley

Data valuation, especially quantifying data value in algorithmic predict...

OPORP: One Permutation + One Random Projection

Consider two D-dimensional data vectors (e.g., embeddings): u, v. In man...

Please sign up or login with your details

Forgot password? Click here to reset