Bitmap Filter: Speeding up Exact Set Similarity Joins with Bitwise Operations

11/20/2017
by   Edans F. O. Sandes, et al.
0

The Exact Set Similarity Join problem aims to find all similar sets between two collections of sets, with respect to a threshold and a similarity function such as overlap, Jaccard, dice or cosine. The naive approach verifies all pairs of sets and it is often considered impractical due the high number of combinations. So, Exact Set Similarity Join algorithms are usually based on the Filter-Verification Framework, that applies a series of filters to reduce the number of verified pairs. This paper presents a new filtering technique called Bitmap Filter, which is able to accelerate state-of-the-art algorithms for the exact Set Similarity Join problem. The Bitmap Filter uses hash functions to create bitmaps of fixed b bits, representing characteristics of the sets. Then, it applies bitwise operations (such as xor and population count) on the bitmaps in order to infer a similarity upper bound for each pair of sets. If the upper bound is below a given similarity threshold, the pair of sets is pruned. The Bitmap Filter benefits from the fact that bitwise operations are efficiently implemented by many modern general-purpose processors and it was easily applied to four state-of-the-art algorithms implemented in CPU: AllPairs, PPJoin, AdaptJoin and GroupJoin. Furthermore, we propose a Graphic Processor Unit (GPU) algorithm based on the naive approach but using the Bitmap Filter to speedup the computation. The experiments considered 9 collections containing from 100 thousands up to 10 million sets and the joins were made using Jaccard thresholds from 0.50 to 0.95. The Bitmap Filter was able to improve 90 experiments in CPU, with speedups of up to 4.50x and 1.43x on average. Using the GPU algorithm, the experiments were able to speedup the original CPU algorithms by up to 577x using an Nvidia Geforce GTX 980 Ti.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/21/2017

Scalable and robust set similarity join

Set similarity join is a fundamental and well-studied database operator....
research
12/21/2018

Speeding-up the Verification Phase of Set Similarity Joins in the GPGPU paradigm

We investigate the problem of exact set similarity joins using a co-proc...
research
10/29/2018

Efficient Taxonomic Similarity Joins with Adaptive Overlap Constraint

A similarity join aims to find all similar pairs between two collections...
research
04/25/2019

GPU-based Efficient Join Algorithms on Hadoop

The growing data has brought tremendous pressure for query processing an...
research
03/06/2020

LSF-Join: Locality Sensitive Filtering for Distributed All-Pairs Set Similarity Under Skew

All-pairs set similarity is a widely used data mining task, even for lar...
research
06/13/2017

Preference-driven Similarity Join

Similarity join, which can find similar objects (e.g., products, names, ...
research
04/20/2023

KOIOS: Top-k Semantic Overlap Set Search

We study the top-k set similarity search problem using semantic overlap....

Please sign up or login with your details

Forgot password? Click here to reset