Similarity Join and Self-Join Size Estimation in a Streaming Environment

06/08/2018
by   Davood Rafiei, et al.
0

We study the problem of similarity self-join and similarity join size estimation in a streaming setting where the goal is to estimate, in one scan of the input and with sublinear space in the input size, the number of record pairs that have a similarity within a given threshold. The problem has many applications in data cleaning and query plan generation, where the cost of a similarity join may be estimated before actually doing the join. On unary input where two records either match or don't match, the problem becomes join and self-join size estimation for which one-pass algorithms are readily available. Our work addresses the problem for d-ary input, for d >= 1, where the degree of similarity can vary from 1 to d. We show that our proposed algorithm gives an accurate estimate and scales well with the input size. We provide error bounds and time and space costs, and conduct an extensive experimental evaluation of our algorithm, comparing its estimation accuracy to a few competitors, including some multi-pass algorithms. Our results show that given the same space, the proposed algorithm has an order of magnitude less error for a large range of similarity thresholds.

READ FULL TEXT
research
06/08/2018

Similarity Join and Similarity Self-Join Size Estimation in a Streaming Environment

We study the problem of similarity self-join and similarity join size es...
research
04/16/2011

Similarity Join Size Estimation using Locality Sensitive Hashing

Similarity joins are important operations with a broad range of applicat...
research
06/13/2017

Preference-driven Similarity Join

Similarity join, which can find similar objects (e.g., products, names, ...
research
05/15/2019

Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

Given two sets of objects, metric similarity join finds all similar pair...
research
03/07/2021

Auto-FuzzyJoin: Auto-Program Fuzzy Similarity Joins Without Labeled Examples

Fuzzy similarity join is an important database operator widely used in p...
research
07/02/2018

Distributed Statistical Estimation of Matrix Products with Applications

We consider statistical estimations of a matrix product over the integer...
research
10/29/2018

Efficient Taxonomic Similarity Joins with Adaptive Overlap Constraint

A similarity join aims to find all similar pairs between two collections...

Please sign up or login with your details

Forgot password? Click here to reset