HyperLogLog Sketch Acceleration on FPGA

05/24/2020
by   Amit Kulkarni, et al.
0

Data sketches are a set of widely used approximated data summarizing techniques. Their fundamental property is sub-linear memory complexity on the input cardinality, an important aspect when processing streams or data sets with a vast base domain (URLs, IP addresses, user IDs, etc.). Among the many data sketches available, HyperLogLog has become the reference for cardinality counting (how many distinct data items there are in a data set). Although it does not count every data item (to reduce memory consumption), it provides probabilistic guarantees on the result, and it is, thus, often used to analyze data streams. In this paper, we explore how to implement HyperLogLog on an FPGA to benefit from the parallelism available and the ability to process data streams coming from high-speed networks. Our multi-pipelined high-cardinality HyperLogLog implementation delivers 1.8x higher throughput than an optimized HyperLogLog running on a dual-socket Intel Xeon E5-2630 v3 system with a total of 16 cores and 32 hyper-threads.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/16/2019

A new Frequency Estimation Sketch for Data Streams

In data stream applications, one of the critical issues is to estimate t...
research
02/09/2016

Graphical Model Sketch

Structured high-cardinality data arises in many domains, and poses a maj...
research
10/23/2017

HyperMinHash: Jaccard index sketching in LogLog space

In this extended abstract, we describe and analyse a streaming probabili...
research
08/20/2020

Simple and Efficient Cardinality Estimation in Data Streams

We study sketching schemes for the cardinality estimation problem in dat...
research
09/12/2017

Data Sketches for Disaggregated Subset Sum and Frequent Item Estimation

We introduce and study a new data sketch for processing massive datasets...
research
02/10/2023

Predicting the cardinality of a reduced Gröbner basis

We use ansatz neural network models to predict key metrics of complexity...

Please sign up or login with your details

Forgot password? Click here to reset