Ranking and benchmarking framework for sampling algorithms on synthetic data streams

06/17/2020
by   József Dániel Gáspár, et al.
0

In the fields of big data, AI, and streaming processing, we work with large amounts of data from multiple sources. Due to memory and network limitations, we process data streams on distributed systems to alleviate computational and network loads. When data streams with non-uniform distributions are processed, we often observe overloaded partitions due to the use of simple hash partitioning. To tackle this imbalance, we can use dynamic partitioning algorithms that require a sampling algorithm to precisely estimate the underlying distribution of the data stream. There is no standardized way to test these algorithms. We offer an extensible ranking framework with benchmark and hyperparameter optimization capabilities and supply our framework with a data generator that can handle concept drifts. Our work includes a generator for dynamic micro-bursts that we can apply to any data stream. We provide algorithms that react to concept drifts and compare those against the state-of-the-art algorithms using our framework.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/26/2022

RMLStreamer-SISO: an RDF stream generator from streaming heterogeneous data

Stream-reasoning query languages such as CQELS and C-SPARQL enable query...
research
05/31/2021

System-aware dynamic partitioning for batch and streaming workloads

When processing data streams with highly skewed and nonstationary key di...
research
12/16/2018

"When and Where?": Behavior Dominant Location Forecasting with Micro-blog Streams

The proliferation of smartphones and wearable devices has increased the ...
research
03/02/2022

Pattern Recognition and Event Detection on IoT Data-streams

Big data streams are possibly one of the most essential underlying notio...
research
10/04/2022

Sampling Streaming Data with Parallel Vector Quantization – PVQ

Accumulation of corporate data in the cloud has attracted more enterpris...
research
11/17/2016

Stream Packing for Asynchronous Multi-Context Systems using ASP

When a processing unit relies on data from external streams, we may face...
research
01/24/2023

Distinct Elements in Streams: An Algorithm for the (Text) Book

Given a data stream 𝒟 = ⟨ a_1, a_2, …, a_m ⟩ of m elements where each a_...

Please sign up or login with your details

Forgot password? Click here to reset