Quantitative Impact Evaluation of an Abstraction Layer for Data Stream Processing Systems

by   Guenter Hesse, et al.

With the demand to process ever-growing data volumes, a variety of new data stream processing frameworks have been developed. Moving an implementation from one such system to another, e.g., for performance reasons, requires adapting existing applications to new interfaces. Apache Beam addresses these high substitution costs by providing an abstraction layer that enables executing programs on any of the supported streaming frameworks. In this paper, we present a novel benchmark architecture for comparing the performance impact of using Apache Beam on three streaming frameworks: Apache Spark Streaming, Apache Flink, and Apache Apex. We find significant performance penalties when using Apache Beam for application development in the surveyed systems. Overall, usage of Apache Beam for the examined streaming applications caused a high variance of query execution times with a slowdown of up to a factor of 58 compared to queries developed without the abstraction layer. All developed benchmark artifacts are publicly available to ensure reproducible results.



There are no comments yet.


page 1

page 2

page 3

page 4


ESPBench: The Enterprise Stream Processing Benchmark

Growing data volumes and velocities in fields such as Industry 4.0 or th...

Apache Spark Streaming and HarmonicIO: A Performance and Architecture Comparison

Studies have demonstrated that Apache Spark, Flink and related framework...

Pilot-Streaming: A Stream Processing Framework for High-Performance Computing

An increasing number of scientific applications rely on stream processin...

Inviwo - A Visualization System with Usage Abstraction Levels

The complexity of today's visualization applications demands specific vi...

Smart Resource Management for Data Streaming using an Online Bin-packing Strategy

Data stream processing frameworks provide reliable and efficient mechani...

Portable high-order finite element kernels I: Streaming Operations

This paper is devoted to the development of highly efficient kernels per...

Accurate and confident prediction of electron beam longitudinal properties using spectral virtual diagnostics

Longitudinal phase space (LPS) provides a critical information about ele...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.