Poster: Benchmarking Financial Data Feed Systems

10/29/2020
by   Manuel Coenen, et al.
0

Data-driven solutions for the investment industry require event-based backend systems to process high-volume financial data feeds with low latency, high throughput, and guaranteed delivery modes. At vwd we process an average of 18 billion incoming event notifications from 500+ data sources for 30 million symbols per day and peak rates of 1+ million notifications per second using custom-built platforms that keep audit logs of every event. We currently assess modern open source event-processing platforms such as Kafka, NATS, Redis, Flink or Storm for the use in our ticker plant to reduce the maintenance effort for cross-cutting concerns and leverage hybrid deployment models. For comparability and repeatability we benchmark candidates with a standardized workload we derived from our real data feeds. We have enhanced an existing light-weight open source benchmarking tool in its processing, logging, and reporting capabilities to cope with our workloads. The resulting tool wrench can simulate workloads or replay snapshots in volume and dynamics like those we process in our ticker plant. We provide the tool as open source. As part of ongoing work we contribute details on (a) our workload and requirements for benchmarking candidate platforms for financial feed processing; (b) the current state of the tool wrench.

READ FULL TEXT

page 1

page 2

research
10/29/2020

Poster: A Real-World Distributed Infrastructure for Processing Financial Data at Scale

Financial markets are event- and data-driven to an extremely high degree...
research
06/23/2022

The DEBS 2022 Grand Challenge: Detecting Trading Trends in Financial Tick Data

The DEBS Grand Challenge (GC) is an annual programming competition open ...
research
08/08/2019

Managing the Complexity of Processing Financial Data at Scale – an Experience Report

Financial markets are extremely data-driven and regulated. Participants ...
research
11/18/2019

Understanding Open Source Serverless Platforms: Design Considerations and Performance

Serverless computing is increasingly popular because of the promise of l...
research
12/09/2020

JANUS: Benchmarking Commercial and Open-Source Cloud and Edge Platforms for Object and Anomaly Detection Workloads

With diverse IoT workloads, placing compute and analytics close to where...
research
04/10/2020

On Strong Scaling and Open Source Tools for Analyzing Atom Probe Tomography Data

Atom probe tomography (APT) has matured to a versatile nanoanalytical ch...

Please sign up or login with your details

Forgot password? Click here to reset