Railgun: streaming windows for mission critical systems

09/01/2020
by   João Oliveirinha, et al.
0

Some mission critical systems, such as fraud detection, require accurate, real-time metrics over long time windows on applications that demand high throughputs and low latencies. As these applications need to run "forever", cope with large and spiky data loads, they further require to be run in a distributed setting. Unsurprisingly, we are unaware of any distributed streaming system that provides all those properties. Instead, existing systems take large simplifications, such as implementing sliding windows as a fixed set of partially overlapping windows, jeopardizing metric accuracy (violating financial regulator rules) or latency (breaching service agreements). In this paper, we propose Railgun, a fault-tolerant, elastic, and distributed streaming system supporting real-time sliding windows for scenarios requiring high loads and millisecond-level latencies. We benchmarked an initial prototype of Railgun using real data, showing significant lower latency than Flink, and low memory usage, independent of window size.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/23/2021

Railgun: managing large streaming windows under MAD requirements

Some mission critical systems, e.g., fraud detection, require accurate, ...
research
09/15/2016

Virtualizing System and Ordinary Services in Windows-based OS-Level Virtual Machines

OS-level virtualization incurs smaller start-up and run-time overhead th...
research
10/05/2018

Memento: Making Sliding Windows Efficient for Heavy Hitters

Cloud operators require real-time identification of Heavy Hitters (HH) a...
research
12/08/2019

A study on Modern Messaging Systems- Kafka, RabbitMQ and NATS Streaming

Distributed messaging systems form the core of big data streaming, cloud...
research
12/19/2018

Fast Botnet Detection From Streaming Logs Using Online Lanczos Method

Botnet, a group of coordinated bots, is becoming the main platform of ma...
research
09/15/2016

Confining Windows Inter-Process Communications for OS-Level Virtual Machine

As OS-level virtualization technology usually imposes little overhead on...
research
08/13/2018

Interactive Launch of 16,000 Microsoft Windows Instances on a Supercomputer

Simulation, machine learning, and data analysis require a wide range of ...

Please sign up or login with your details

Forgot password? Click here to reset