Log In Sign Up

Online Analysis of Distributed Dataflows with Timely Dataflow

by   Malte Sandstede, et al.

We present ST2, an end-to-end solution to analyze distributed dataflows in an online setting. It is powered by Timely Dataflow, a low-latency, distributed data-parallel dataflow computational framework, and expands on its predecessor SnailTrail 1, a system to run online critical path analysis on program activity graphs derived from dataflow execution traces. ST2 connects to a running Timely computation, creates the program activity graph representation, and runs multiple analyses on top of it. Analyses include aggregate metrics, progress and temporal invariant checking, and graph pattern matching. Through a command-line interface and a real-time dashboard, users are able to interact with and visualize ST2's analysis results. For ST2's implementation, we discuss Differential Dataflow, a framework that uses differential computation to incrementalize even complex relational dataflow operators, as an alternative to Timely Dataflow, but ultimately settle on using Timely. In our performance evaluations, we are able to show that ST2 is able to comfortably keep up with common streaming computations in offline and online settings, even exceeding SnailTrail 1's performance. We also showcase and evaluate ST2 from a functional standpoint in a case study. Using the dashboard to profile a faulty source computation, we manage to successfully detect the issues' root cause. We argue that ST2 is an extendable system that paves the way for users to debug, monitor, and optimize online distributed dataflows.


page 1

page 2

page 3

page 4


A Visual Analytics Framework for Distributed Data Analysis Systems

This paper proposes a visual analytics framework that addresses the comp...

Real-time regression analysis with deep convolutional neural networks

We discuss the development of novel deep learning algorithms to enable r...

Scalable Online Conformance Checking Using Incremental Prefix-Alignment Computation

Conformance checking techniques aim to collate observed process behavior...

Graphsurge: Graph Analytics on View Collections Using Differential Computation

This paper presents the design and implementation of a new open-source v...

ScalAna: Automating Scaling Loss Detection with Graph Analysis

Scaling a parallel program to modern supercomputers is challenging due t...

Predicting Workout Quality to Help Coaches Support Sportspeople

The support of a qualified coach is crucial to keep the motivation of sp...

Automated System Performance Testing at MongoDB

Distributed Systems Infrastructure (DSI) is MongoDB's framework for runn...