DeepAI
Log In Sign Up

Online Analysis of Distributed Dataflows with Timely Dataflow

12/20/2019
by   Malte Sandstede, et al.
0

We present ST2, an end-to-end solution to analyze distributed dataflows in an online setting. It is powered by Timely Dataflow, a low-latency, distributed data-parallel dataflow computational framework, and expands on its predecessor SnailTrail 1, a system to run online critical path analysis on program activity graphs derived from dataflow execution traces. ST2 connects to a running Timely computation, creates the program activity graph representation, and runs multiple analyses on top of it. Analyses include aggregate metrics, progress and temporal invariant checking, and graph pattern matching. Through a command-line interface and a real-time dashboard, users are able to interact with and visualize ST2's analysis results. For ST2's implementation, we discuss Differential Dataflow, a framework that uses differential computation to incrementalize even complex relational dataflow operators, as an alternative to Timely Dataflow, but ultimately settle on using Timely. In our performance evaluations, we are able to show that ST2 is able to comfortably keep up with common streaming computations in offline and online settings, even exceeding SnailTrail 1's performance. We also showcase and evaluate ST2 from a functional standpoint in a case study. Using the dashboard to profile a faulty source computation, we manage to successfully detect the issues' root cause. We argue that ST2 is an extendable system that paves the way for users to debug, monitor, and optimize online distributed dataflows.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/10/2021

A Visual Analytics Framework for Distributed Data Analysis Systems

This paper proposes a visual analytics framework that addresses the comp...
05/07/2018

Real-time regression analysis with deep convolutional neural networks

We discuss the development of novel deep learning algorithms to enable r...
12/22/2020

Scalable Online Conformance Checking Using Incremental Prefix-Alignment Computation

Conformance checking techniques aim to collate observed process behavior...
04/11/2020

Graphsurge: Graph Analytics on View Collections Using Differential Computation

This paper presents the design and implementation of a new open-source v...
09/03/2020

ScalAna: Automating Scaling Loss Detection with Graph Analysis

Scaling a parallel program to modern supercomputers is challenging due t...
01/25/2021

Predicting Workout Quality to Help Coaches Support Sportspeople

The support of a qualified coach is crucial to keep the motivation of sp...
04/17/2020

Automated System Performance Testing at MongoDB

Distributed Systems Infrastructure (DSI) is MongoDB's framework for runn...