CircuitFlow: A Domain Specific Language for Dataflow Programming (with appendices)

11/24/2021
by   Riley Evans, et al.
0

Dataflow applications, such as machine learning algorithms, can run for days, making it desirable to have assurances that they will work correctly. Current tools are not good enough: too often the interactions between tasks are not type-safe, leading to undesirable run-time errors. This paper presents a new declarative Haskell Embedded DSL (eDSL) for dataflow programming: CircuitFlow. Defined as a Symmetric Monoidal Preorder (SMP) on data that models dependencies in the workflow, it has a strong mathematical basis, refocusing on how data flows through an application, resulting in a more expressive solution that not only catches errors statically, but also achieves competitive run-time performance. In our preliminary evaluation, CircuitFlow outperforms the industry-leading Luigi library of Spotify by scaling better with the number of inputs. The innovative creation of CircuitFlow is also of note, exemplifying how to create a modular eDSL whose semantics necessitates effects, and where storing complex type information for program correctness is paramount.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/04/2017

Introspection for C and its Applications to Library Robustness

Context: In C, low-level errors, such as buffer overflow and use-after-f...
research
08/30/2011

Domain-specific Languages in a Finite Domain Constraint Programming System

In this paper, we present domain-specific languages (DSLs) that we devis...
research
04/01/2021

Idris 2: Quantitative Type Theory in Practice

Dependent types allow us to express precisely what a function is intende...
research
05/18/2017

Exploiting Term Hiding to Reduce Run-time Checking Overhead

One of the most attractive features of untyped languages is the flexibil...
research
06/07/2019

Towards Run Time Estimation of the Gaussian Chemistry Code for SEAGrid Science Gateway

Accurate estimation of the run time of computational codes has a number ...
research
10/12/2022

Specializing Scope Graph Resolution Queries: Extended Edition

To warrant programmer productivity, type checker results should be corre...
research
07/01/2019

Type Checking Program Code using SHACL (Extended Version)

It is a strength of graph-based data formats, like RDF, that they are ve...

Please sign up or login with your details

Forgot password? Click here to reset