An Order-aware Dataflow Model for Extracting Shell Script Parallelism

12/31/2020

∙

by   Shivam Handa, et al.

∙

0

∙

share

We present a dataflow model for extracting data parallelism latent in Unix shell scripts. To accurately capture the semantics of Unix shell scripts, the dataflow model is order-aware, i.e., the order in which a node in the dataflow graph consumes inputs from different edges plays a central role in the semantics of the computation and therefore in the resulting parallelization. We use this model to capture the semantics of transformations that exploit data parallelism available in Unix shell computations and prove their correctness. We additionally formalize the translations from the Unix shell to the dataflow model and from the dataflow model back to a parallel shell script. We use a large number of real scripts to evaluate the parallel performance delivered by the dataflow transformations, including the contributions of individual transformations, achieving an average speedup of 6.14× and a maximum of 61.1× on a 64-core machine.