DeepAI AI Chat
Log In Sign Up

Toward Common Components for Open Workflow Systems

by   Jay Jay Billings, et al.
Rutgers University
Oak Ridge National Laboratory

The role of scalable high-performance workflows and flexible workflow management systems that can support multiple simulations will continue to increase in importance. For example, with the end of Dennard scaling, there is a need to substitute a single long running simulation with multiple repeats of shorter simulations, or concurrent replicas. Further, many scientific problems involve ensembles of simulations in order to solve a higher-level problem or produce statistically meaningful results. However most supercomputing software development and performance enhancements have focused on optimizing single- simulation performance. On the other hand, there is a strong inconsistency in the definition and practice of workflows and workflow management systems. This inconsistency often centers around the difference between several different types of workflows, including modeling and simulation, grid, uncertainty quantification, and purely conceptual workflows. This work explores this phenomenon by examining the different types of workflows and workflow management systems, reviewing the perspective of a large supercomputing facility, examining the common features and problems of workflow management systems, and finally presenting a proposed solution based on the concept of common building blocks. The implications of the continuing proliferation of workflow management systems and the lack of interoperability between these systems are discussed from a practical perspective. In doing so, we have begun an investigation of the design and implementation of open workflow systems for supercomputers based upon common components.


page 1

page 2

page 3

page 4


Middleware Building Blocks for Workflow Systems

This paper describes a building blocks approach to the design of scienti...

The Eclipse Integrated Computational Environment

Problems in modeling and simulation require significantly different work...

Towards Advanced Monitoring for Scientific Workflows

Scientific workflows consist of thousands of highly parallelized tasks e...

Evaluation of tools for describing, reproducing and reusing scientific workflows

In the field of computational science and engineering, workflows often e...

Extreme Scale Survey Simulation with Python Workflows

The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) wil...

Theory-plus-code documentation of the DEPAM workflow for soundscape description

In the Big Data era, the community of PAM faces strong challenges, inclu...