A Bespoke Workflow Management System for Data-Driven Urgent HPC

10/23/2020
by   Gordon P. S. Gibb, et al.
0

In this paper we present a workflow management system which permits the kinds of data-driven workflows required by urgent computing, namely where new data is integrated into the workflow as a disaster progresses in order refine the predictions as time goes on. This allows the workflow to adapt to new data at runtime, a capability that most workflow management systems do not possess. The workflow management system was developed for the EU-funded VESTEC project, which aims to fuse HPC with real-time data for supporting urgent decision making. We first describe an example workflow from the VESTEC project, and show why existing workflow technologies do not meet the needs of the project. We then go on to present the design of our Workflow Management System, describe how it is implemented into the VESTEC system, and provide an example of the workflow system in use for a test case.

READ FULL TEXT

Authors

page 10

03/31/2017

The Eclipse Integrated Computational Environment

Problems in modeling and simulation require significantly different work...
04/15/2018

Applying Distributed Ledgers to Manage Workflow Provenance

Sharing provenance across workflow management systems automatically is n...
01/11/2018

The archive solution for distributed workflow management agents of the CMS experiment at LHC

The CMS experiment at the CERN LHC developed the Workflow Management Arc...
03/30/2020

A Framework for Online Investment Algorithms

The artificial segmentation of an investment management process into a w...
05/22/2017

Nucleus: A Pilot Project

Early in 2016, an environmental scan was conducted by the Research Libra...
08/30/2021

ExaWorks: Workflows for Exascale

Exascale computers will offer transformative capabilities to combine dat...
10/23/2020

An Intermediate Data-driven Methodology for Scientific Workflow Management System to Support Reusability

In this thesis first we propose an intermediate data management scheme f...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.