Workflow environments for advanced cyberinfrastructure platforms

06/12/2020
by   Rosa M. Badia, et al.
0

Progress in science is deeply bound to the effective use of high-performance computing infrastructures and to the efficient extraction of knowledge from vast amounts of data. Such data comes from different sources that follow a cycle composed of pre-processing steps for data curation and preparation for subsequent computing steps, and later analysis and analytics steps applied to the results. However, scientific workflows are currently fragmented in multiple components, with different processes for computing and data management, and with gaps in the viewpoints of the user profiles involved. Our vision is that future workflow environments and tools for the development of scientific workflows should follow a holistic approach, where both data and computing are integrated in a single flow built on simple, high-level interfaces. The topics of research that we propose involve novel ways to express the workflows that integrate the different data and compute processes, dynamic runtimes to support the execution of the workflows in complex and heterogeneous computing infrastructures in an efficient way, both in terms of performance and energy. These infrastructures include highly distributed resources, from sensors and instruments, and devices in the edge, to High-Performance Computing and Cloud computing resources. This paper presents our vision to develop these workflow environments and also the steps we are currently following to achieve it.

READ FULL TEXT

page 6

page 8

research
10/04/2022

EdgeFaaS: A Function-based Framework for Edge Computing

The rapid growth of data generated from Internet of Things (IoTs) such a...
research
12/23/2020

Library of efficient algorithms for phylogenetic analysis

Evolutionary relationships between species are usually inferred through ...
research
02/25/2019

Towards A Methodology and Framework for Workflow-Driven Team Science

Scientific workflows are powerful tools for management of scalable exper...
research
01/28/2021

Best Practices in Scientific Computing

The world is becoming increasingly complex, both in terms of the rich so...
research
01/18/2023

A Workflow Model for Holistic Data Management and Semantic Interoperability in Quantitative Archival Research

Archival research is a complicated task that involves several diverse ac...
research
04/27/2023

Developing Distributed High-performance Computing Capabilities of an Open Science Platform for Robust Epidemic Analysis

COVID-19 had an unprecedented impact on scientific collaboration. The pa...
research
09/17/2016

Applications of Data Mining (DM) in Science and Engineering: State of the art and perspectives

The continuous increase in the availability of data of any kind, coupled...

Please sign up or login with your details

Forgot password? Click here to reset