Designing for Recommending Intermediate States in A Scientific Workflow Management System

10/10/2020
by   Debasish Chakroborti, et al.
0

To process a large amount of data sequentially and systematically, proper management of workflow components (i.e., modules, data, configurations, associations among ports and links) in a Scientific Workflow Management System (SWfMS) is inevitable. Managing data with provenance in a SWfMS to support reusability of workflows, modules, and data is not a simple task. Handling such components is even more burdensome for frequently assembled and executed complex workflows for investigating large datasets with different technologies (i.e., various learning algorithms or models). However, a great many studies propose various techniques and technologies for managing and recommending services in a SWfMS, but only a very few studies consider the management of data in a SWfMS for efficient storing and facilitating workflow executions. Furthermore, there is no study to inquire about the effectiveness and efficiency of such data management in a SWfMS from a user perspective. In this paper, we present and evaluate a GUI version of such a novel approach of intermediate data management with two use cases (Plant Phenotyping and Bioinformatics). The technique we call GUI-RISPTS (Recommending Intermediate States from Pipelines Considering Tool-States) can facilitate executions of workflows with processed data (i.e., intermediate outcomes of modules in a workflow) and can thus reduce the computational time of some modules in a SWfMS. We integrated GUI-RISPTS with an existing workflow management system called SciWorCS. In SciWorCS, we present an interface that users use for selecting the recommendation of intermediate states (i.e., modules' outcomes). We investigated GUI-RISP's effectiveness from users' perspectives along with measuring its overhead in terms of storage and efficiency in workflow execution.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 12

page 20

page 23

page 24

page 26

10/23/2020

An Intermediate Data-driven Methodology for Scientific Workflow Management System to Support Reusability

In this thesis first we propose an intermediate data management scheme f...
01/11/2018

BioWorkbench: A High-Performance Framework for Managing and Analyzing Bioinformatics Experiments

Advances in sequencing techniques have led to exponential growth in biol...
12/10/2019

Managing Machine Learning Workflow Components

Machine Learning Workflows (MLWfs) have become essential and a disruptiv...
05/11/2021

Distributed In-memory Data Management for Workflow Executions

Complex scientific experiments from various domains are typically modele...
04/16/2020

A Workflow Manager for Complex NLP and Content Curation Pipelines

We present a workflow manager for the flexible creation and customisatio...
01/11/2018

The archive solution for distributed workflow management agents of the CMS experiment at LHC

The CMS experiment at the CERN LHC developed the Workflow Management Arc...
08/27/2017

RIOT: a Novel Stochastic Method for Rapidly Configuring Cloud-Based Workflows

Traditional tools for configuring cloud services can run much slower tha...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.