DeepAI AI Chat
Log In Sign Up

Task Runtime Prediction in Scientific Workflows Using an Online Incremental Learning Approach

by   Muhammad H. Hilman, et al.
The University of Melbourne

Many algorithms in workflow scheduling and resource provisioning rely on the performance estimation of tasks to produce a scheduling plan. A profiler that is capable of modeling the execution of tasks and predicting their runtime accurately, therefore, becomes an essential part of any Workflow Management System (WMS). With the emergence of multi-tenant Workflow as a Service (WaaS) platforms that use clouds for deploying scientific workflows, task runtime prediction becomes more challenging because it requires the processing of a significant amount of data in a near real-time scenario while dealing with the performance variability of cloud resources. Hence, relying on methods such as profiling tasks' execution data using basic statistical description (e.g., mean, standard deviation) or batch offline regression techniques to estimate the runtime may not be suitable for such environments. In this paper, we propose an online incremental learning approach to predict the runtime of tasks in scientific workflows in clouds. To improve the performance of the predictions, we harness fine-grained resources monitoring data in the form of time-series records of CPU utilization, memory usage, and I/O activities that are reflecting the unique characteristics of a task's execution. We compare our solution to a state-of-the-art approach that exploits the resources monitoring data based on regression machine learning technique. From our experiments, the proposed strategy improves the performance, in terms of the error, up to 29.89


page 1

page 2

page 3

page 4


Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures

Scientific workflows typically comprise a multitude of different process...

Modular Resource Centric Learning for Workflow Performance Prediction

Workflows provide an expressive programming model for fine-grained contr...

Adaptive Scheduling for Efficient Execution of Dynamic Stream Workflows

Stream workflow application such as online anomaly detection or online t...

Profiling Resource Utilization of Bioinformatics Workflows

We present a software tool, the Container Profiler, that measures and re...

BottleMod: Modeling Data Flows and Tasks for Fast Bottleneck Analysis

In the recent years, scientific workflows gained more and more popularit...

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters

Many scientific workflow scheduling algorithms need to be informed about...

Keeping Track of User Steering Actions in Dynamic Workflows

In long-lasting scientific workflow executions in HPC machines, computat...