Bioinformatics Computational Cluster Batch Task Profiling with Machine Learning for Failure Prediction

12/22/2018
by   Christopher Harrison, et al.
0

Motivation: Traditional computational cluster schedulers are based on user inputs and run time needs request for memory and CPU, not IO. Heavily IO bound task run times, like ones seen in many big data and bioinformatics problems, are dependent on the IO subsystems scheduling and are problematic for cluster resource scheduling. The problematic rescheduling of IO intensive and errant tasks is a lost resource. Understanding the conditions in both successful and failed tasks and differentiating them could provide knowledge to enhancing cluster scheduling and intelligent resource optimization. Results: We analyze a production computational cluster contributing 6.7 thousand CPU hours to research over two years. Through this analysis we develop a machine learning task profiling agent for clusters that attempts to predict failures between identically provision requested tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/09/2020

CASH: A Credit Aware Scheduling for Public Cloud Platforms

The public cloud offers a myriad of services which allows its tenants to...
research
01/10/2022

A Simulation Platform for Multi-tenant Machine Learning Services on Thousands of GPUs

Multi-tenant machine learning services have become emerging data-intensi...
research
05/21/2019

Exploring the Fairness and Resource Distribution in an Apache Mesos Environment

Apache Mesos, a cluster-wide resource manager, is widely deployed in mas...
research
04/06/2020

Optimal Virtual Cluster-based Multiprocessor Scheduling

Scheduling of constrained deadline sporadic task systems on multiprocess...
research
05/26/2017

Design and Implementation of Modified Fuzzy based CPU Scheduling Algorithm

CPU Scheduling is the base of multiprogramming. Scheduling is a process ...
research
04/10/2019

R-Storm: Resource-Aware Scheduling in Storm

The era of big data has led to the emergence of new systems for real-tim...
research
12/15/2010

Customer Appeasement Scheduling

Almost all of the current process scheduling algorithms which are used i...

Please sign up or login with your details

Forgot password? Click here to reset