DRESS: Dynamic RESource-reservation Scheme for Congested Data-intensive Computing Platforms

05/22/2018
by   Ying Mao, et al.
0

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1 meanwhile, maintains a stable overall system performance.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/25/2021

Node-Based Job Scheduling for Large Scale Simulations of Short Running Jobs

Diverse workloads such as interactive supercomputing, big data analysis,...
research
03/09/2019

Machine Learning Based Prediction and Classification of Computational Jobs in Cloud Computing Centers

With the rapid growth of the data volume and the fast increasing of the ...
research
08/05/2020

Best of Both Worlds: High Performance Interactive and Batch Launching

Rapid launch of thousands of jobs is essential for effective interactive...
research
04/09/2018

PingAn: An Insurance Scheme for Job Acceleration in Geo-distributed Big Data Analytics System

Geo-distributed data analysis in a cloud-edge system is emerging as a da...
research
10/24/2020

Differentiate Quality of Experience Scheduling for Deep Learning Applications with Docker Containers in the Cloud

With the prevalence of big-data-driven applications, such as face recogn...
research
04/07/2023

Runtime Variation in Big Data Analytics

The dynamic nature of resource allocation and runtime conditions on Clou...
research
04/17/2019

Terra: Scalable Cross-Layer GDA Optimizations

Geo-distributed analytics (GDA) frameworks transfer large datasets over ...

Please sign up or login with your details

Forgot password? Click here to reset