SAF: Simulated Annealing Fair Scheduling for Hadoop Yarn Clusters

08/28/2020
by   Mahsa Ghanavatinasab, et al.
0

Apache introduced YARN as the next generation of the Hadoop framework, providing resource management and a central platform to deliver consistent data governance tools across Hadoop clusters. Hadoop YARN supports multiple frameworks like MapReduce to process different types of data and works with different scheduling policies such as FIFO, Capacity, and Fair schedulers. DRF is the best option that uses short-term, without considering history information, convergence to fairness for multi-type resource allocation. However, DRF performance is still not satisfying due to trade-offs between fairness and performance regarding resource utilization. To address this problem, we propose Simulated Annealing Fair scheduling, SAF, a long-term fair scheme in resource allocation to have fairness and excellent performance in terms of resource utilization and MakeSpan. We introduce a new parameter as entropy, which is an approach to indicates the disorder in the fairness of allocated resources of the whole cluster. We implemented SAF as a pluggable scheduler in Hadoop Yarn Cluster and evaluated it with standard MapReduce benchmarks in Yarn Scheduler Load Simulator (SLS) and CloudSim Plus simulation framework. Finally, the results of both simulation tools are evidence to prove our claim. Compared to DRF, SAF increases resource utilization of YARN clusters significantly and decreases MakeSpan to an appropriate level.

READ FULL TEXT
research
05/21/2019

Exploring the Fairness and Resource Distribution in an Apache Mesos Environment

Apache Mesos, a cluster-wide resource manager, is widely deployed in mas...
research
08/24/2018

Performance evaluation of job schedulers on Hadoop YARN

To solve the limitation of Hadoop on scalability, resource sharing, and ...
research
06/21/2019

Fairness and Utilization in Allocating Resources with Uncertain Demand

Resource allocation problems are a fundamental domain in which to evalua...
research
07/11/2022

IaaS Procurement by Simulated Annealing

Considering the problem of resource allocation for potentially complex a...
research
05/17/2023

Defining a canonical unit for accounting purposes

Compute resource providers often put in place batch compute systems to m...
research
01/05/2018

Selective Fair Scheduling over Fading Channels

Imposing fairness in resource allocation incurs a loss of system through...
research
03/15/2021

Megha: Decentralized Global Fair Scheduling for Federated Clusters

Increasing scale and heterogeneity in data centers have led to the devel...

Please sign up or login with your details

Forgot password? Click here to reset