DeepAI AI Chat
Log In Sign Up

Differential Approximation and Sprinting for Multi-Priority Big Data Engines

by   Robert Birke, et al.

Today's big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90 reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30


page 3

page 4


BoPF: Mitigating the Burstiness-Fairness Tradeoff in Multi-Resource Clusters

Simultaneously supporting latency- and throughout-sensitive workloads in...

Improving the Effective Utilization of Supercomputer Resources by Adding Low-Priority Containerized Jobs

We propose an approach to utilize idle computational resources of superc...

Computation Resource Leasing for Priority Aggregation Local Computing Network

In large scale smart edge networks, computation resource is generally un...

Efficient Two-Level Scheduling for Concurrent Graph Processing

With the rapidly growing demand of graph processing in the real scene, t...

Optimal Resource Allocation for Elastic and Inelastic Jobs

Modern data centers are tasked with processing heterogeneous workloads c...

Genie: An Open Box Counterfactual Policy Estimator for Optimizing Sponsored Search Marketplace

In this paper, we propose an offline counterfactual policy estimation fr...

PipeTune: Pipeline Parallelism of Hyper and System Parameters Tuning for Deep Learning Clusters

DNN learning jobs are common in today's clusters due to the advances in ...