Differential Approximation and Sprinting for Multi-Priority Big Data Engines

09/12/2019
by   Robert Birke, et al.
0

Today's big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90 reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30

READ FULL TEXT

Authors

page 3

page 4

12/07/2019

BoPF: Mitigating the Burstiness-Fairness Tradeoff in Multi-Resource Clusters

Simultaneously supporting latency- and throughout-sensitive workloads in...
09/01/2019

Improving the Effective Utilization of Supercomputer Resources by Adding Low-Priority Containerized Jobs

We propose an approach to utilize idle computational resources of superc...
10/31/2019

Computation Resource Leasing for Priority Aggregation Local Computing Network

In large scale smart edge networks, computation resource is generally un...
06/03/2018

Efficient Two-Level Scheduling for Concurrent Graph Processing

With the rapidly growing demand of graph processing in the real scene, t...
09/11/2021

Sharp Waiting-Time Bounds for Multiserver Jobs

Multiserver jobs, which are jobs that occupy multiple servers simultaneo...
06/05/2018

Blocking time under basic priority inheritance: Polynomial bound and exact computation

The Priority Inheritance Protocol (PIP) is arguably the best-known proto...
08/07/2019

Analytical Performance Models for NoCs with Multiple Priority Traffic Classes

Networks-on-chip (NoCs) have become the standard for interconnect soluti...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.