Differential Approximation and Sprinting for Multi-Priority Big Data Engines

by   Robert Birke, et al.

Today's big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90 reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30



page 3

page 4


BoPF: Mitigating the Burstiness-Fairness Tradeoff in Multi-Resource Clusters

Simultaneously supporting latency- and throughout-sensitive workloads in...

Improving the Effective Utilization of Supercomputer Resources by Adding Low-Priority Containerized Jobs

We propose an approach to utilize idle computational resources of superc...

Computation Resource Leasing for Priority Aggregation Local Computing Network

In large scale smart edge networks, computation resource is generally un...

Efficient Two-Level Scheduling for Concurrent Graph Processing

With the rapidly growing demand of graph processing in the real scene, t...

Sharp Waiting-Time Bounds for Multiserver Jobs

Multiserver jobs, which are jobs that occupy multiple servers simultaneo...

Blocking time under basic priority inheritance: Polynomial bound and exact computation

The Priority Inheritance Protocol (PIP) is arguably the best-known proto...

Analytical Performance Models for NoCs with Multiple Priority Traffic Classes

Networks-on-chip (NoCs) have become the standard for interconnect soluti...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.