Decoupled Strategy for Imbalanced Workloads in MapReduce Frameworks

10/09/2018
by   Sergio Rivas-Gomez, et al.
0

In this work, we consider the integration of MPI one-sided communication and non-blocking I/O in HPC-centric MapReduce frameworks. Using a decoupled strategy, we aim to overlap the Map and Reduce phases of the algorithm by allowing processes to communicate and synchronize using solely one-sided operations. Hence, we effectively increase the performance in situations where the workload per process is unexpectedly unbalanced. Using a Word-Count implementation and a large dataset from the Purdue MapReduce Benchmarks Suite (PUMA), we demonstrate that our approach can provide up to 23 improvement on average compared to a reference MapReduce implementation that uses state-of-the-art MPI collective communication and I/O.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/22/2017

Lightweight MPI Communicators with Applications to Perfectly Balanced Schizophrenic Quicksort

MPI uses the concept of communicators to connect groups of processes. It...
research
10/09/2018

MPI Windows on Storage for HPC Applications

Upcoming HPC clusters will feature hybrid memories and storage devices p...
research
11/15/2021

Quo Vadis MPI RMA? Towards a More Efficient Use of MPI One-Sided Communication

The MPI standard has long included one-sided communication abstractions ...
research
10/26/2020

Leveraging MPI RMA to optimise halo-swapping communications in MONC on Cray machines

Remote Memory Access (RMA), also known as single sided communications, p...
research
02/18/2022

SKaMPI-OpenSHMEM: Measuring OpenSHMEM Communication Routines

Benchmarking is an important challenge in HPC, in particular, to be able...
research
03/31/2022

Efficient and Eventually Consistent Collective Operations

Collective operations are common features of parallel programming models...
research
06/05/2018

Energy-efficient localised rollback after failures via data flow analysis

Exascale systems will suffer failures hourly. HPC programmers rely mostl...

Please sign up or login with your details

Forgot password? Click here to reset