Decoupled Strategy for Imbalanced Workloads in MapReduce Frameworks

10/09/2018
by   Sergio Rivas-Gomez, et al.
0

In this work, we consider the integration of MPI one-sided communication and non-blocking I/O in HPC-centric MapReduce frameworks. Using a decoupled strategy, we aim to overlap the Map and Reduce phases of the algorithm by allowing processes to communicate and synchronize using solely one-sided operations. Hence, we effectively increase the performance in situations where the workload per process is unexpectedly unbalanced. Using a Word-Count implementation and a large dataset from the Purdue MapReduce Benchmarks Suite (PUMA), we demonstrate that our approach can provide up to 23 improvement on average compared to a reference MapReduce implementation that uses state-of-the-art MPI collective communication and I/O.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

10/22/2017

Lightweight MPI Communicators with Applications to Perfectly Balanced Schizophrenic Quicksort

MPI uses the concept of communicators to connect groups of processes. It...
10/09/2018

MPI Windows on Storage for HPC Applications

Upcoming HPC clusters will feature hybrid memories and storage devices p...
11/15/2021

Quo Vadis MPI RMA? Towards a More Efficient Use of MPI One-Sided Communication

The MPI standard has long included one-sided communication abstractions ...
10/26/2020

Leveraging MPI RMA to optimise halo-swapping communications in MONC on Cray machines

Remote Memory Access (RMA), also known as single sided communications, p...
02/18/2022

SKaMPI-OpenSHMEM: Measuring OpenSHMEM Communication Routines

Benchmarking is an important challenge in HPC, in particular, to be able...
06/05/2018

Energy-efficient localised rollback after failures via data flow analysis

Exascale systems will suffer failures hourly. HPC programmers rely mostl...
12/01/2020

Enhancing Scalability of a Matrix-Free Eigensolver for Studying Many-Body Localization

In [Van Beeumen, et. al, HPC Asia 2020, https://www.doi.org/10.1145/3368...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.