Desynchronization and Wave Pattern Formation in MPI-Parallel and Hybrid Memory-Bound Programs

by   Ayesha Afzal, et al.

Analytic, first-principles performance modeling of distributed-memory parallel codes is notoriously imprecise. Even for applications with extremely regular and homogeneous compute-communicate phases, simply adding communication time to computation time does often not yield a satisfactory prediction of parallel runtime due to deviations from the expected simple lockstep pattern caused by system noise, variations in communication time, and inherent load imbalance. In this paper, we highlight the specific cases of provoked and spontaneous desynchronization of memory-bound, bulk-synchronous pure MPI and hybrid MPI+OpenMP programs. Using simple microbenchmarks we observe that although desynchronization can introduce increased waiting time per process, it does not necessarily cause lower resource utilization but can lead to an increase in available bandwidth per core. In case of significant communication overhead, even natural noise can shove the system into a state of automatic overlap of communication and computation, improving the overall time to solution. The saturation point, i.e., the number of processes per memory domain required to achieve full memory bandwidth, is pivotal in the dynamics of this process and the emerging stable wave pattern. We also demonstrate how hybrid MPI-OpenMP programming can prevent desirable desynchronization by eliminating the bandwidth bottleneck among processes. A Chebyshev filter diagonalization application is used to demonstrate some of the observed effects in a realistic setting.



There are no comments yet.


page 1

page 2

page 3

page 4


HDOT – an Approach Towards Productive Programming of Hybrid Applications

MPI applications matter. However, with the advent of many-core processor...

Analytic Modeling of Idle Waves in Parallel Programs: Communication, Cluster Topology, and Noise Impact

Most distributed-memory bulk-synchronous parallel programs in HPC assume...

Scalable Communication Endpoints for MPI+Threads Applications

Hybrid MPI+threads programming is gaining prominence as an alternative t...

Open-MPI over MOSIX: paralleled computing in a clustered world

Recent increased interest in Cloud computing emphasizes the need to find...

Integrating Blocking and Non-Blocking MPI Primitives with Task-Based Programming Models

In this paper we present the Task-Aware MPI library (TAMPI) that integra...

Framework for the hybrid parallelisation of simulation codes

Writing efficient hybrid parallel code is tedious, error-prone, and requ...

Visualizing the world's largest turbulence simulation

In this exploratory submission we present the visualization of the large...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.