Multi-GPU Performance Optimization of a CFD Code using OpenACC on Different Platforms

06/04/2020
by   Weicheng Xue, et al.
0

This paper investigates the multi-GPU performance of a 3D buoyancy driven cavity solver using MPI and OpenACC directives on different platforms. The paper shows that decomposing the total problem in different dimensions affects the strong scaling performance significantly for the GPU. Without proper performance optimizations, it is shown that 1D domain decomposition scales poorly on multiple GPUs due to the noncontiguous memory access. The performance using whatever decompositions can be benefited from a series of performance optimizations in the paper. Since the buoyancy driven cavity code is latency-bounded on the clusters examined, a series of optimizations both agnostic and tailored to the platforms are designed to reduce the latency cost and improve memory throughput between hosts and devices efficiently. First, the parallel message packing/unpacking strategy developed for noncontiguous data movement between hosts and devices improves the overall performance by about a factor of 2. Second, transferring different data based on the stencil sizes for different variables further reduces the communication overhead. These two optimizations are general enough to be beneficial to stencil computations having ghost changes on all of the clusters tested. Third, GPUDirect is used to improve the communication on clusters which have the hardware and software support for direct communication between GPUs without staging CPU's memory. Finally, overlapping the communication and computations is shown to be not efficient on multi-GPUs if only using MPI or MPI+OpenACC. Although we believe our implementation has revealed enough overlap, the actual running does not utilize the overlap well due to a lack of asynchronous progression.

READ FULL TEXT
research
12/05/2020

An Improved Framework of GPU Computing for CFD Applications on Structured Grids using OpenACC

This paper is focused on improving multi-GPU performance of a research C...
research
09/12/2019

PittPack: An Open-Source Poisson's Equation Solver for Extreme-Scale Computing with Accelerators

We present a parallel implementation of a direct solver for the Poisson'...
research
02/25/2022

HipBone: A performance-portable GPU-accelerated C++ version of the NekBone benchmark

We present hipBone, an open source performance-portable proxy applicatio...
research
09/13/2023

GPU Scheduler for De Novo Genome Assembly with Multiple MPI Processes

De Novo Genome assembly is one of the most important tasks in computatio...
research
04/10/2023

An Experimental Study of Two-Level Schwarz Domain Decomposition Preconditioners on GPUs

The generalized Dryja–Smith–Widlund (GDSW) preconditioner is a two-level...
research
09/04/2017

From MPI to MPI+OpenACC: Conversion of a legacy FORTRAN PCG solver for the spherical Laplace equation

A real-world example of adding OpenACC to a legacy MPI FORTRAN Precondit...
research
12/22/2022

The Gaia AVU-GSR parallel solver: preliminary studies of a LSQR-based application in perspective of exascale systems

The Gaia Astrometric Verification Unit-Global Sphere Reconstruction (AVU...

Please sign up or login with your details

Forgot password? Click here to reset