Accelerating Parallel Write via Deeply Integrating Predictive Lossy Compression with HDF5

06/29/2022
by   Sian Jin, et al.
0

Lossy compression is one of the most efficient solutions to reduce storage overhead and improve I/O performance for HPC applications. However, existing parallel I/O libraries cannot fully utilize lossy compression to accelerate parallel write due to the lack of deep understanding on compression-write performance. To this end, we propose to deeply integrate predictive lossy compression with HDF5 to significantly improve the parallel-write performance. Specifically, we propose analytical models to predict the time of compression and parallel write before the actual compression to enable compression-write overlapping. We also introduce an extra space in the process to handle possible data overflows resulting from prediction uncertainty in compression ratios. Moreover, we propose an optimization to reorder the compression tasks to increase the overlapping efficiency. Experiments with up to 4,096 cores from Summit show that our solution improves the write performance by up to 4.5X and 2.9X over the non-compression and lossy compression solutions, respectively, with only 1.5 HPC applications.

READ FULL TEXT
research
05/27/2021

cuSZ(x): Optimizing Error-Bounded Lossy Compression for Scientific Data on GPUs

Error-bounded lossy compression is a critical technique for significantl...
research
05/24/2015

Lossless Layout Image Compression Algorithms for Electron-Beam Direct-Write Lithography

Electron-beam direct-write (EBDW) lithography systems must in the future...
research
04/13/2023

accelerating wrf i/o performance with adios2 and network-based streaming

With the approach of Exascale computing power for large-scale High Perfo...
research
01/20/2022

High Performance Parallel I/O and In-Situ Analysis in the WRF Model with ADIOS2

As the computing power of large-scale HPC clusters approaches the Exasca...
research
10/06/2022

DeltaFS: Pursuing Zero Update Overhead via Metadata-Enabled Delta Compression for Log-structured File System on Mobile Devices

Data compression has been widely adopted to release mobile devices from ...
research
05/07/2019

Efficient Similarity-aware Compression to Reduce Bit-writes in Non-Volatile Main Memory for Image-based Applications

Image bitmaps have been widely used in in-memory applications, which con...
research
09/18/2023

Significant improvement of lossy compression rate and speed of HPC data using perceptron parallelized compression

The escalating surge in data generation presents formidable challenges t...

Please sign up or login with your details

Forgot password? Click here to reset