SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors

11/04/2021
by   Xin Liang, et al.
0

Today's scientific simulations require a significant reduction of data volume because of extremely large amounts of data they produce and the limited I/O bandwidth and storage space. Error-bounded lossy compressor has been considered one of the most effective solutions to the above problem. In practice, however, the best-fit compression method often needs to be customized/optimized in particular because of diverse characteristics in different datasets and various user requirements on the compression quality and performance. In this paper, we develop a novel modular, composable compression framework (namely SZ3), which involves three significant contributions. (1) SZ3 features a modular abstraction for the prediction-based compression framework such that the new compression modules can be plugged in easily. (2) SZ3 supports multialgorithm predictors and can automatically select the best-fit predictor for each data block based on the designed error estimation criterion. (3) SZ3 allows users to easily compose different compression pipelines on demand, such that both compression quality and performance can be significantly improved for their specific datasets and requirements. (4) In addition, we evaluate several lossy compressors composed from SZ3 using the real-world datasets. Specifically, we leverage SZ3 to improve the compression quality and performance for different use-cases, including GAMESS quantum chemistry dataset and Advanced Photon Source (APS) instrument dataset. Experiments show that our customized compression pipelines lead to up to 20 the same data distortion compared with the state-of-the-art approaches.

READ FULL TEXT

page 2

page 3

page 4

page 5

page 6

page 7

page 8

page 9

research
04/01/2022

TAC: Optimizing Error-Bounded Lossy Compression for Three-Dimensional Adaptive Mesh Refinement Simulations

Today's scientific simulations require a significant reduction of data v...
research
01/05/2023

TAC+: Drastically Optimizing Error-Bounded Lossy Compression for 3D AMR Simulations

Today's scientific simulations require a significant reduction of data v...
research
07/11/2023

Optimizing Scientific Data Transfer on Globus with Error-bounded Lossy Compression

The increasing volume and velocity of science data necessitate the frequ...
research
11/18/2021

Improving Prediction-Based Lossy Compression Dramatically Via Ratio-Quality Modeling

Error-bounded lossy compression is one of the most effective techniques ...
research
06/12/2017

Z-checker: A Framework for Assessing Lossy Compression of Scientific Data

Because of vast volume of data being produced by today's scientific simu...
research
10/12/2020

MGARD+: Optimizing Multilevel Methods for Error-bounded Scientific Data Reduction

Data management is becoming increasingly important in dealing with the l...
research
05/15/2023

Black-Box Statistical Prediction of Lossy Compression Ratios for Scientific Data

Lossy compressors are increasingly adopted in scientific research, tackl...

Please sign up or login with your details

Forgot password? Click here to reset