Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

11/27/2021
by   David Krasowska, et al.
0

Lossy compression plays a growing role in scientific simulations where the cost of storing their output data can span terabytes. Using error bounded lossy compression reduces the amount of storage for each simulation; however, there is no known bound for the upper limit on lossy compressibility. Correlation structures in the data, choice of compressor and error bound are factors allowing larger compression ratios and improved quality metrics. Analyzing these three factors provides one direction towards quantifying lossy compressibility. As a first step, we explore statistical methods to characterize the correlation structures present in the data and their relationships, through functional models, to compression ratios. We observed a relationship between compression ratios and statistics summarizing correlation structure of the data, which are a first step towards evaluating the theoretical limits of lossy compressibility used to eventually predict compression performance and adapt compressors to correlation structures present in the data.

READ FULL TEXT
research
05/15/2023

Black-Box Statistical Prediction of Lossy Compression Ratios for Scientific Data

Lossy compressors are increasingly adopted in scientific research, tackl...
research
09/07/2023

SRN-SZ: Deep Leaning-Based Scientific Error-bounded Lossy Compression with Super-resolution Neural Networks

The fast growth of computational power and scales of modern super-comput...
research
05/27/2021

cuSZ(x): Optimizing Error-Bounded Lossy Compression for Scientific Data on GPUs

Error-bounded lossy compression is a critical technique for significantl...
research
01/31/2022

SZx: an Ultra-fast Error-bounded Lossy Compressor for Scientific Datasets

Today's scientific high performance computing (HPC) applications or adva...
research
10/15/2022

Tensor-Train Compression of Discrete Element Method Simulation Data

We propose a framework for discrete scientific data compression based on...
research
03/18/2019

A Parallel Data Compression Framework for Large Scale 3D Scientific Data

Large scale simulations of complex systems ranging from climate and astr...
research
08/21/2023

Ultra Dual-Path Compression For Joint Echo Cancellation And Noise Suppression

Echo cancellation and noise reduction are essential for full-duplex comm...

Please sign up or login with your details

Forgot password? Click here to reset