Log In Sign Up

Exploring Autoencoder-Based Error-Bounded Compression for Scientific Data

by   Jinyang Liu, et al.

Error-bounded lossy compression is becoming an indispensable technique for the success of today's scientific projects with vast volumes of data produced during the simulations or instrument data acquisitions. Not only can it significantly reduce data size, but it also can control the compression errors based on user-specified error bounds. Autoencoder (AE) models have been widely used in image compression, but few AE-based compression approaches support error-bounding features, which are highly required by scientific applications. To address this issue, we explore using convolutional autoencoders to improve error-bounded lossy compression for scientific data, with the following three key contributions. (1) We provide an in-depth investigation of the characteristics of various autoencoder models and develop an error-bounded autoencoder-based framework in terms of the SZ model. (2) We optimize the compression quality for main stages in our designed AE-based error-bounded compression framework, fine-tuning the block sizes and latent sizes and also optimizing the compression efficiency of latent vectors. (3) We evaluate our proposed solution using five real-world scientific datasets and comparing them with six other related works. Experiments show that our solution exhibits a very competitive compression quality from among all the compressors in our tests. In absolute terms, it can obtain a much better compression quality (100   800 with SZ2.1 and ZFP in cases with a high compression ratio.


page 1

page 2

page 9


SZx: an Ultra-fast Error-bounded Lossy Compressor for Scientific Datasets

Today's scientific high performance computing (HPC) applications or adva...

SDC Resilient Error-bounded Lossy Compressor

Lossy compression is one of the most important strategies to resolve the...

Improving Prediction-Based Lossy Compression Dramatically Via Ratio-Quality Modeling

Error-bounded lossy compression is one of the most effective techniques ...

Z-checker: A Framework for Assessing Lossy Compression of Scientific Data

Because of vast volume of data being produced by today's scientific simu...

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

Efficient error-controlled lossy compressors are becoming critical to th...

Scalable Hybrid Learning Techniques for Scientific Data Compression

Data compression is becoming critical for storing scientific data becaus...

Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

Lossy compression plays a growing role in scientific simulations where t...