Streaming Compression of Scientific Data via weak-SINDy

by   Benjamin P. Russo, et al.

In this paper a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing an stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline" compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online" while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well-suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.


page 11

page 16

page 22


Efficient Computation of Tucker Decomposition for Streaming Scientific Data Compression

The Tucker decomposition, an extension of singular value decomposition f...

Enhancing Dynamic Mode Decomposition Workflow with In-Situ Visualization and Data Compression

Modern computational science and engineering applications are being impr...

Statistical Inference in High-dimensional Generalized Linear Models with Streaming Data

In this paper we develop an online statistical inference approach for hi...

Streaming Low-Rank Matrix Approximation with an Application to Scientific Simulation

This paper argues that randomized linear sketching is a natural tool for...

Compression of enumerations and gain

We study the compressibility of enumerations, and its role in the relati...

Bias Correction for Regularized Regression and its Application in Learning with Streaming Data

We propose an approach to reduce the bias of ridge regression and regula...

Please sign up or login with your details

Forgot password? Click here to reset