Data Deduplication with Random Substitutions

07/01/2021
by   Hao Lou, et al.
0

Data deduplication saves storage space by identifying and removing repeats in the data stream. Compared with traditional compression methods, data deduplication schemes are more time efficient and are thus widely used in large scale storage systems. In this paper, we provide an information-theoretic analysis on the performance of deduplication algorithms on data streams in which repeats are not exact. We introduce a source model in which probabilistic substitutions are considered. More precisely, each symbol in a repeated string is substituted with a given edit probability. Deduplication algorithms in both the fixed-length scheme and the variable-length scheme are studied. The fixed-length deduplication algorithm is shown to be unsuitable for the proposed source model as it does not take into account the edit probability. Two modifications are proposed and shown to have performances within a constant factor of optimal with the knowledge of source model parameters. We also study the conventional variable-length deduplication algorithm and show that as source entropy becomes smaller, the size of the compressed string vanishes relative to the length of the uncompressed string, leading to high compression ratios.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/22/2022

Universal Variable-to-Fixed Length Lossy Compression at Finite Blocklengths

We consider universal variable-to-fixed length compression of memoryless...
research
03/31/2023

The Many Qualities of a New Directly Accessible Compression Scheme

We present a new variable-length computation-friendly encoding scheme, n...
research
01/07/2020

Complexity Issues of String to Graph Approximate Matching

The problem of matching a query string to a directed graph, whose vertic...
research
07/10/2020

Third-Order Asymptotics of Variable-Length Compression Allowing Errors

This study investigates the fundamental limits of variable-length compre...
research
07/18/2018

An Information-theoretic Framework for the Lossy Compression of Link Streams

Graph compression is a data analysis technique that consists in the repl...
research
07/19/2021

Sensitivity of string compressors and repetitiveness measures

The sensitivity of a string compression algorithm C asks how much the ou...
research
08/30/2010

Prediction by Compression

It is well known that text compression can be achieved by predicting the...

Please sign up or login with your details

Forgot password? Click here to reset