Indexing Highly Repetitive String Collections

by   Gonzalo Navarro, et al.

Two decades ago, a breakthrough in indexing string collections made it possible to represent them within their compressed space while at the same time offering indexed search functionalities. As this new technology permeated through applications like bioinformatics, the string collections experienced a growth that outperforms Moore's Law and challenges our ability of handling them even in compressed form. It turns out, fortunately, that many of these rapidly growing string collections are highly repetitive, so that their information content is orders of magnitude lower than their plain size. The statistical compression methods used for classical collections, however, are blind to this repetitiveness, and therefore a new set of techniques has been developed in order to properly exploit it. The resulting indexes form a new generation of data structures able to handle the huge repetitive string collections that we are facing. In this survey we cover the algorithmic developments that have led to these data structures. We describe the distinct compression paradigms that have been used to exploit repetitiveness, the fundamental algorithmic ideas that form the base of all the existing indexes, and the various structures that have been proposed, comparing them both in theoretical and practical aspects. We conclude with the current challenges in this fascinating field.


page 1

page 2

page 3

page 4


Improved Compressed String Dictionaries

We introduce a new family of compressed data structures to efficiently s...

Fast, Small, and Simple Document Listing on Repetitive Text Collections

Document listing on string collections is the task of finding all docume...

A theoretical and experimental analysis of BWT variants for string collections

The extended Burrows-Wheeler-Transform (eBWT), introduced by Mantaci et ...

On the Reproducibility of Experiments of Indexing Repetitive Document Collections

This work introduces a companion reproducible paper with the aim of allo...

A Probabilistic Framework for Lexicon-based Keyword Spotting in Handwritten Text Images

Query by String Keyword Spotting (KWS) is here considered as a key techn...

A New Class of String Transformations for Compressed Text Indexing

Introduced about thirty years ago in the field of Data Compression, the ...

Tunneling on Wheeler Graphs

The Burrows-Wheeler Transform (BWT) is an important technique both in da...