Fast, Small, and Simple Document Listing on Repetitive Text Collections

02/20/2019 ∙ by Dustin Cobas, et al. ∙ Universidad de Chile 0

Document listing on string collections is the task of finding all documents where a pattern appears. It is regarded as the most fundamental document retrieval problem, and is useful in various applications. Many of the fastest-growing string collections are composed of very similar documents, such as versioned code and document collections, genome repositories, etc. Plain pattern-matching indexes designed for repetitive text collections achieve orders-of-magnitude reductions in space. Instead, there are not many analogous indexes for document retrieval. In this paper we present a simple document listing index for repetitive string collections of total length n that lists the ndoc distinct documents where a pattern of length m appears in time O(m+ndoc · n). We exploit the repetitiveness of the document array (i.e., the suffix array coarsened to document identifiers) to grammar-compress it while precomputing the answers to nonterminals, and store them in grammar-compressed form as well. Our experimental results show that our index sharply outperforms existing alternatives in the space/time tradeoff map.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Document retrieval is a family of problems aimed at retrieving documents from a set that are relevant to a query pattern. In a rather general setting, both documents and patterns are arbitrary strings. This encompasses the well-known application of natural language and Web searching, but also many others of interest in bioinformatics, software development, and multimedia retrieval, to name a few [22].

The most fundamental document retrieval problem, on top of which more sophisticated ones are built, is document listing. This problem aims at simply returning the list of documents where the pattern appears. An obvious solution to document listing resorts to pattern matching: find all the positions where the pattern appears, and then return the different documents where those positions lie. This solution requires time and the output is of size , so the approach is very inefficient if (i.e., if the pattern appears many times in the same documents). A better solution, which however applies only in natural language settings, resorts to inverted indexes [1]. These restrict the possible patterns to sequences of words and store the list of the documents where each word appears, thereby solving document listing via intersections of the lists of the pattern words.

Muthunkishnan [20] designed the first linear-space and optimal-time index for general string collections. Given a collection of total length , he builds an index of words that lists the documents where a pattern of length appears in time . While linear space is deemed as sufficiently small in classic scenarios, the solution is impractical for very large text collections unless one resorts to disk, which is orders of magnitude slower. Sadakane [26] showed how to reduce the space of Muthukrishnan’s index to that of the statistically-compressed text plus bits, while raising the time complexity to only if the appropriate underlying pattern-matching index is used [2].

The sharp growth of text collections is a concern in many recent applications, outperforming Moore’s Law in some cases [27]. Fortunately, many of the fastest-growing text collections are highly repetitive: each document can be obtained from a few large blocks of other documents. These collections arise in different areas, such as repositories of genomes of the same species (which differ from each other by a small percentage only) like the 100K-genome project222, software repositories that store all the versions of the code arranged in a tree or acyclic graph like GitHub333, versioned document repositories where each document has a timeline of versions like Wikipedia444, etc. On such text collections, statistical compression is ineffective [14] and even bits of extra space can be unaffordable.

Repetitiveness is the key to tackle the fast growth of these collections: their amount of new material grows much slower than their size. For example, version control systems compress those collections by storing the list of edits with respect to some reference document that is stored in plain form, and reconstruct it by applying the edits to the reference version. Much more challenging, however, is to index those collections in small space so as to support fast pattern matching or document retrieval tasks. To date, there exist several pattern matching indexes for repetitive text collections (see a couple of studies [21, 10] and references therein). However, there are not many document retrieval indexes for repetitive text collections [5, 8, 23]. Most of these indexes [26, 8] rely on a pattern-matching index needs bits in order to offer time per retrieved document.

In this paper we introduce new simple and efficient document listing indexes aimed at highly repetitive text collections. Like various preceding indexes, we achieve search time, yet our indexes are way faster and/or smaller than previous ones on various repetitive datasets, because they escape from the space/time tradeoff of the pattern-matching index. Our main idea is as follows: we use the document array [20], which projects the entries of the suffix array [19] to the document where each position belongs. Document listing boils down to listing the distinct integers in a range , where and are found in time . Array must be grammar-compressible since the differential suffix array is known to be so on repetitive texts [11, 10]. We then build a balanced binary context-free grammar that generates (only) . This allows us retrieve any individual cell of in time and any range in time . We can then implement existing indexes [26, 8] within much less space and without affecting their time complexities. Further, we propose a new simple index based on the grammar-compressed array . Our compression guarantees that any range is covered by nonterminals. For each nonterminal of the grammar, we store the list of the distinct documents appearing in it. The set of all the lists is grammar-compressed as well, as done in previous work [5, 8]. We then merge the lists of the nonterminals that cover , in time .

2 Preliminaries

A document is a sequence of symbols over an alphabet , terminated by a special symbol that is lexicographically smaller than any symbol of .

A collection is a set of documents . is commonly represented as the concatenation of all its documents, , of length .

A pattern is a string over the same alphabet with length . It occurs times in , and appears in documents.

Text indexes. The suffix tree [28] of a string is a compressed digital tree storing all the suffixes , for all . The suffix tree node reached by following the symbols of a pattern is called the locus of and is the ancestor of all the leaves corresponding to the positions of in . The suffix tree uses bits and lists all the occurrences of in time .

The suffix array [19] of a string is a permutation of the starting positions of all the suffixes of in lexicographic order, for all . can be binary searched to obtain the range of all the suffixes prefixed by (note ). Thus the occurrences of can be listed in time . The suffix array takes bits.

Compressed suffix arrays (s) [24] are space-efficient representations of the suffix array. They find the interval corresponding to in time , and access any cell in time . Their size in bits, , is usually bounded by .

Grammar compression. Grammar compression of a string replaces it by a context-free grammar (CFG) that uniquely generates . This CFG may require less space than the original sequence , especially when is repetitive.

Finding the smallest CFG generating the input is NP-hard [16], but various -approximations exist. In particular, we are interested in approximations that are binary (i.e., the maximum arity of the parse tree is 2) and balanced (i.e., any substring is covered by maximal nodes of the parse tree) [25, 3, 13].

3 Related Work

Muthukrishnan [20] proposed the first optimal-time linear-space solution to the document listing problem. He defines the document array of , where stores the identifier of the document to which belongs. The document listing problem is then translated into computing the distinct identifiers in the interval corresponding to the pattern . He uses a suffix tree to find and in time , and then an algorithm that finds the distinct numbers in the range in time .

Sadakane [26] adapts the method of Muthukrishnan to use much less space. He replaces the suffix tree by a CSA, and mimics the algorithm to find the distinct numbers in using only bits of space. Within bits, he performs document listing in time . Using a particular CSA [2] the space is bits and the time is .

There are many other classical and compact indexes for document listing. We refer the reader to a survey [22] and focus on those aimed at repetitive text collections.

Gagie et al. [8] proposed a technique adapting Sadakane’s solution to highly repetitive collections. They show that the technique to find the distinct elements of can be applied almost verbatim on an array they call interleaved longest-common-prefix array (). On repetitive collections, this array can be decomposed into a small number of equal values, which allows them represent it in little space. The ILCP index requires bits of space and solves document listing in time .

Gagie et al. [8] proposed another radically different approach, called Precomputed Document Lists (PDL). The idea is to store the list of the documents where (the corresponding substring of) each suffix tree node appears. Then the search consists of finding the locus of and returning its list. To reduce space, however, only some sampled nodes store their lists, and so document listing requires merging the lists of the maximal sampled nodes descending from the locus node. To further save space, the lists are grammar-compressed, which is effective when the collection is repetitive.

To bound the query time, the deepest sampled nodes cover at most leaves, and a factor restricts the work done per document to be merged in the unions of the lists. The index then requires bits, and the document listing time is , where is the height of the suffix tree.

A problem in all the revisited -based solutions are the extra bits that must be included in in order to get time per document. This space does not decrease with repetitiveness, forcing all these indexes to use bits to obtain time , for example.

Claude and Munro [5] propose the first index for document listing based on grammar compression, which escapes from the problem above. They extend a grammar-based pattern-matching index [6] by storing the list of the documents where each nonterminal appears. Those lists are grammar-compressed as well. The index searches for the minimal nonterminals that contain and merges their lists. While it does not offer relevant space or query time guarantees, the index performs well in practice. Navarro [23] extends this index in order to obtain space guarantees and time, but the scheme is difficult to implement.

4 Our Document Listing Index

Like most of the previous work, we solve the document listing problem by computing the distinct documents in the interval corresponding to the pattern , found with a in time . Instead of also using the to compute the values of (and thus facing the problem of using bits to compute a cell in time , as it happens in previous work [26, 8]), we store the array directly, yet in grammar-compressed form. This is promising because the suffix array of repetitive collections is known to have large areas that appear shifted by 1 elsewhere, , that is, for all [18, 10]. Except for the entries of that point to the ends of the documents, it also holds that . Grammar compression is then expected to exploit those large repeated areas in .

To answer the queries efficiently, we use an idea similar to the one introduced in PDL [8] and the Grammar-index [5]: precomputing and storing the answers of document listing queries, and grammar-compressing those lists as well. An important difference with them is that PDL stores lists for suffix tree nodes and the Grammar-index stores lists for nonterminals of the grammar of . Our index, instead, stores lists for the nonterminals of the grammar of . This is much simpler because we do not store a suffix tree topology (like PDL) nor a complex grammar-based pattern-matching index (like the Grammar-index): we simply find the interval using the , fetch the nonterminals covering it, and merge their lists. By using a binary balanced grammar on , we ensure that any document is obtained in the merging only times, which leads to our worst-case bound of . PDL and the Grammar-index cannot offer such a logarithmic-time guarantee.

4.1 Structure

The first component of our index is a suitable for repetitive collections, of which we are only interested in the functionality of finding the interval corresponding to a pattern . For example, we can use the Run-Length CSA (RLCSA) variant of Gagie et al. [10], which offers times within bits, or within bits, where is the number of equal-letter runs in the Burrows-Wheeler Transform of . This also upper-bounds the number of areas into which can be divided such that each area appears elsewhere shifted by 1 [17].

The second component is the grammar that generates , which must be binary and balanced. Such grammars can be built so as to ensure that their total size is bits [9], which is of the same order of the first component.

The third component are the lists of the distinct documents that appear in the expansion of each nonterminal of . These lists are stored in ascending order to merge them easily. To reduce their size, the set of sequences are grammar-compressed as a whole in a new grammar , ensuring that no nonterminal of expands beyond a list . Each list can then be obtained in optimal time, , from a nonterminal of .

4.2 Document listing

Given a pattern , we use the to find the range where the occurrences of lie in the suffix array, in time . We then find the maximal nodes of the parse tree of that cover . Finally, we decompress the lists of the nonterminals corresponding to those maximal nodes, and compute their union.

Since is binary and balanced, there are maximal nonterminals that cover in the parse tree. By storing the length to which each nonterminal of expands, we can easily find those maximal nonterminals in time , by (virtually) descending in the parse tree from the initial symbol of towards the area .

To merge the lists of documents in ascending order, we use an atomic heap [7] (see practical considerations in the next section). This data structure performs insert and extractmin operations in constant amortized time, when storing elements. We then insert the heads of the lists in the atomic heap, extract the minimum, and insert the next element of its list. If we extract the same document many times, we report only one copy. We then expand and merge the lists in time .

Since each distinct document we report may appear in the lists, our document listing solution takes time . By using the RLCSA that occupies bits, the total time is .

4.3 Plugging-in other indexes

Our grammar-compressed , without the lists , can be used to replace the CSA component that requires bits to compute a cell in time . These indexes actually access cells in order to obtain . Our grammar-compressed offers access time within bits of space.

Therefore, we can implement Sadakane’s solution [26], as well as ILCP and PDL [8] all answering in time , and replacing the part of their space by bits (which also accounts for the RLCSA variant that finds in time . We can also implement the brute-force solution in time and bits by extracting the whole .

5 Practical Considerations

5.1 Compressed suffix array

We use a practical RLCSA [18, called RLFM+ in there] that uses bits of space and offers search time in . Since we do not need to compute cells of with this structure, we do not need to spend the bits, and as a result the contribution of the RLCSA to the total space is negligible.

5.2 Grammar compressor

We choose Re-Pair [15] to obtain both and , since it performs very well in practice. Re-Pair repeatedly replaces the most frequent pair of adjacent symbols with a new nonterminal, until every pair is unique. Upon ties in frequency, we give priority to the pairs whose symbols have been generated earlier, which in practice yielded rather balanced grammars in all the cases we have tried.

Re-Pair yields a binary grammar, but the top-level is a sequence of terminals and nonterminals. We then complete the grammar by artificially adding a parse tree on top of the final sequence left by Re-Pair. To minimize the height of the resulting grammar, we merge first the pairs of nonterminals with shorter parse trees.

We store the grammar rules as an array taking bits of space, so that if is the th nonterminal of the grammar, it holds that .

When building , we concatenate all the lists and separate them with unique numbers larger than , to ensure that Re-Pair will not produce nonterminals that cross from one list to another. After running Re-Pair, we remove the separators but do not complete the grammars, as all we need is to decompress any in optimal time. We represent all the reduced sets as a sequence , marking the beginning of each set in a bitvector . The beginning of is found with operation , which finds the th 1 in . This operation can be implemented in constant time using further bits [4].

5.3 Sampling

The largest component of our index is the set of compressed lists . To reduce this space, we will store those lists only for some sampled nonterminals of . The list of a nonsampled nonterminal is then obtained by merging those of the highest sampled descendants of in the parse tree, which yields a space/time tradeoff.

We use a strategy similar to PDL [8], based on parameters and . We define a sampled tree by sampling some nodes from the parse tree. First, no leaf of the sampled tree can have an expansion larger than , so that we spend time to obtain its sorted list directly from . To this aim, we sample all the nonterminals of with parent such that . Those are the leaves of the sampled tree, which form a partition of .

Second, for any nonsampled node with , we must be able to build by merging other precomputed lists of total length . This implies that generating costs times more than having stored and just decompressing it.

We first assume that the sampled tree contains all the ancestors of the sampled leaves and then proceed bottom-up in the sampled tree, removing some nodes from it. Any node with parent and children is removed if . In this case, the nodes become children of in the sampled tree.

At query time, if a node of interest is not sampled, we collect all the lists of its highest sampled descendants. Therefore, on a parse tree of height we may end up merging many more than the original lists , but have the guarantee that the merged lists add up to size at most . To merge the lists we use a classical binary heap instead of an atomic heap, so the cost per merged element is .

We may then spend time in extracting and sorting the lists of size below . The other lists may lead to merging elements. The total cost over the lists is then . In terms of complexity, if we choose for example , , and the grammar is balanced, , then the total cost of merging is .

6 Experiments and Results

We evaluate different variants of our indexes and compare them with the state of the art. We use the experimental framework proposed by Gagie et al. [8].

6.1 Document collections

To test various kinds of repetitiveness scenarios, we performed several experiments with real and synthetic datasets. We used the same document collections tested by Gagie et al. [8], available at Table 1 summarizes some statistics on the collections and the patterns used in the queries.

l r r r r r r r r Collection & Size & RLCSA & Docs & Doc size & Patterns & Occs & Doc occs & Occs/doc
& () & (bps) & () & () & & () & () & ()

Page & 110 & 0.18 & & & & & & 242.75
& 641 & 0.11 & & & & & & 444.79
& 1037 & 0.13 & & & & & & 429.04

Revision & 110 & 0.18 & & & & & & 2.09
& 640 & 0.11 & & & & & & 2.43
& 1035 & 0.13 & & & & & & 2.42

Influenza & 137 & 0.32 & & & & & & 6.02
& 321 & 0.26 & & & & & & 6.17

Table 1: Statistics for document collections (small, medium, and large variants): Collection name; Size in megabytes; RLCSA bits per symbol (bps); Docs, number of documents; Doc size, average document length; number of Patterns; Occs, average number of occurrences; Doc occs, average number of document occurrences; Occs/doc, average ratio of occurrences to document occurrences.

Real collections. Page and Revision are collections formed by all the revisions of some selected pages from the Wikipedia in Finnish language. In Page, there is a document for each selected article, that also includes all of its revisions. In the case of Revision, each page revision becomes a separate document. Influenza is another repetitive collection composed of sequences of the H. influenzae virus genomes.

Synthetic collections. We also used two types of synthetic collections to explore the effect of collection repetitiveness on document listing performance in more detail. Concat and Version are similar to Page and Revision, respectively. We use 10 and 100 base documents of length each, extracted at random from the English file of Pizza&Chili ( Besides, we include variants of each base document, generated using different mutation probabilities (, , , and ). A mutation is a replacement by a different random symbol. In collection Version, each variant becomes a separate document. In Concat, all variants of the same base document are concatenated into a single document.

Queries. The query patterns for Page and Revision datasets are Finnish words of length that occur in the collections. For Influenza, the queries are substrings of length extracted from the dataset. In the case of Concat and Version, the patterns are terms selected from an MSN query log. See Gagie et al. [8] for a more detailed description.

6.2 Compared indexes

Grammar-Compressed Document Array (Gcda). This is our main proposal. We use the balanced variant of the Re-Pair compressor implemented by Navarro555 To sample the parse tree, we test several parameter configurations for the block size and factor .

Brute force (Brute). This family of algorithms is the most basic and simple solution to the document listing problem. They use a to retrieve all the document identifiers in , sort them, and report each of them once. Brute-L uses the to extract the values . Brute-D, instead, uses an explicit document array . Finally, Brute-C is our variant using the grammar-compressed . From the grammar tree of height and storing the length of the expansion of each nonterminal, we extract the range in time .

Sadakane (Sada). Sada-L is the original index proposed by Sadakane [26]. Sada-D speeds up the query time by explicitly storing . Sada-C stores in grammar-compressed form, where each individual cell is extracted in time .

Interleaved Longest Common Prefix (Ilcp). Ilcp-L is an implementation of the ILCP index proposed by Gagie et al. [8] using a run-length encoded Ilcp array. Ilcp-D is a variant that uses the document array instead of the functionality. Ilcp-C uses, instead, our grammar-compressed , which accesses any cell in time .

Precomputed Document Lists (Pdl). Pdl-Bc and Pdl-Rp are implementations of the PDL algorithm proposed by Gagie et al. [8]. The first one uses a Web graph compressor [12] on the set of lists, whereas Pdl-Rp uses Re-Pair compression. Both variants use block size and factor , as recommended by their authors.

Grammar-based (Grammar). This is an implementation of the index by Claude and Munro [5]. It uses Re-Pair on the collection and on the set of lists. This index is the only tested solution that does not use a .

We implemented GCDA on C++, using several succinct data structures from the SDSL library666 We used existing C++ implementations of the indexes Brute, Sada, ILCP and PDL, which were tested by Gagie et al. [8]777, and modified the versions -C by using in grammar-compressed instead of in plain form.

All tested indexes except Grammar use a suffix array to compute the interval corresponding to pattern . We used a RLCSA implementation888 that is optimized for highly repetitive text collections. To compute entries , the RLCSA uses a suffix array sampling, which requires significant space as explained. Our index does not use this operation, but it is required for the indexes Brute-L, Sada-L, ILCP-L, and both variants of PDL. We use 32 as the value for this sample rate, as it gave good results in previous tests [8]. The exception is Brute-L, which uses a RLCSA optimized to extract whole ranges [10].999 The column RLCSA of Table 1 gives the space used by the RLCSA without suffix array samples.

Our machine has two Intel(R) Xeon(R) CPU E5-2407 processors running at and of RAM. The operating system was Debian with Linux kernel 4.9.0-8-amd64. All indexes were compiled using g++ version 6.3.0 with flags -O3 -DNDEBUG.

6.3 Tuning our main index

Figure 1 shows the tradeoff between time and space of GCDA on small real collections. We tested GCDA with 4 different sizes of block : , , , and . For each block size, we used 3 different factors (, , and ), which are represented with increasing color darkness in the plots. The configuration and shows to be a good general-purpose choice of parameter values, and we stick to it from now on.

Figure 1: GCDA on small real collections with different configurations. The axis shows the total size of the index in bits per symbol. The axis shows the average time per query in microseconds. Beware that the plots do not start at zero.

The lower-right plot of Figure 2 shows the space required by the main components of our index. As the number of documents in the collection grows and their size decreases, the weight of the grammar-compressed , and even more, of the grammar-compressed lists of documents, becomes dominant. Note also that Influenza is the least repetitive collection.

6.4 Comparison on real collections

Figure 2: Document listing indexes on real repetitive collections. The axis shows the total size of the index in bits per symbol. The axis shows the average time per query. Combinations with excessively high time are omitted in some plots. The lower-right plot shows the size of the main components of GCDA on the small collections; the axis shows the size in megabytes.

Figure 2 shows the tradeoff between time and space for all tested indexes on the real collections. Our main index, GCDA, and the -C variants of the other indexes we adapted, are clearly dominant in a large portion of the space/time map. Most of the previous indexes are way slower, way larger, or both, than ours. The best previous tradeoffs, PDL-BC and PDL-RP [8], are much closer, but still they are almost always slower and larger than GCDA.

For all versions of Page, where there are few large documents and our grammars compress very well, GCDA requires only bits per symbol (bps) and answers queries in less than 16 microseconds. The index using the least space is Grammar, which requires bps. Grammar is way out of the plot, however, because it requires 1.2–3.4 milliseconds to solve the queries, that is, 205–235 times slower than GCDA.101010As in previous work [8], Grammar was not built on the largest dataset of Page. The next smallest index is our variant Brute-C, which uses 0.35–0.55 bps and is generally smaller than GCDA, but slower by a factor of 2.6–6.7. Brute-L, occupying 0.38–0.60 bps, is also smaller in some cases, but much slower (180–1080 microseconds, out of the plot). GCDA sharply outperforms all the other indexes in space, and also in time (only Sada-D is 6% faster in the small collection, yet using 18 times more space). The closest competitors, PDL-BC and PDL-RP, are 4.4–5.0 times larger and times slower than GCDA.

In the case of Revision, where there are more and smaller documents, GCDA uses 0.73–0.88 bps and answers queries in less than 150 microseconds. Again Grammar uses the least space, 0.26–0.42 bps, but once again at the price of being times slower than GCDA. The case of Brute-L is analogous: 0.38–0.60 bps but over 8 times slower than GCDA. Instead, our variant Brute-C is a relevant competitor, using 0.45–0.76 bps and being less than 60% slower than GCDA. The other relevant index is our variant ILCP-C, using almost the same space and time of GCDA. The group GCDA/Brute-C/ILCP-C forms a clear sweetpoint in this collection. The closest competitors, again PDL-BC and PDL-RP, are 3.1–3.8 times larger and 1.2–1.9 times slower than GCDA.

Influenza, with many small documents, is the worst case for the indexes. GCDA uses 4.46–4.67 bps and answers queries within 115 milliseconds. Many indexes are smaller than GCDA, but only our variants form a relevant space/time tradeoff: ILCP-C uses 2.88–3.37 bps, Brute-C uses 2.42–2.86 bps, and Sada-C uses 4.96–5.40 bps. All the -C variants obtain competitive times, and ILCP-C even dominates GCDA (it answers queries within 65 milliseconds, taking less than 60% of the time of GCDA). The other indexes outperforming GCDA in time are -D variants, which are at least 3.7 times larger than GCDA and 5.2 times larger than ILCP-C.

6.5 Comparison on synthetic collections

Figure 3 compares the indexes on synthetic collections. These allow us study how the indexes evolve as the repetitiveness decreases, in a scenario of few large documents (Concat) and many smaller documents (Version). We combine in a single plot the results for different mutation rates of a given collection and number of base documents. The plots show the increasing mutation rates using variations of the same color, from lighter to darker. All the -L variants and Grammar are omitted because they were significantly slower.

Figure 3: Document listing on synthetic collections. The axis shows the total size of the index in bits per symbol. The axis shows the average time per query in microseconds. Combinations with excessively high time are omitted in some plots.

On collection Concat, GCDA essentially outperforms all the other indexes. In the case of the version composed by base documents, our index obtains the best space/time tradeoff by a wide margin. Only Brute-C is smaller than GCDA, but 8–9 times slower. On the other hand, various indexes are slightly faster than GCDA, but much larger (from Sada-D, which is up to 30% faster but 7 times larger, to Sada-C, which is 15% faster but at least 4 times larger). With the other variant of Concat ( base documents), our index offers the best space and time for all mutation rates. Only PDL-RP is 6% faster in its best case, but 2.2 times larger. Further, GCDA retains its space/time performance as repetitiveness decreases, whereas the competing indexes worsen fast in one or both aspects.

On Version, composed by documents of length , GCDA is also a dominant solution, retaining its time performance as repetitiveness decreases and outperforming all the -D variants in space up to a mutation rate of 1%. Other competing indexes are our variants Brute-C and ILCP-C (the only one dominating GCDA in some cases), as well as PDL-BC and PDL-RP in the case of 100 base documents. The strange behavior of the PDL indexes in both collections with 10 base documents is briefly discussed in the original article [8].

7 Conclusions

We have presented simple and efficient indexes for document listing on repetitive string collections. They find the documents where a pattern of length appears in a collection of size in time . The indexes uses grammar-compression of the document array, and perform better as the collection is more repetitive.

Our experimental results show that our main index, GCDA, outperforms the best previous solutions by a fair margin in time and/or space on various repetitive collections. From the previous indexes, only PDL [8] gets close, but it is almost always dominated by GCDA in both space and time. GCDA performs well in space for mutation rates up to 1%, whereas its query time is mostly insensitive to the repetitiveness. Other previous solutions (especially ILCP [8] and brute force) that we adapted to run on our grammar-compressed document array also display unprecedented performance on repetitive texts, competing with GCDA.

For the final version of this paper, we plan to combine the PDL indexes with a grammar-compressed document array as well, which we omitted for lack of time. A line of future work is to further reduce the space of GCDA and our index variants that use the grammar-compressed document array, by using a more clever encoding of the grammars that may nearly halve their space at a modest increase in time [11]. Another line is to extend the index to support top- document retrieval, that is, find the documents where appears most often. For example, following previous ideas [8], we can store the list of documents where each nonterminal appears in decreasing order of frequency, and use algorithms developed for inverted indexes [1] on the lists involved in a query. The frequency of the candidates can be efficiently counted on repetitive collections [8].


  • [1] R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley, 2nd edition, 2011.
  • [2] D. Belazzougui and G. Navarro. Alphabet-independent compressed text indexing. ACM Transactions on Algorithms, 10(4):article 23, 2014.
  • [3] M. Charikar, E. Lehman, D. Liu, R. Panigrahy, M. Prabhakaran, A. Sahai, and A. Shelat. The smallest grammar problem. IEEE Transactions on Information Theory, 51(7):2554–2576, 2005.
  • [4] D. R. Clark. Compact PAT Trees. PhD thesis, University of Waterloo, Canada, 1996.
  • [5] F. Claude and J. I. Munro. Document listing on versioned documents. In Proc. 20th Symposium on String Processing and Information Retrieval (SPIRE), LNCS 8214, pages 72–83, 2013.
  • [6] F. Claude and G. Navarro. Self-indexed grammar-based compression. Fundamenta Informaticae, 111(3):313–337, 2010.
  • [7] M. L. Fredman and D. E. Willard. Trans-dichotomous algorithms for minimum spanning trees and shortest paths. Journal of Computer and System Sciences, 48(3):533–551, 1994.
  • [8] T. Gagie, A. Hartikainen, K. Karhu, J. Kärkkäinen, G. Navarro, S. J. Puglisi, and J. Sirén. Document retrieval on repetitive collections. Information Retrieval, 20:253–291, 2017.
  • [9] T. Gagie, G. Navarro, and N. Prezza. Fully-functional suffix trees and optimal text searching in BWT-runs bounded space. CoRR, abs/1809.02792, 2018.
  • [10] T. Gagie, G. Navarro, and N. Prezza. Optimal-time text indexing in BWT-runs bounded space. In Proc. 29th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1459–1477, 2018.
  • [11] R. González, G. Navarro, and H. Ferrada. Locally compressed suffix arrays. ACM Journal of Experimental Algorithmics, 19(1):article 1, 2014.
  • [12] C. Hernández and G. Navarro. Compressed representations for web and social graphs. Knowledge and Information Systems, 40(2):279–313, 2014.
  • [13] A. Jez. A really simple approximation of smallest grammar. Theoretical Computer Science, 616:141–150, 2016.
  • [14] S. Kreft and G. Navarro. On compressing and indexing repetitive sequences. Theoretical Computer Science, 483:115–133, 2013.
  • [15] J. Larsson and A. Moffat. Off-line dictionary-based compression. Proceedings of the IEEE, 88(11):1722–1732, 2000.
  • [16] E. Lehman and A. Shelat. Approximation algorithms for grammar-based compression. In Proc. 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 205–212, 2002.
  • [17] V. Mäkinen and G. Navarro. Succinct suffix arrays based on run-length encoding. Nordic Journal of Computing, 12(1):40–66, 2005.
  • [18] V. Mäkinen, G. Navarro, J. Sirén, and N. Välimäki. Storage and retrieval of highly repetitive sequence collections. Journal of Computational Biology, 17(3):281–308, 2010.
  • [19] U. Manber and G. Myers. Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing, 22(5):935–948, 1993.
  • [20] S. Muthukrishnan. Efficient algorithms for document retrieval problems. In Proc. 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 657–666, 2002.
  • [21] G. Navarro. Indexing highly repetitive collections. In Proc. 23rd International Workshop on Combinatorial Algorithms (IWOCA), LNCS 7643, pages 274–279, 2012.
  • [22] G. Navarro. Spaces, trees and colors: The algorithmic landscape of document retrieval on sequences. ACM Computing Surveys, 46(4):article 52, 2014.
  • [23] G. Navarro. Document listing on repetitive collections with guaranteed performance. In Proc. 28th Annual Symposium on Combinatorial Pattern Matching (CPM), LIPIcs 78, page article 4, 2017.
  • [24] G. Navarro and V. Mäkinen. Compressed full-text indexes. ACM Computing Surveys, 39(1):article 2, 2007.
  • [25] W. Rytter. Application of Lempel-Ziv factorization to the approximation of grammar-based compression. Theoretical Computer Science, 302(1-3):211–222, 2003.
  • [26] K. Sadakane. Succinct data structures for flexible text retrieval systems. Journal of Discrete Algorithms, 5:12–22, 2007.
  • [27] Z. D. Sthephens, S. Y. Lee, F. Faghri, R. H. Campbell, Z. Chenxiang, M. J. Efron, R. Iyer, S. Sinha, and G. E. Robinson. Big data: Astronomical or genomical? PLoS Biology, 17(7):e1002195, 2015.
  • [28] P. Weiner. Linear pattern matching algorithm. In Proc. 14th Annual IEEE Symposium on Switching and Automata Theory, pages 1–11, 1973.