Local Decodability of the Burrows-Wheeler Transform
share
The Burrows-Wheeler Transform (BWT) is among the most influential discoveries in text compression and DNA storage. It is a reversible preprocessing step that rearranges an n-letter string into runs of identical characters (by exploiting context regularities), resulting in highly compressible strings, and is the basis for the ubiquitous bzip program. Alas, the decoding process of BWT is inherently sequential and requires Ω(n) time even to retrieve a single character. We study the succinct data structure problem of locally decoding short substrings of a given text under its compressed BWT, i.e., with small redundancy r over the Move-To-Front based (bzip) compression. The celebrated BWT-based FM-index (FOCS '00), and other related literature, gravitate toward a tradeoff of r=Õ(n/√(t)) bits, when a single character is to be decoded in O(t) time. We give a near-quadratic improvement r=Õ(n· t/t). As a by-product, we obtain an exponential (in t) improvement on the redundancy of the FM-index for counting pattern-matches on compressed text. In the interesting regime where the text compresses to n^1-o(1) bits, these results provide an (t) overall space reduction. For the local decoding problem, we also prove an Ω(n/t^2) cell-probe lower bound for "symmetric" data structures. We achieve our main result by designing a compressed Rank (partial-sums) data structure over BWT. The key component is a locally-decodable Move-to-Front (MTF) code: with only O(1) extra bits per block of length n^Ω(1), the decoding time of a single character can be decreased from Ω(n) to O( n). This result is of independent interest in algorithmic information theory.
READ FULL TEXT