Fast decoder for VByte-compressed integers
Arrays of integers are often compressed in search engines. Though there are many ways to compress integers, we are interested in the popular byte-oriented integer compression techniques (e.g., VByte or Google's Varint-GB). They are appealing due to their simplicity and engineering convenience. Amazon's varint-G8IU is one of the fastest byte-oriented compression technique published so far. It makes judicious use of the powerful single-instruction-multiple-data (SIMD) instructions available in commodity processors. To surpass varint-G8IU, we present Stream VByte, a novel byte-oriented compression technique that separates the control stream from the encoded data. Like varint-G8IU, Stream VByte is well suited for SIMD instructions. We show that Stream VByte decoding can be up to twice as fast as varint-G8IU decoding over real data sets. In this sense, Stream VByte establishes new speed records for byte-oriented integer compression, at times exceeding the speed of the memcpy function. On a 3.4GHz Haswell processor, it decodes more than 4 billion differentially-coded integers per second from RAM to L1 cache.READ FULL TEXT VIEW PDF
Fast decoder for VByte-compressed integers
A port of Stream VByte to Go
We frequently represent sets of document or row identifiers by arrays of integers. Compressing these arrays can keep the data closer to the processor and reduce bandwidth usage, and fit more data in memory or on a disk. Though data can always be compressed using generic algorithms such as Lempel-Ziv coding (LZ77), specialized compression algorithms for integers can be orders of magnitude faster. We consider codecs to compress 32-bit unsigned integers (in ). We are especially interested in compressing arrays where most integers are small. In addition to such arrays that arise naturally, sorted arrays of non-small integers can often be treated as arrays of small integers by considering the successive differences (“deltas”). So instead of compressing the integers directly, we can compress the gaps between them There are many integer-compression algorithms applicable to arrays of small integers. One option are byte-oriented techniques Stepanov et al. (2011). In these formats, the main data corresponding to an integer is stored in consecutive whole bytes, and all bits within a given byte correspond to only one such integer. Though byte-oriented formats do not offer the best compression ratios or the best speeds, they are in widespread use within databases, search engines and data protocols in part because of their simplicity.
Historically, the most widely known byte-oriented compression algorithm is VByte. It writes a non-negative integer starting from the least significant bits, using seven bits in each byte, with the most significant bit set to zero when the following byte continues the current integer. Thus integers in are coded using a single byte, integers in in two bytes and so on. For example, the integer 32 is written using a single byte (00100000) and the integer 128 is written using two bytes (10000000 and 00000001).
To decode VByte data, it suffices to iterate over the compressed bytes while checking the value of the most significant bit. Whenever the number of bytes required per integer is easily predictable—such as when most integers fit in —the absence of branch prediction errors allows high decoding speeds on modern-day superscalar processors capable of speculative execution.
Unfortunately, not all data is easily predictable. In some of these case, the decoding speed of VByte compressed data becomes a bottleneck. For this reason, Google developed varint-GB Dean (2009). In varint-GB, numbers are compressed and decompressed in blocks of four. Instead of having to check one bit per compressed byte, only one control byte for every four integers needs to be processed. For unpredictable patterns, this block-based design can reduce the number of branch mispredictions by a factor of four.
Modern processors (e.g., ARM, POWER, Intel, AMD) have instructions that perform the same operation on multiple scalar values (e.g., the addition of two sets of four 32-bit integers). These instructions are said to be single-instruction-multiple-data (SIMD) while the algorithms designed to take advantage of SIMD instructions are said to be vectorized.
Stepanov et al. Stepanov et al. (2011) considered the problem of vectorizing byte-oriented compression. They reported that there was little benefit to vectorizing VByte decoding. As an alternative to VByte and varint-GB, Stepanov et al. proposed a new patented byte-oriented format (varint-G8IU) designed with SIMD instructions in mind. It proved more than three times faster than VByte on realistic sets of document identifiers.
Plaisance et al. Plaisance et al. (2015) revisted the VByte decoding problem: unlike Stepanov et al., their Masked VByte decoder is twice as fast as a scalar VByte decoder. Though it should not be expected to be as fast as a decoder working on a format designed for SIMD instructions (e.g., varint-G8IU), it can help in systems where the data format is fixed.
There remained an open question: could varint-G8IU be surpassed? Could byte-oriented decoding be even faster? We answer this question by the positive. In fact, our proposal (Stream VByte) can be twice as fast as varint-G8IU on realistic data.
Since the introduction of the Pentium 4 in 2001, x64 processors have had vector instructions operating on 16-byte SIMD registers (called XMM registers). These registers can store four 32-bit integers.
A Stream VByte decoder (§ 4) can be written with just two x64 SIMD assembly instructions:
The movdqu instruction can load 16 bytes from memory into a SIMD register, or write such a register to memory. On recent Intel processors (e.g., Haswell) these operations have multicycle latency (5–6 cycles from L1 cache), but they also have high throughput. In ideal cases, two XMM loads and an XMM write can all be issued each CPU cycle.
The shuffle (pshufb) instruction can selectively copy the byte values of one SIMD register to another according to a mask . If are the values of the 16 individual bytes in , and are the bytes within (), then pshufb outputs where . Once its operands are in registers, the pshufb instruction is fast: it has a latency of one cycle and a reciprocal throughput of one instruction per cycle.
Both of these SIMD instructions are available in all processors supporting the SSSE3 instruction set, i.e., almost all x64 processors produced since 2010.
When we have compressed “deltas”, or successive differences instead of the original integers, we need to compute a prefix sum to recover the original integers (). The computation of the prefix sum can be accelerated by vector shifts and additions. We can compute the prefix sum of a vector of size using shifts and additions Lemire et al. (2016). For example, consider the vector of delta values . We add to this vector a version of itself shifted by one integer () to get . Finally, we add to this last vector a version of itself shifted by two integers () to get the prefix sum of the original vector.
The VByte format uses the most significant bit of each byte as a control bit. Thus an integer is compressed to an array of bytes such that have their most significant bits set to 1 whereas the most significant bit of is zero. That is, if we view bytes as unsigned integers in , we have that and . We can decode the compressed integer as . Integers in are coded using bytes for The software to decode the VByte format is easy to write in standard C, but performance on inputs with mixed byte lengths can suffer due to poor branch prediction.
For greater speed, Plaisance et al. Plaisance et al. (2015) proposed the Masked VByte decoder that uses SIMD instructions. It works directly on the standard VByte format. The Masked VByte decoder gathers the most significant bits of an array of consecutive bytes using the pmovmskb x64/SSE2 instruction. Using look-up tables and a shuffle instruction (pshufb), Masked VByte permutes the bytes to arrive at the decoded integers. We refer the interested reader to Plaisance et al. Plaisance et al. (2015) for a detailed description.
|name of decoder||data format||SIMD|
|VByte||7 data bits per byte, 1 bit as continuation flag||no|
|Masked VByte Plaisance et al. (2015)||identical to VByte||yes|
|varint-GB Dean (2009)||fixed number of integers (4) compressed to a variable number of bytes (4–16), prefixed by a control byte||no|
|varint-G8IU Stepanov et al. (2011)||fixed number of compressed bytes (8) for a variable number of integers (2–8), prefixed by a control byte||yes|
|Stream VByte (novel)||control bytes and data bytes in separate streams||yes|
Whereas the VByte format interleaves actual integer data (using the least significant 7 bits of each byte) with control data (using the most significant bit of each byte), varint-GB stores the control data using a distinct control byte corresponding to a block of four integers Dean (2009). See Table 1 and Fig. 3. To store an integer , we use continuous bytes, since always holds. Given four integers, , the control byte stores the lengths for each of the four integers . Indeed, because there are only four possible byte lengths (), each of the four individual byte lengths can be stored using just two bits. Thus a block in varint-GB used to store integers contains a control byte followed by bytes. When the number of integers is not divisible by four, an incomplete block might be used. For this reason, it might be necessary to record how many integers are compressed (e.g., as a format header).
To decode varint-GB data, the control byte is first parsed, from which we extract the four lengths. The integers are then extracted (without any branching) and we advance to the next control byte. To achieve higher speed for highly compressible data, we find it useful to include an optimized code path for when all integers are small ().
Like varint-GB, varint-G8IU also uses a control byte Stepanov et al. (2011), but it does not describe a fixed number (four) of integers. Rather, varint-G8IU divides the compressed output into blocks of eight data bytes for each control byte. Each block can contain between two and eight compressed integers. See Fig. 6. The control byte describes the next eight data bytes. Thus a compressed block in varint-G8IU is always made of exactly nine compressed bytes. Each bit in the control byte corresponds to one of the eight data bytes. A bit value of 0 indicates that the data byte completes a compressed integer, whereas a bit value of 1 indicates that the data byte is part of a compressed byte or is “wasted”. Indeed, as shown in Fig. (b)b, if the control byte ends with trailing ones, then they correspond to bytes that are wasted (they do not correspond to a compressed integer). A case with waste like the example of Fig. (b)b might arise if the next integer to be coded is larger or equal to so that it cannot fit in the remaining byte.
varint-G8IU can sometimes have worse compression than VByte due to the wasted bytes. However, varint-G8IU can compress a stream of integers in using only nine bits per integer on average, against bytes for VByte, so varint-G8IU can also, in principle, compress better. varint-GB has a slight size disadvantage compared to varint-G8IU when compressing streams of integers in as it uses ten bits per integer against only nine bits per integer for varint-G8IU. However, this is offset by varint-GB’s better compression ratio for larger integers.
Stepanov et al. Stepanov et al. (2011) attempted to accelerate varint-GB by using SIMD instructions. The results were far inferior to varint-G8IU—despite the apparent similarity between the formats, with each having one control byte followed by some data.
To understand why it might be difficult to accelerate the decoding of data compressed in the varint-GB format compared to the varint-G8IU format, consider that we cannot decode faster than we can access the control bytes. In varint-G8IU, the control bytes are conveniently always located nine compressed bytes apart. Thus while a control byte is being processed, or even before, our superscalar processor can load and start processing upcoming control bytes, as their locations are predictable. Instructions depending on these control bytes can be reordered by the processor for best performance. However, in the varint-GB format, there is a strong data dependency: the location of the next control byte depends on the current control byte. This increases the risk that the processor remains underutilized, delayed by the latency between issuing the load for the next control byte and waiting for it to be ready.
But the varint-G8IU format has its own downside: it decodes a variable number of integers per control bytes (between two and eight inclusively). We expect that it is faster to store full SIMD registers (e.g., four integers) to memory with each iteration. Moreover, when using differential coding, it is more convenient and efficient to reconstruct the original integers from their deltas when they come in full SIMD registers.
Thankfully, we can combine the benefits of the varint-GB and the varint-G8IU formats: (1) having control bytes at predictable locations so that the processor can access series of control bytes simultaneously, without data dependency and (2) decoding integers in full SIMD registers. We use a format that is identical to that of varint-GB with the small, but important, difference that the control bytes are stored continuously in a separate stream from the data bytes. See Fig. 7. Since we record how many integers are compressed, we can use the first compressed bytes to store the control bytes followed by the data bytes.
1// "databytes" is a byte pointer to compressed data
2// "control" contains control byte
3uint8_t C = lengthTable[control]; // C is between 4 and 16
4__m128i Data = _mm_loadu_si128((__m128i *) databytes);
5__m128i Shuf = _mm_loadu_si128(shuffleTable[control]);
6Data = _mm_shuffle_epi8(Data, Shuf); // final decoded data
7datasource += C;
Decoding a block requires no more than a handful of lines of code (see Fig. 8). At all times, we maintain a pointer into the stream of control bytes and a pointer into the stream of data bytes. They are initialized to the respective beginning of their streams.
From the control byte, we load a 16-byte shuffling mask for the pshufb instruction (line 8). There are 256 such masks, one for each possible value of the control byte.
Both pointers are advanced. The control-byte pointer is advanced by one byte whereas the data-byte pointer is advanced by bytes (line 8).
Incomplete blocks (containing fewer than four integers) are decoded using a scalar function similar to that used for varint-GB. Likewise, when we detect that fewer than 16 data bytes remain, we use a scalar function. Further, we found it useful to use an optimized code path when we have four zero control bytes in sequence.
When the data is highly compressible, the Stream VByte format stores long runs of control bytes set to zero. In some applications, it might be beneficial to compress such runs to improve compression (e.g., using run-length encoding).
Though details are outside our scope, we have implemented fast functions to append new integers to a compressed Stream VByte array without having to recompress the data. We append extra data, and occasionally grow the control bytes stream.
We implemented our compression software in C. We use the GNU GCC 4.8 compiler with the -O3 flag. To ease reproducibility, our software is available online.111https://goo.gl/6Op1t4 and https://github.com/lemire/streamvbyte
We run the benchmark program on a Linux server with an Intel i7-4770 processor (). This Haswell processor has of L1 data cache and of L2 cache per core with of shared L3 cache. The machine has ample memory ( of dual-channel DDR3 1600 RAM). Turbo Boost and Speed Step are disabled, so the processor runs consistently at its rated speed of . We measure wall-clock times to decompress data from RAM to L1 cache. All tests are single-threaded.
Search engines typically rely on posting lists: given a term, we create a list of all document identifiers corresponding to documents where the term appears. Instead of directly storing the identifiers, we use differential coding to reduce the average number of bits required to store each compressed integer. The document identifiers are sorted in increasing order ( where for ), and we compress their successive differences (e.g., ). To recover the original identifiers during decompression, we need to compute a prefix sum (). When using vectorized formats, we vectorize the computation of the prefix sum. All decoding times include differential coding to reconstruct the original integers.
We use a collection of posting lists extracted from the ClueWeb09 (Category B) data set. ClueWeb09 includes 50 million web pages. We have one posting list for each of the 1 million most frequent words—after excluding stop words and applying lemmatization. Documents are sorted lexicographically based on their URL prior to attributing document identifiers.222The posting-list data is freely available (http://goo.gl/DygoQM).
We want to sort the posting lists by compressibility. For this purpose, the posting lists are grouped based on length: we store and process lists of lengths to together for all values of needed by the corpus. While the correlation is not perfect, shorter lists tend to be less compressible than longer lists since their gaps tend to be larger.
We decode the compressed data sequentially to a buffer of 32-bit integers (half the size of the L1 cache). For each group and each decoder, we compute the average decoding speed in billions of 32-bit integers per second (Bis). So that disk access is not an issue, the data is loaded in memory prior to processing. The length of the arrays, their compressibility and the volume of data varies from group to group.
Our results are summarized in Fig. (a)a. For Stream VByte, the reported speed ranges from for highly compressible data to for less compressible data. The second fastest codec is varint-G8IU with speeds ranging from to . Next we have Masked VByte and varint-GB with speeds ranging from to . Though varint-GB is more than faster than Masked VByte when the data is poorly compressible, they are almost tied speed-wise when the data is more compressible. Finally, we find VByte with speed ranging from to . For all groups of posting lists, Stream VByte is fastest of the algorithms considered. It is always at least faster than the conventional VByte decoder and sometimes nearly faster. Compared to varint-G8IU, Stream VByte can be twice as fast. For reference, we also provide the copy speed of the uncompressed data by blocks of up to integers using the C function memcpy. For highly compressible data, Stream VByte is faster than memcpy because fewer bytes are read. In our worst case, Stream VByte decompresses at of the corresponding memcpy speed.
Though they are faster, both Stream VByte and varint-G8IU use .5 to 2 more bits per integer than VByte in these tests: see Fig. (b)b. Stream VByte and varint-GB have almost exactly the same storage requirements due to their similar format. Also Masked VByte and VByte have exactly the same compressed format.
Byte-oriented codecs make it convenient to program advanced operations directly on the compressed stream. We benchmark two such operations: (1) we seek the location of the first value greater or equal to a given target, and retrieve this value, and (2) we select the integer. Given a bit width , we first generate an array of 256 random integers in : . The prefix sum is computed () and used as input data. We omit varint-G8IU in this experiment. In Fig. (a)a, we randomly seek a value in range. In Fig. (b)b, we randomly select the value at one of the indexes. In these tests, Stream VByte offers the best speed and is up to three times faster than VByte, with intermediate results for other codecs. The performance is noticeably better when all deltas compress down to a single byte due the simple and predictable code path: it happens when all deltas fit in 7 bits for VByte and Masked VByte, and when they all fit in 8 bits for varint-GB and Stream VByte.
This work is supported by the National Research Council of Canada, under grant 26143. We thank L. Boystov from CMU for sharing the posting list collection.