Hopping over Big Data: Accelerating Ad-hoc OLAP Queries with Grasshopper Algorithms

This paper presents a family of algorithms for fast subset filtering within ordered sets of integers representing composite keys. Applications include significant acceleration of (ad-hoc) analytic queries against a data warehouse without any additional indexing. The algorithms work for point, range and set restrictions on multiple attributes, in any combination, and are inherently multidimensional. The main idea consists in intelligent combination of sequential crawling with jumps over large portions of irrelevant keys. The way to combine them is adaptive to characteristics of the underlying data store.



There are no comments yet.


page 5

page 28


Structured abbreviation expansion in context

Ad hoc abbreviations are commonly found in informal communication channe...

A Line in the Sand: Recommendation or Ad-hoc Retrieval?

The popular approaches to recommendation and ad-hoc retrieval tasks are ...

Intelligent Emergency Message Broadcasting in VANET Using PSO

The new type of Mobile Ad hoc Network which is called Vehicular Ad hoc N...

Raptor Zonal Statistics: Fully Distributed Zonal Statistics of Big Raster + Vector Data [Pre-Print]

Recent advancements in remote sensing technology have resulted in petaby...

Efficient Path Routing Over Road Networks in the Presence of Ad-Hoc Obstacles (Technical Report)

Nowadays, the path routing over road networks has become increasingly im...

Dynamic Interleaving of Content and Structure for Robust Indexing of Semi-Structured Hierarchical Data (Extended Version)

We propose a robust index for semi-structured hierarchical data that sup...

An Extended Analytic Model for the Heating of Bondwires

We present an extended analytic formula for the calculation of the tempe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Business Intelligence (BI) applications, in their quest to provide decision support, rely on OLAP and Data Mining techniques to extract actionable information from collected data. Modern applications are characterized by massive volumes of data and extensive empowerment of users resulting in requirements to ensure fast response for ad-hoc queries against this data.

Classical relational data warehousing techniques [5] are often combined or replaced nowadays with non-relational distributed processing systems. One way to interpret this trend in the context of ad-hoc queries is to observe that maintaining adequate indexing for massive data volumes becomes impractical and perhaps the only remaining strategy for ad-hoc queries is the “brute force” approach, i.e. full scan of data distributed over a large number of nodes. Performance improvements can be achieved this way, but at a high price.

Scalability and performance requirements have always been critical for BI. Multidimensional OLAP techniques have been used to address performance problems, but scalability had been limited. Other popular ways to address the ad-hoc query performance problems have been in-memory and columnar databases. All of these techniques are beneficial and some work better than the others for ad-hoc OLAP queries.

The family of algorithms suggested in this paper, “Grasshopper algorithms”, is aimed at acceleration of ad-hoc OLAP queries without any additional indexing. They work for point, range and set restrictions on dimensional attributes, in any combination. The algorithms combine sequential scanning with long hops over irrelevant data; in the majority of cases they perform much faster than full scan, but practically never worse. A grasshopper can crawl or can jump quite far. This explains the name.

An OLAP cube

is a unit of logical data organization reflecting a vector functional dependency

in which independent variables are called dimensions, and dependent variables are called measures. Dimensions often have hierarchical structure induced by additional (scalar) functional dependencies on independent variables. Various aspects of OLAP have been extensively studied in the literature (see e.g. [3, 5, 10] and references in there, as well as on-line OLAP bibliography [7]).

An ad-hoc OLAP query is a query against the cube in which various filters may be placed on some of the participating variables, and measure values may have to be aggregated. Since it is not known in advance, to which variables filters will be applied, and what restrictions the filter will pose, techniques like materialized views may not be helpful.

OLAP implementations tend to use dictionaries to encode dimensional attribute values with surrogate keys. We will assume that all surrogate keys are integers. For unordered attributes we prefer these integers to be consecutive; ordered attributes with integer values do not have to be encoded, but if any naturally ordered attribute is encoded, encoding must preserve the order. We then assume all dimensions to be integer-valued. Uniformity of such encoding provides additional advantages for our techniques.

The Cartesian product of the dimensional attribute domains forms the composite key space. The vector dependency then maps a composite key to (a vector of) measures. Multidimensional database techniques are based on endowing the composite key space with a space filling curve, so that each element of the space corresponds to a single point on the curve, and vice versa. There are multiple ways to choose such a curve; for our purposes we choose a class of curves called “generalized z-curve” (gz-curve), following [9]. Each point on such curve is encoded with an integer that is derived from the values of the components of the composite key. Any query with point, range or set filters against the cube translates into a pattern search problem on the gz-curve. Such reduction is fairly standard and has been in use for many years in multidimensional databases, such as e.g. Oracle Essbase [8]. Precise definitions and explanations will be given in the next section.

We consider the following problems:

PROBLEM 1: Present a computer algorithm and an associated cost model, for efficiently retrieving elements matching a given pattern in the absence of additional indices.

PROBLEM 2: If data is partitioned by keys, present an appropriate parallelizable algorithm.

Grasshopper algorithms are applied after the described reduction of the OLAP problem to a pattern search problem in (composite) key-value space. They are very simple, and any database system based on a key-value store that supports a simple functional interface can take advantage of these algorithms. We have tested the algorithms with a few such stores, both standalone and distributed, memory and disk based, including in particular Apache HBase [1]. The results we have seen are consistent with the theory behind the Grasshopper algorithms. Before each query, the grasshopper takes a decision when to hop, based on the characteristics of the underlying key-value store and the query.

Particular cases of the algorithms - for certain encodings - translate into well known techniques. For example, if fact data is ordered by dimensional attributes, query processing is straightforward when leading attributes are filtered. Other cases, related to z-curves, have been considered in [2, 6, 9]. However, to the best of our knowledge, Grasshopper algorithms have not been presented in the literature or implemented in relevant products.

The paper is organized as follows. Section 2 contains the necessary formalizations and reduction procedures. Related work is discussed. In section 3 we describe Grasshopper strategy and provide a cost model. Then we provide Grasshopper algorithms for queries with point, range and set filters. In section 4 we present testing results and discuss them.

2 Background and Related Work

We provide the necessary background describing transformation of the OLAP fact data to key-value form (section 2.1), discuss related work in section 2.2 and reformulate the problem as a pattern search problem in section 2.3.

2.1 Reduction to key-value form

Reduction from general cube schema to integer encoded dimensions is fairly common in the OLAP world. For the facts, usage of composite keys is more typical in multidimensional OLAP (MOLAP). We outline the corresponding setup, in order to set the context for further exposition. A similar setup is described in detail in Volker Markl’s thesis [6]. Our algorithms do not depend on particular ways of encoding dimensions.

In OLAP field, the main subject of study is a vector functional dependency . Here independent variables are called dimensions (dimensional attributes) and dependent variables are called measures. This dependency is augmented with additional, typically scalar, functional dependencies involving dimensional attributes, e.g. . Dependent attributes are called higher level dimensional attributes. They induce groupings within domains of the attributes they depend on and, by virtue of that - aggregation operations on measures. Altogether dependencies are assumed to form a DAG.

In the assumed setup, all the dimensional attributes are encoded with integers. If the attribute is integer valued, it can be used without additional encoding. Otherwise an encoding dictionary is created. For attributes which are naturally ordered, encoding is required to preserve the order. Dense encoding (by consecutive integers) is preferred in most cases. How exactly encoding is organized is beyond the subject of this paper. For simplicity, we assume the cardinality of each dimensional attribute to be a power of 2.

All dimensional dependencies are then expressed in encoded terms, often simply as arrays of integers, sometimes as graphs. It is supposed that both dictionaries and dependencies provide constant lookup time.

All the dimensional attributes that are of interest for analysis can participate in the formation of a composite key. Including higher level attributes in the key adds sparsity to the model, but often eliminates the need for joins at query time.

Encoding with composite key transforms the data for functional dependency into key-value format. A storage component is responsible for maintaining data in key-value format and retrieving relevant key-value pairs at query time.

A simple query against the cube restricts certain attributes to a subset of their values and asks for data satisfying the restrictions. We study a class of filters on dimensional attributes, more precisely, point, range and set restrictions on the attributes’ domains.

When such query arrives, the system looks up the attribute values involved in the restriction against the dictionary and translates them into integers. These integers are then used to form restrictions on the composite key; those are passed to the storage component for retrieving relevant key-value pairs, which are aggregated, sorted, etc. as required by the query. Finally, dictionaries are used again to translate final results back to original attribute domains.

The above setup is common in many OLAP systems. Our algorithms fit in this picture as a vehicle for storage component to retrieve relevant key-value pairs quickly.

Let us explore composite key composition more closely. No matter how the composite key is produced, it provides integer encoding of all potential points in the key search space which is a Cartesian product of encoded attribute domains. Thus it provides (integer) parameterization for a space filling curve in the search space. We restrict our attention here to a specific class of space filling curves, generalized z-curves (gz-curves for brevity) in the sense of [9]. See also [6] for extensive exposition on the topic. Basically, the integer composite key in binary representation is built from bits of participating components’ keys in a way that the order of bits of each component is preserved. This procedure produces keys of fixed length which is convenient for our purposes.

Figure 1: Examples of gz-curves with bit orderings yxyxyx, yyyxxx, yyxyxx and xxyyyx respectively. Shaded areas are examples of fundamental regions (of order 4), to be described in section 3.3.

The shape of the gz-curve depends on the way component’s bits are shuffled into the composite key. Figure 1 shows a few possible shapes for two variables, where bits for horizontal and vertical dimensions are marked with x and y respectively. The first example is the classical “isotropic” z-curve, and the second one is known as “odometer curve” and corresponds to sorting keys by y, then by x. The latter case strongly favors one dimension over the other. It is very well known that answers to queries with filters on the leading dimension(s) of the odometer are located within a single contiguous segment of the curve, whereas for filters on the trailing dimension(s) they are scattered across the curve. Our algorithms become trivial in this case: the grasshopper either crawls all the time or hops every time, from one horizontal segment to another.

2.2 Related work

Indexing techniques based on gz-curve have been presented in [2, 6, 9] along with relevant data structures such as e.g. the UB-Tree. Besides the odometer ordering, the case of the isotropic curve has been well studied, especially for range filters on all of the dimensions (see e.g. [11, 12]). In the latter case, the bounding box of the query is often fairly densely covered by a single interval of the curve, and even full scan of that interval is very efficient. This is often exploited in geo-spatial applications. Our grasshopper typically does not have to jump in that case either.

There are certainly plenty of similarities between our algorithms and UB-Tree methods of [6] and [11]. Efficiency of query processing is the key requirement for ad-hoc queries. Ability to perform all calculations directly in terms of the composite keys, without decomposing back into dimensional coordinates, is crucial. In [6] such methods are briefly outlined for the case when all dimensions have range or point restrictions. They use the specialized UB-Tree data structure that splits keys into regions (pages). The query is executed after determining which regions cover its bounding box. The number of such regions can often be quite high (for incomplete range queries nearly all pages may qualify, for example) and determining all of them upfront can be quite lengthy, resulting in inefficiency.

Knowing which pages to retrieve is extremely important for systems with disk storage because of high I/O costs. How the search is done within the page already in memory is also important, because it allows controlling CPU costs. In the recent years, hardware has changed dramatically, and these days data can often completely fit into memory on one or more machines. As a result, for a particular storage organization, costs of certain operations can vary significantly.

Taking that into account, we have decided to take a different stance. Since we cannot change the storage in many cases, we focus on scanning algorithms rather than data structures in the hope that they will be useful for many relevant systems. As a result, our algorithms, which do not determine the cover in advance and act in a streaming fashion, seem simpler to implement. Besides, our methods apply to any combination of point, range and set filters on any subset of dimensions, not necessarily on the full set. We can therefore argue that our algorithms can improve performance of ad-hoc OLAP queries for any underlying key-value storage system that keeps data in the order of composite keys and supports certain simple operations. This is especially relevant in today’s distributed storage systems, like e.g. HBase, where full scan is often the only option to answer queries. Of course, significant performance gains can be expected when the storage system efficiently supports the required operations. We discuss this in more detail when test results are presented. Basically, the grasshopper has in possession only certain characteristics of the storage, such as the ratio of sequential access and random access costs. Given a query, it computes a certain threshold beyond which it will jump if it encounters an appropriate “obstacle” (unqualified key) while crawling. The threshold can be determined algebraically and explained geometrically.

As already mentioned, we consider two problems. The first problem is very generic and does not require much from the storage system, except the ability to jump. The grasshopper does not know much about specifics of the storage. It may jump over keys within the same unit of storage, or it may jump landing on a different unit of storage. For the second problem, the grasshopper can take advantage of the additional information provided by the storage, namely, the boundaries of the partitions by key intervals on the gz-curve. These partitions may correspond e.g. to pages of the UB-Tree or HBase regions, etc. Partitioning can be hierarchical, and is specific to storage. The grasshopper can then decide whether it needs to examine contents of the region or can skip it. In the case of the UB-Tree, this is essentially what the algorithms in [6, 11] are doing. Our techniques apply actually to a more general class of factorizable partitions, to be defined later. Each partition can be processed in parallel, as is the case with HBase regions. Moreover, within a partition, we can typically reduce the dimensionality of the problem and operate directly on the reduced (factorized) keys, without the need to restore the original keys which is a costly operation. More on this in the upcoming sections.

2.3 Pattern Search Problems

In this section we describe our problems as pattern search problems on the gz-curve.

Any point restriction on one or more of the attributes means fixing a pattern of bits in the key, so the query problem translates into a fixed pattern search problem (PSP) on a set of keys. Similarly, range and set restrictions result in patterns, albeit more complex. Definitions provided below are aimed at expressing everything in the pattern search related terminology.

Let be the total number of bits in the composite key. Consider the space of all keys as , an -dimensional linear space over the group of residues . The bits form an ordered basis in , and elements of are ordered lexicographically by coefficients (which trivially coincides with the order of integers).

Define mask to be an operator of projection onto a -dimensional coordinate linear subspace of . Given basis vectors , such operator simply masks out the remaining coordinates. Denote by the subspace onto which the mask projects.

Two or more masks are called disjoint, if the subspaces onto which they project are pairwise disjoint, i.e. do not have any common basis elements.

To emphasize similarity with bit masking, we will use the notation for . We will also use notation instead of the sum of vectors and belonging to two disjoint subspaces, and similarly, for masks.

For the case of gz-curves, to each dimensional attribute corresponds a mask that defines its bit positions in the composite key. Applying the mask to the composite key retrieves the contributing value of . Obviously, masks corresponding to different dimensional attributes are disjoint.

Let be a subset of representing composite keys of the cube fact data. Any query against the cube with point filter on attribute translates into a pattern search problem (PSP): find all such that . Queries with point filters on multiple attributes , translate into a similar problem of finding solutions to , for the union of attribute masks and union of corresponding patterns.

It is convenient to consider any mask on as corresponding to some “virtual” attribute. Thus a query with multiple point filters is equivalent to a query with a single point filter on an appropriate virtual attribute.

A query with range filter in this setting also translates into a PSP: find all such that . However, unlike the point case, combining two or more such queries into a single similarly expressed query against some virtual attribute is generally impossible. So we will have to look for elements that simultaneously satisfy some number of patterns.

Set queries are transformed in a similar fashion. Given a set , the filter corresponds to the PSP . Set filters for multiple attributes can be combined into a similarly expressed query against some virtual attribute, but since the resulting restriction set is a Cartesian product of coordinate restrictions, its cardinality may be too large for practical purposes, so multi-pattern search is used for them as well. Obviously, any solution to the search condition also satisfies a range restriction .

To summarize, we consider pattern search problems (PSP) with restrictions of the following kinds: (P), (R), (S).

A brute force solution to the pattern search problem on a set is achieved via checking pattern restrictions on each element of (full scan). A solution to the pattern search problem will be called efficient

, if on average it is faster than the brute force solution and is never slower than that. To comply with ad-hoc query requirements, the average here is taken with respect to a set of random pattern restrictions on any fixed combination of the appropriate number of attribute restrictions and then over all such combinations. According to this definition, an efficient algorithm is allowed to lose to the full scan on some pattern, but is not allowed to lose on average. We provide cost estimates explaining why our algorithms are efficient and confirm this by experiments.

A set is called factorizable if it can be represented as a Cartesian product of at least two subsets of . Besides itself, the set of all its elements satisfying a restriction of kind (P) is clearly factorizable. This is also true for restrictions of kind (R) and (S), when .Examples of factorizable subsets include intervals with common prefix or sets with common pattern.

Let be a partition of into factorizable subsets (perhaps each with its own factors). The induced partition of any by sets is also called factorizable.

Our algorithms get additional advantage when dealing with factorizable partitions, in particular with partitions by key intervals or by sets with common pattern. They become especially efficient when the underlying storage implements prefix or common pattern compression.

The problems Grasshopper algorithms are intended for can now be more precisely formulated.

PROBLEM 1: Let be an arbitrary collection of disjoint masks on space . Let, for each of these masks, a pattern restriction of kind (P), (R) or (S) be given. Further, let be an arbitrary subset of . Find an efficient solution to pattern search problem on .

PROBLEM 2: In the same setting, when is partitioned in a factorizable manner, present an efficient parallelizable algorithm.

3 Analysis and Algorithms

In this section we first present the Grasshopper strategy for solving our Problem 1. In section 3.2 we develop some notation and definitions needed for its implementation. Then, for each kind of pattern restrictions, we introduce the algorithm that helps to implement this strategy. Grasshopper algorithms use geometric properties of PSP solution loci on the gz-curve, so we outline them before describing the algorithms in each case. Geometric properties for point PSP are presented in section 3.3. Point matchers are introduced in section 3.4. We then explain in section 3.5, how to handle our Problem 2 for point restrictions. In sections 3.6 and 3.7 we introduce range and set matchers respectively. In section 3.8 we outline how queries with multiple restrictions of various kinds are handled.

3.1 Grasshopper strategy

Here we present Grasshopper strategy for scanning data in search of particular patterns. It is designed to avoid doing a full scan of the data by intelligently combining sequential crawling with jumps over large portions of irrelevant keys

Desire to have algorithms applicable to any underlying data structure led us to split powers between the devices used in the pattern search: the data store and the pattern matcher. We will formulate our algorithms using these concepts. The data store is typically not in our control; pattern matchers together with the Grasshopper strategy make up the Grasshopper algorithms.

A key-value store is assumed to contain key-value pairs whose keys are elements of and to be aware of the key ordering. In terms of its capabilities, a data store will be called basic if it supports the following operations:


given a key , retrieve the appropriate value.


given a key , retrieve the next key in .


given a key , retrieve the next key in larger than or equal to .

We also suppose that statistics of (such as cardinality, first and last key) are available at negligible cost.

A partitioned data store is supposed to be able to provide partitioning criteria and to possess the above basic capabilities for each element of the partition.

Besides the data store, the other player in the game is a matcher. The matcher is designed to assist with pattern search assuming the following functionality:


given , tell whether satisfies the given pattern restrictions.


given , return 0 if satisfies the given pattern restrictions or the (signed) position of the highest bit in responsible for mismatch, with sign indicating whether mismatch is from above or from below.


given an element with its mismatch , suggest the next element , that can theoretically satisfy the pattern restrictions.

The roles are quite distinct: the store knows everything about the set , but nothing about the masks and patterns; for the matcher it is the other way round. All the algorithms follow the same pattern, but matcher variations for different cases of pattern problems are quite different.

A race over the data store is announced. Participants must collect data matching a given set of patterns, into a bag. At the start of the race, there is a crawler, a frog and a grasshopper. Each one is given a matcher. Using the matcher, they can compute the theoretical query bounding interval on and intersect it with the interval to obtain the actual bounding interval .

The crawler’s strategy is sequential scan:

bag = ;


if , add to bag;


The frog’s strategy is jumping as soon as possible:

bag = ;


y =

if , add to bag, ;

else ;


The grasshopper’s strategy is to jump only when the absolute value of the mismatch is above a certain threshold ; otherwise crawl:

bag = ;


y =

if , add to bag, ;

else if ;

else ;


When the Hint operation has nothing to suggest, it returns and the corresponding loop terminates.

Some modifications on the grasshopper strategy will be outlined later on.

The following simple cost model can be suggested. First, observe that all three racers are doing the same number of and operations for those that do match the PSP restrictions. We can exclude these from the cost estimates. The only difference for matching elements may be in the cost of and operations.

For those elements, that do not solve the PSP, the crawler performs and operations, the frog performs , and operations, and the grasshopper sometimes does the same as the crawler and sometimes - the same as the frog.

Let us assume that matcher’s operations take negligible time compared to the data store’s operations (this is often true in reality). So the essential costs to compare are:







Here is the number of mismatched elements, is the number of frog’s jumps, is the number of grasshopper’s jumps, and is the number of times grasshopper continues to crawl. These values are functions of and .

Recall that, by our definition of efficiency, we need to compare average costs. Fix the PSP mask , and let be the sum of the corresponding over all possible patterns. Let be the corresponding PSP solution.

Let . is a property of the data store that can be experimentally determined (it may depend on the data set). When data is physically stored sorted, usually , but if navigation is via an index, it may happen that . For in-memory data store, is typically closer to 1 than for disk-based one.

It is clear that the frog will finish (on average) ahead of the crawler if .

The term is the sum of cardinalities of , and since form a disjoint cover of , we obtain .

So the frog really hopes that


Obviously, the opposite inequality is in favor of the crawler.

The right hand side of (1) does not depend on the geometry of the mask(s), i.e. on the way the attributes participate in the key composition. By contrast, is heavily dependent on the mask.

If the grasshopper determines ahead of the race that the frog is guaranteed to win over the crawler, it can solidarize with the frog by deciding to use the threshold value . However there are cases when the frog definitely loses to the crawler. For example, if the mask consists only of the first (most junior) bit, then theoretically every second point of solves the PSP. The matcher cannot propose anything better than jumping exactly to the next point, definitely a losing strategy.

Well, in the worst case the grasshopper can solidarize with the crawler by deciding to use the threshold value prohibiting any jumps. But the grasshopper cannot always solidarize with the crawler, because then its strategy will not meet the “efficiency” criteria. And the grasshopper must make its decision before the race! So it must be able to at least verify if (1) holds.

In order to help the grasshopper to make the right decision, we need to examine the problem more closely.

3.2 Masks and Patterns

In this section we develop additional notation and terminology in order to formulate our solutions.

For any mask , there exists a complementary mask (co-mask) , that projects onto the remaining coordinates. Certainly, any -dimensional vector can be restored from its projections onto and , i.e. .

It is clear that co-mask definition can be extended to complement a set of masks (just pick a subspace orthogonal to the span of ).

We say that mask is covered by masks if both and jointly have the same complement . A cover is a partition if spaces do not have common basis vectors.

Let mask be projecting onto bits in , listed in ascending order. Define and . Mask is called contiguous if it projects onto adjacent bits, or, equivalently, .

For a mask and some element of the basis of , denote by , and projections of onto basis vectors , onto and onto basis vectors respectively. Thus , with similar relationships for the projection spaces. One or more of the projections may be empty. Similarly, any pattern can be decomposed as .

An element of a subspace all of whose coordinates are 0 (1) is denoted (), and we will write instead of .

Define a partial order on the set of masks as . Among partitions of mask by contiguous masks, canonical partition of is the one with the smallest number of parts. We will always enumerate them in descending order, from senior bits to junior ones.

The smallest element matching a fixed pattern in is of the form and the largest such element - is . These elements form the bounding interval for the fixed pattern search problem. Although they depend on , their difference, , does not:

For any range or set pattern restriction, with minimal element and maximal element , we have and . For contiguous masks, the spread depends only on the difference , however this is not true in the general case.

3.3 Geometrical properties of the point PSP locus on gz-curve

In this section we obtain certain geometric properties of the point PSP locus on the gz-curve.

Recall that the gz-curve is used as a space filling curve for the Cartesian product of attribute domains of integers. The cardinality of each domain is a power of 2, and the way it participates in forming the element of the gz-curve is expressed by the domain mask . In the previous section, the space filling curve was algebraically expressed as an -dimensional space, , with being the total number of bits of all domains. How do and relate to each other geometrically?

As the gz-curve traverses the space , one can identify fundamental regions of order for as the rectangular boxes with volume corresponding to intervals of the gz-curve (each region is a single point, and ). The ends of the intervals are aligned with the corresponding power of 2. See figure 1 for an example. Each fundamental region contains fundamental regions of lower orders. All the regions of given order are replicas of each other, the shape of the gz-curve in them is the same and their number in is . When no confusion arises, corresponding intervals on the gz-curve will also be called fundamental.

Solution locus of any PSP on a gz-curve consists of certain intervals (clusters), which in some cases degenerate to a point. We will use the term lacuna for the gap between the clusters (excluding gaps at the ends of the curve).

We would need to compute certain quantities characterizing the locus of any point PSP: cluster count, cluster lengths, total lacunae length, and individual lacunae lengths.

Proposition 1

Let be an arbitrary mask projecting onto dimensions, and let be its canonical partition. Let be a point PSP.

Then the locus of the PSP consists of intervals of length each, separated by lacunae of total length . The spread can be calculated as where . Individual lacunae lengths are partial sums

Figure 2: Structure of the locus for point PSP.

We briefly outline the proof which follows by induction on the number of the elements in the canonical partition of the mask. For a contiguous mask with bits, it is easy to see that only one out of adjacent intervals of size within any fundamental region qualifies for the given fixed pattern restriction with mask and is also a fundamental region . There are such regions. Thus the locus of points on the gz-curve satisfying the restriction consists of clusters of length separated by lacunae of length . See figure 2 for an illustration.

The step of the induction follows by repeating the base argument within each of the fundamental regions (previously identified matching cluster) considered instead of for the next component of the mask. This time however we need to take into account the gaps at the edges of the regions. These additional gaps have the length which yields (2).

Note. It is easy to see that partial sums in (2) have the following bounds:


3.4 Grasshopper algorithm for point queries

In this section we describe Grasshopper algorithm for the point matcher.

Let be some mask projecting onto dimensions, and let be an element of . For a subset , consider the fixed pattern search problem , that is, finding all elements such that . Any mask partition also induces pattern partition . An element of matches on if and only if it matches on for each .

Here is how matcher operations for this PSP are defined. For the Mismatch operation, the matcher works by examining , , one at a time. If , let be the most senior bit, on which the sides disagree. The matcher returns if and if . If , the matcher proceeds on to , and so on. If no mismatch is detected, the matcher returns 0.

Let be the identity mask on , i.e. the mask projecting onto entire .

For the Hint operation, given an element and mismatch position , the matcher acts as follows.

If mismatch is negative and indicates mismatch at bit , the matcher returns . The highest bit position that changes is . Geometrically this means that the point belongs to some fundamental region , and by changing the bit, we are placing the result into the next such region. Since none of the bits above is changed, we are staying within the same fundamental region that contained .

If the mismatch is positive, geometric meaning of the operation is similar, but the next fundamental region

intersecting with the PSP locus, is located in a different fundamental region of higher order than the one containing . In order to find such region, we need to determine the “growth point” which is the smallest position above of an unset (0) bit in . If such position does not exist, the search is over ( returned), otherwise the value is returned.

We need to estimate from above the total number of times when the frog jumps, . Note that the jump occurs only when a mismatch is detected, i.e. when belongs to some lacuna. After the jump, the frog lands on the next cluster. Clearly the number of jumps cannot exceed the number of lacunae, which, by proposition 1, is and does not depend on . Hence

If the right hand side is less than the right hand side of (1), the frog finishes (on average) ahead of the crawler. Such estimate obviously holds for some masks, for example, for contiguous headless masks, since for those and there are no lacunae. Rewrite the condition (1) as


The above estimate is instrumental in the dense case, when the cardinality of is comparable to the number of lacunae. In the sparse

case, when we have much less points than lacunae, a different estimate is needed. Consider the probability distribution

of points of with respect to fundamental regions and let . The average number of points per region is For a fixed pattern, each region may participate in at most one lacuna.

For a given mask , define region co-frequency as the number of times the -th region participates in lacunae across all patterns, . The co-frequencies are easy to determine given the mask. For example, for contiguous masks, co-frequencies are . For non-contiguous masks, co-frequencies consist of several series of similar kind with various exponents up to d.

The total number of points in all lacunae across all patterns is . Since condition (1) can be rewritten as



Observe that

We have arrived at the following conclusion:

Proposition 2

Let be a mask projecting onto dimensions (bits) of , and let be a nonempty subset of . Let be fundamental regions with co-frequencies , and let be the distribution of with respect to those fundamental regions. Let and be defined by (4) and (5) respectively.

If the scan to seek ratio of the data store satisfies the estimate

then the frog strategy is more efficient than the crawler strategy.

The condition would be meaningful if The value of might not be straightforward to compute, let us examine particular situations when the estimates can be simplified.

First, examine the case when the distribution is uniform, i.e. Note that is proportional (with coefficient ) to the length of all lacunae for all patterns. By proposition 1, the length of lacunae for each pattern is

Hence, over all patterns,

Observe that the minimum value of is achieved when is a contiguous tailless mask projecting onto most junior bits. Thus , and

It follows that


Clearly, the same would be true when distribution is sufficiently close to uniform. Observe that, for a given

The first term can be estimated as


is the variance of

, it follows that the number of terms in the sum for which cannot exceed Hence



First choose an that would minimize the value of the right hand side. The right hand side has the form with and . Taking derivative to find the minimum, one obtains or which yields the minimum value of

Since , in order for the above expression to be less than 1, we need to have


Substitute the expressions for and to see that the last inequality is equivalent to


The right hand side of (6) is


Sufficient conditions are summarized in the following

Proposition 3

Let be a mask projecting onto dimensions (bits) of , and let be a nonempty subset of . Let

the standard deviation of the distribution of

with respect to fundamental regions satisfy (7). Let quantities and be defined by (4) and (8) respectively.

If the scan to seek ratio of the data store satisfies the estimate

then the frog strategy is more efficient than the crawler strategy.

The grasshopper is clever enough to verify the conditions of Proposition 2 (or 3) and, if they hold, set its strategic threshold to 0. The grasshopper will then arrive with the frog, ahead of the crawler.

Proposition 3 can be interpreted in the following manner: if the mask has sufficiently high tail, the dense case condition kicks in, and the relatively small number of lacunae awards victory to the frog and grasshopper; if the mask has low tail, the sparse case condition must be checked, and the victory is guaranteed when the distribution over is close to uniform. For the frog, the chance of winning is limited to these cases. For the grasshopper, however, there are still opportunities to win over both.

The idea is, even when the mask tail is low, but there are contiguous components with sufficiently high tails, setting the right strategy threshold would allow the grasshopper to win over the crawler because it jumps over large lacunae and to win over the frog because it scans small lacunae. Let us examine the situation in more detail.

Grasshopper jumps when the matcher detects mismatch exceeding the threshold. Mismatch value for a non-qualified point carries information about the size of its lacuna, and the threshold is designed to ensure that only large lacunae trigger the jumps. Let be the number of bits in the mask above . For a given pattern, PSP locus is contained in only of fundamental regions (crawl regions). The regions between crawl regions are jump regions. Geometric interpretation of the grasshopper strategy is to crawl when the current point is within a crawl region and to jump if it belongs to a jump region. Adjacent jump regions form lacunae, and grasshopper jumps over the entire lacuna in one hop.

First, consider competition against the crawler. Within smaller size lacunae both strategies perform scans, so crawl regions have no influence on the competition. Recall that lacuna sizes correspond to contiguous components of the mask. Removing components below the threshold would only affect the PSP locus in the crawl regions, hence trailing components can be safely dropped from consideration without affecting the outcome of the competition between the crawler and the grasshopper. So assume that all components of the mask lie at or above . Under this assumption, , and both frog and grasshopper strategies coincide.

The total number of all jump lacunae for all patterns is , and . Rewrite the estimate (1) as


The value can thus serve as threshold. This argument is similar to the one used to define .

To summarize, we have arrived at the following result:

Proposition 4

Let be a mask projecting onto dimensions (bits) of with canonical partition and let be a nonempty subset of . Let, further, be the scan-to-seek ratio of the data store, and let

Then grasshopper strategy with threshold is more efficient than the crawler strategy.

Another way to arrive at a similar threshold value is to observe that lacunae must be sufficiently long to contain a large enough number of elements of . The grasshopper will skip over of them as soon as it stumbles upon the first one. The crawler, however, would have to visit them all. Hence the difference between crawler and grasshopper strategy costs would amount to comparing and . Grasshopper wins when .

Let be the maximal value of all such for which the partial sum in (2) exceeds . If such value exists, in view of bounds (3),

so lacunae would be large enough (to contain points on average) if Thus serve as an alternative definition of the threshold. If there is no such the threshold is set to .

Assuming that the mask has components below the threshold, consider competition between the grasshopper and the frog. The difference is in the behavior below the threshold where lacunae correspond to lower contiguous components of the mask, and it amounts to comparing and . Here is the number of jumps above the threshold. We can drop mask components above the threshold without affecting competition outcome and assume that and So essentially we are again comparing frog and crawler behavior, however this time the condition ensuring frog’s victory is not satisfied and the crawler (and hence grasshopper) has advantage.

Overall, situation can become bad for the frog when the following takes place: the tail of the mask is low so that the sparse case kicks in; within lacunae there is at most one non-empty jump region. Then the frog would be forced to jump a number of times comparable to which loses to sequential scan unless For the grasshopper, however, this is not critical if there are higher mask components corresponding to the dense case.

Note. As we have seen, if the matcher returns a negative mismatch value for some , it means that belongs to a lacuna within some fundamental region and the next cluster of the PSP locus is located in the same fundamental region. If the mismatch is positive, it means that is in a larger lacuna located between two fundamental regions of order higher than . This indicates that, in principle, the grasshopper could have operated two different thresholds and jumped more often upon encountering positive mismatch.

Another possible variation of the algorithm is to try to enhance the scanning portion of it, by determining, upon seeing an element that qualifies, the end point of the cluster, in the PSP locus, to which it belongs, and then blindly picking the elements encountered before that end point. This means that instead of verifying the match, we will be verifying the inequality. These two operations have roughly the same cost. Besides, upon encountering an element that does not satisfy the inequality, we would still have to check if it matches the pattern. Calculating the end of the cluster is easy, but also bears additional cost. So upon first glance, this variation does not bring any benefit. However, its efficiency really depends on how the appropriate storage interface is implemented. In many data stores, extra comparisons are made anyway. But some of the data store’s costs may be avoided, if it is partitioned - the search simply stops at the end of the partition, and we may not even have to check that in the loop.

3.5 Partitioned case

In this section we describe grasshopper strategy modifications for Problem 2 (partitioned case).

If the data is partitioned, and it is possible to scan partitions in parallel, there is an obvious benefit to all of the strategies, because a bunch of crawlers, frogs and grasshoppers can participate in the race and fill their bags faster. However, grasshoppers have the additional benefit of coming up with a proper threshold specific to a particular part.

When a partition is factorizable, there is a common pattern that all elements possess. The case of partitioning by intervals is discussed here as it is used most often. If an interval in is factorizable, there is a common prefix pattern and a corresponding prefix mask projecting on dimensions, such that , where is an interval in an -dimensional space. Prefix compression techniques are used by some stores to keep a single copy of the prefix and only bits per key. If the store can also provide access to truncated keys (dimensionality reduction), efficiency increases. Unfortunately, unless the store is in our control, such access is often unavailable. As a result, the store performs multiple memory allocations and copies in order to assemble full-length keys which is counterproductive.

Nevertheless, computing the prefix from the boundaries of is easy, and additional reductions are possible. First, form . This is achieved through an easy mask operation. If , let be the corresponding intersection mask. If , the entire interval lies outside the PSP locus (trivial mismatch), and can be safely skipped. If , it means that , and hence the entire interval lies within the PSP locus (trivial match), so all the points in it are added to the bag without checking. Otherwise, the problem is non-trivial, but the mask in PSP can be replaced with and the pattern - with . When computing the threshold, dimensionality, , can be reduced by the dimensionality of .

3.6 Range queries

In this section we describe geometric properties and matcher implementation for range restrictions.

Before presenting appropriate geometric considerations for range queries, it makes sense to mention our reduction techniques.

We deal with pattern restrictions of kind (R): We first perform a trivial check , otherwise it is a point restriction. Next we determine if the interval is factorizable. For that we compute the maximal common prefix of and . If such common prefix exists, then , and all points within the interval have the same prefix . This induces splitting of mask into prefix and suffix masks: , and the original PSP is transformed into a system of two PSPs and . For the first problem we already know the locus structure, and locus of the original PSP is a subset of it. Hence considerations of the previous section would apply. For range specific techniques, it is then sufficient to consider the case of non-factorizable interval which is what we further assume.

Observe that, by our assumption, elements and have different senior bits, 0 and 1 respectively (otherwise they have common prefix). An interval is called complete, if all bits of are 0 and all bits of are 1. Obviously, for a complete interval, the PSP is trivial: all elements of are solutions.

An interval is called suffix-complete if it is factorizable, and its suffix interval is complete. For example, interval is suffix-complete, since , but interval is not, since . For suffix-complete intervals, via the mentioned reduction, the original range PSP is thereby converted into a point PSP.

Assume finally that the interval is incomplete and non-factorizable. It may still sweep almost the entire corresponding -dimensional subspace, and hence PSP locus may be almost the entire space . The smaller the interval, the more close the problem is to the point case, and the more chances for grasshopper strategy to find large lacunae to jump over.

Let us examine the corresponding geometry. As we have seen, the locus of the point PSP consists of intervals of equal length with gaps between them. This is not the case for range restrictions.

Proposition 5

Let be an arbitrary mask projecting onto dimensions and let be its canonical partition. Let be a range PSP, be the cardinality of , and let be the cardinality of .

Then the locus of the PSP generally consists of clusters of varying lengths, which are separated by lacunae of total length . The spread can be calculated as . Individual lacunae lengths are partial sums


Since the proof is more complex than in the point PSP case, we explain it a little bit.

First, suppose that mask is contiguous. In that case, as in the point case, within each fundamental region in , the locus of the PSP is a single interval of size , where is the length of the interval. The lacuna between the intervals is thus .

Figure 3: Structure of the locus for range PSP.

The picture becomes more complex in non-contiguous case. For simplicity, consider the case of two components. Unlike the point case, the partial PSPs for each of the masks are not independent; the second PSP depends on the state of the first problem. Consider the PSP1: . Note that if , then definitely solves the original PSP, and PSP for the second mask is not even considered. If , or , is definitely not a solution. If ( respectively), then is a solution whenever it solves the second PSP (PSP2) of the form ( respectively). Denote these PSPs as and respectively. Of course, one or both of the corresponding intervals may degenerate to a point.

Let and be the lengths of the intervals involved in and . As we have mentioned, within each fundamental region , the locus of PSP1 consists of a single interval of length , called order 1 interval. The locus of the original PSP within that fundamental region is contained in that interval, and contains the interval of length which corresponds to the inner part , if the latter is not empty. Within the interval of order 1 but outside this inner part, in every fundamental region , there are two series of order 2 intervals corresponding to PSP2(a) and PSP2(b), located to the left and to the right respectively of the inner part of the order 1 interval. One of the order 2 intervals in each of two series would be adjacent to the order 1 interval from the corresponding side, and the total number of intervals within that fundamental domain would be at most . See figure 3 for illustration.

The order 2 intervals in each series have lengths and respectively, and lacunae between these intervals are of sizes and respectively. Now, there was a lacuna between the order 1 intervals in different fundamental regions . To this gap one needs to add the gap between the beginning of that interval and the closest intervals of the order2 interval series from each such fundamental region. This gap corresponds to intervals and , and has combined length , where is the length of . Adding the gap to the length of the order 1 lacuna produces the partial sum (9).

If the mask has 3 components, the picture changes in a similar manner: each of the order 2 intervals would have its inner part belonging to the PSP locus, and within the space between order 2 interval and its inner part there would be two series of order 3 intervals, and so on.

We now describe how the matcher works for the range PSP case.

For the Mismatch operation, the matcher examines each , , one at a time. If , the matcher returns 0, indicating a match. If , let be the most senior bit, on which they disagree. The matcher returns if and if . If or , the matcher proceeds on to , where the interval is either or , and so on.

Let be the identity mask on , i.e. the mask, projecting onto entire .

For the Hint operation, given an element and mismatch position , the matcher acts as follows.

If mismatch is negative, the matcher computes a preliminary hint of the form . If preliminary hint is not within (which depends on the senior part of ), the hint is corrected as . The highest bit position that changes is .

If the mismatch is positive, the matcher determines the “growth point” which is the smallest position, above , of an unset (0) bit in . If such position does not exist, the search is over ( returned), otherwise the hint is computed as above with instead of .

The way to compute the threshold for the grasshopper strategy is similar to the point queries case, but uses Proposition 5. The threshold would be where is the maximal for which in (9) exceeds

Treatment of the partitioned case is similar to point PSPs, but for each interval in the partition we can also compute a proper range restriction. For a given interval, besides the trivial cases when the entire interval qualifies, or the entire interval does not qualify, there may be other interesting situations, when, e.g. range restriction becomes point restriction within the interval, or when restriction with incomplete range becomes a restriction with complete range, and so on.

3.7 Set queries

In this section we describe geometric properties and matcher implementation for set restrictions. We have similar reduction techniques for them.

We now deal with pattern restrictions of the kind (S): where is some set. For convenience, we assume the set to be ordered. We first check that the spread of is not equal to its cardinality, otherwise is actually a range. This also excludes single element sets. Next we determine if the set is factorizable. For that we compute the maximal common pattern of all elements of . If such common pattern exists, then , with the splitting of the mask . The original PSP is transformed into a system of two PSPs and . For the first problem we already know the locus structure, and locus of the original PSP is a subset of it. Hence considerations of the point queries section would apply. For set specific techniques, it is then sufficient to consider the case of non-factorizable set , which is what we further assume.

Since a set consists of individual points, it is clear that the locus of the set PSP is a union of loci of corresponding point PSPs. This suggests that all clusters in PSP locus have the same size, and their total number differs from the point case by a factor . As in the range case, the solution space can be quite large if the set is almost the entire space . Moreover, individual lacunae sizes differ greatly depending on the distances between the set elements. However since the set is fully contained in the range