1 Introduction
Datalog [Maier:2018:DCH:3191315.3191317] is a logic language for deductive databases [DBLP:books/aw/AbiteboulHV95], security [DBLP:conf/sp/DeTreville02], networking [DBLP:journals/cacm/LooCGGHMRRS09], semantic web [DBLP:journals/ws/CaliGL12], and many other applications [WarLiu17AppLParxiv].
Given a set of Datalog rules, facts, and a query, answers to the query can be inferred using bottomup evaluation starting with the facts or topdown evaluation starting with the query. The dominant strategies for efficient evaluations are topdown evaluation with variant tabling [DBLP:conf/iclp/TamakiS86] to memoize and reuse answers to subqueries, and bottomup evaluation with the magic set transformation (MST) [DBLP:conf/pods/BancilhonMSU86] so that evaluation of the transformed rules are driven by demand from the query.
The performance of Datalog engines is difficult to understand due to the multitude of factors involved. For example, the performance of different tabling strategies vary drastically [DBLP:conf/iclp/RaoRR96], and bottomup evaluation after MST may be much slower than the bottomup evaluation of the original rules [DBLP:conf/sigmod/SereniAM08]. Choosing the best evaluation method for a given set of rules requires precise complexity analysis of each evaluation method. Recently, demand transformation (DT), resulting in simpler rules with better space complexity than MST has been introduced, and the time and space complexities of topdown evaluation with tabling and bottomup evaluation after DT has been precisely established.
Stratified negation is a simple kind of negation with a simple and universally accepted semantics, and it is handled gracefully by topdown evaluation with tabling requiring almost no change. However, MST applied to rules with stratified negation is known to result in rules with nonstratified negation, which has many complex semantics, such as wellfounded semantics [DBLP:journals/jacm/GelderRS91] and stable model semantics [DBLP:conf/iclp/GelfondL88]. Under such semantics, precise complexity analysis is difficult, and worstcase behavior is prohibitive, as in the quadratic time complexity in the size of the grounded rules for wellfounded semantics. A more recent semantics, founded semantics [DBLP:conf/lfcs/LiuS18], can evaluate a certain class of rules with arbitrary negation in linear time, but not for all rules. Other efforts to address this problem have been made, as discussed in Section 7, but they are more complicated than even MST, and none of them provide precise complexity guarantees, or proofs that they match topdown evaluation with tabling.
This paper presents a new method consisting of (1) a simple extension to demand transformation, a transformation to make rules demanddriven for Datalog without negation, to support stratified negation, and (2) a simple extension to an optimal bottomup evaluation method for Datalog with stratified negation, to handle nonstratified negation in the resulting rules. We show that the method provides precise complexity guarantees, which is worstcase linear time in the size of the grounded rules, in contrast to existing quadratic time results. The method only infers facts needed for topdown evaluation of the query, and each firing of a rule to infer such a fact takes worstcase constant time. We extend the precise relationship between topdown evaluation and demanddriven bottomup evaluation to Datalog with stratified negation. Finally, we show experimental results for performance, as well as applications to previously challenging examples.
2 Datalog with negation
Datalog [Maier:2018:DCH:3191315.3191317] is a language for defining rules, facts, and queries, where rules can be used with facts to answer queries.
A Datalog rule is of the form:
where is a finite natural number, each (respectively ) is a predicate of finite number (respectively ) arguments, each and is either a constant or a variable, and each variable in the arguments of must also be in the arguments of some .
If , then each must be a constant, in which case is called a fact.
For the rest of the paper, rule refers only to the case where , in which case each is called a hypothesis, also known as premise, and is called the conclusion.
The meaning of a set of rules and facts is the set of facts that are given or can be inferred using the rules.
A query is of the form where each argument is either a constant or a variable. Answers to a query is the set of facts, given or inferred, that match the query, where a fact matches a query if there is a substitution of the variables in to constants such that under the substitution is the same as .
We call a predicate that appears in the conclusion of a rule an intensional predicate, and a predicate that does not appear in the conclusion of any rule but appears only in given facts an extensional predicate. We assume without loss of generality that intensional and extensional predicates are distinct.
In examples, we use letters for variables in arguments of predicates, and we use literal numbers for constants.
Example.
We use the following rules for transitive closure as a running example.
p(x,y) \(\leftarrow\) e(x,y). (R1) p(x,z) \(\leftarrow\) e(x,y), p(y,z). (R2)
We consider the query p(1,x)?. It asks for all nodes reachable from 1 following the edges.
Stratified negation.
Datalog can be extended with stratified negation by extending rules to be of the form, where [not] indicates that not is optional:
and imposing the constraint that there is no cyclic dependency between any predicate and a negated predicate, i.e., a predicate preceded with a not. For example, replacing (R2) in the running example with the following rule, where q is an extensional predicate, results in Datalog with stratified negation:
p(x,z) \(\leftarrow\) e(x,y), p(y,z), not q(x,z). (R2’S)
because there is no cyclic dependency between p and q.
Formally, we say that predicate (negatively) depends on predicate if there is a rule whose conclusion’s predicate is and there is a (negative) hypothesis in the rule whose predicate is . A set of rules is stratified if for any two predicates and , if negatively depends on , then does not transitively depend on . Predicates in a stratified set of rules can be split into numbered strata such that for any two strata and , if a predicate in negatively depends on a predicate in , then has a higher number than .
Datalog with stratified negation has a simple and universally accepted semantics, where rules having predicates in a lower strata are evaluated first, and negated hypotheses under a substitution is considered false if the corresponding facts have not been inferred. This semantics coincides with all wellknown semantics for negation, including perfect model semantics [DBLP:journals/jar/Przymusinski89], wellfounded semantics [DBLP:journals/jacm/GelderRS91], and stable model semantics [DBLP:conf/iclp/GelfondL88].
Nonstratified negation.
Removing the constraint that there is no cyclic dependency between any predicate and a negated predicate may result in nonstratified negation, which has no universally accepted semantics. Consider the following rule:
t(x) \(\leftarrow\) not t(x).
For a constant c, if t(c) is false, then it is also true by this rule, therefore standard semantics does not apply. Many sophisticated semantics have been developed, including wellfounded semantics [DBLP:journals/jacm/GelderRS91] and stable model semantics [DBLP:conf/iclp/GelfondL88]. Also, under sophisticated semantics, precise complexity analysis is difficult, and worstcase behavior is prohibitive, e.g., quadratic time in the size of the grounded rules for wellfounded semantics [berman1995computing]. Therefore, rules with nonstratified negation must be avoided whenever possible.
3 Demanddriven evaluation and challenge of stratified negation
This section describes topdown and bottomup methods for answering queries, and challenge of stratified negation.
Topdown evaluation with tabling.
To answer a query, topdown evaluation starts with the query, generates subqueries from hypotheses of rules whose conclusions match the query, considering rules in the order given, and considering hypotheses from left to right, and does so repeatedly until the subqueries match given facts.
Straightforward topdown evaluation may lead to repeated subqueries, or even infinite recursion for recursive rules such as (R2) in the running example. To address this problem, tabling memoizes answers to queries encountered, and reuses the answers when a query is encountered again.
In this paper, we consider topdown evaluation using the dominant tabling strategy, variant tabling without early completion, which exploits all ways to infer the answers to a query modulo variable renaming [DBLP:journals/jacm/ChenW96]. We refer to this evaluation as Topdown in the rest of the paper.
Stratified negation is generally supported for queries that are nonfloundering [DBLP:journals/jacm/ChenW96]. A query is nonfloundering with respect to a set of rules if during Topdown of the query, all subqueries for negated hypotheses have all of their arguments bound.
In this paper, we only consider nonfloundering queries of Datalog with stratified negation, because these queries can be evaluated by Topdown with a trivial extension to test the negation.
Optimal bottomup evaluation and demanddriven transformations.
Bottomup evaluation starts with given facts, infers new facts from conclusions of rules whose hypotheses match existing facts, and does so repeatedly until no more facts can be inferred.
An optimal method [DBLP:journals/toplas/LiuS09] transforms any given set of Datalog rules into an efficient specialized procedural implementation with guaranteed worstcase time and space complexities, and computes the complexities from the rules.
In particular, rules are first transformed to remove singleton variables (variables that occur in only one hypothesis) and multiple occurrences of the same variable in a single hypothesis. Next, rules with more than two hypotheses are decomposed into rules of two hypotheses. Then, the least fixedpoint specification of the rules is transformed to a whileloop that considers given and inferred facts incrementally, one at a time, where auxiliary indices are used to find each matching fact in constant time.
The evaluation is optimal in that only combinations of facts that make the hypotheses of a rule simultaneously true are considered, and each such combination, which leads to a firing of the rule, is considered once in constant time. The time complexity is precisely the sum of the number of firings over all rules plus the size of extensional predicates for reading given facts.
In this paper, we only consider the decomposition of rules that takes the two leftmost hypotheses into a new rule at a time. We call this leftoptimal bottomup evaluation, because the time complexity of evaluation using this decomposition is optimal for the lefttoright ordering of the hypotheses in a rule. We refer to this evaluation as Bottomup in the rest of the paper.
Note that the optimal bottomup evaluation in [DBLP:journals/toplas/LiuS09] is not limited to the particular decomposition used by Bottomup. It can perform time and space complexity calculation for all possible decompositions, and select an optimal one trading space for time, including space for intermediate predicates from the decompositions. We consider Bottomup in this paper in order to establish correspondence with the lefttoright evaluation of hypotheses in Topdown.
Bottomup infers all facts possible from the given facts, and thus may infer many facts not needed for answering the query. For efficient evaluation, the query can be used to limit the facts that can be inferred. This is achieved by transforming the rules to be restricted by the query, like in the wellknown magic set transformation (MST) [DBLP:conf/pods/BancilhonMSU86].
Demand transformation (DT) [DBLP:conf/ppdp/TekleL10] is such a transformation that results in rules with the same time complexity and exponentially better space complexity in program size than MST. It transforms a set of rules and a query into a new set of rules and a fact, which can infer only facts that can be inferred during Topdown of the original rules. DT and MST make use of predicate annotations for argument binding patterns. DT differs from MST in that given intensional predicates are not annotated in transformed rules, therefore is much simpler, and much less space is used because a fact of a predicate is stored only once, rather than once for each possible annotation of the predicate.
DT has two stages: (I) compute demand patterns for intensional predicates, that is, a set of pairs indicating that under Topdown for the given query, there is a subquery for predicate p with pattern s, a string of characters for arguments of the subquery, where the th character is b if the th argument is a constant (i.e., bound), and f if it is a variable (i.e., free), and (II) transform given rules and query using the demand patterns computed, in three steps, as follows.

For each demand pattern , and each given rule p(args) h,…,h., generate following rule
p(args) d_p_s(a,…,a), h,…,h.
where args denotes arguments in the conclusion, a,…,a are arguments in args that are bound by s, i.e., that correspond to character b in s.

For the given query of the form p(args)?, generate the following fact:
d_p_s(a,…,a).
where s is the pattern for , and a,…,a are the constant arguments in args.

For each rule generated in Step 1, c h,…,h., and each h whose predicate is an intensional predicate p, generate the following rule:
d_p_s(a,…,a) h,…,h.
where s is the pattern of h, and a,…,a are the bound arguments of h.
Example.
For the running example, the only demand pattern is , because the only type of query encountered during Topdown is one where the first argument of p is a constant. DT yields the following rules and fact:
p(x,y) \(\leftarrow\) d_p_bf(x), e(x,y). (R1’) (DT Step 1) p(x,z) \(\leftarrow\) d_p_bf(x), e(x,y), p(y,z). (R2’) (DT Step 1) d_p_bf(1). (DF) (DT Step 2) d_p_bf(y) \(\leftarrow\) d_p_bf(x), e(x,y). (DR) (DT Step 3)
(R1’) and (R2’) restrict the original (R1) and (R2) using a new demand predicate d_p_bf indicating demand on the first of two arguments of predicate p. (DF) is a new fact that corresponds to demand by the given query where ther first of the two arguments of p is 1. (DR) propagates demand in (R2’) from d_p_bf(x) via e(x,y) to demand on the first argument y in p(y,z).
Challenge of stratified negation for MST and DT.
Not only can Topdown handle nonfloundering queries of stratified Datalog with a trivial extension [DBLP:journals/jacm/ChenW96], which checks whether a negative hypothesis is true under the current substitution; but also can Bottomup be trivially extended and give precise time and space complexity guarantees as for without negation [DBLP:journals/toplas/LiuS09], by checking whether a fully instantiated negative hypothesis is true in time.
It is therefore natural to think that MST and DT would apply to stratified negation as well, adding demands that mimic Topdown, so that Bottomup of the resulting rules has the same optimal complexity as for without negation. Unfortunately, this is not true—applying MST to stratified rules, treating demand for a negated hypotheses as demand for the hypothesis without the negation, may result in nonstratified rules [DBLP:journals/jlp/BalbinPRM91]. We show that DT has the same problem as MST through the following example.
Example.
Consider extending the running example with two additional rules:
p2(x,y) \(\leftarrow\) not p(x,y), e2(x,y). (R3) p2(x,z) \(\leftarrow\) not p(x,z), e2(x,y), p2(y,z). (R4)
p2 is the transitive closure of e2 whose computation does not use any pair in p. The four rules are stratified because only p2 depends negatively on p, and p does not transitively depend on p2.
Consider the query p2(1,2)?. The query is nonfloundering because the only demand pattern for the negated hypotheses is . DT yields the following rules, treating demand for a negated hypothesis as demand for the hypothesis without the negation:
p(x,y) \(\leftarrow\) d_p_bb(x,y), e(x,y). (R1’) (DT Step 1) p(x,z) \(\leftarrow\) d_p_bb(x,z), e(x,y), p(y,z). (R2’) (DT Step 1) p2(x,y) \(\leftarrow\) d_p2_bb(x,y), not p(x,y), e2(x,y). (R3’) (DT Step 1) p2(x,z) \(\leftarrow\) d_p2_bb(x,z), not p(x,z), e2(x,y), p2(y,z). (R4’) (DT Step 1) d_p2_bb(1,2). (DF) (DT Step 2) d_p_bb(y,z) \(\leftarrow\) d_p_bb(x,z), e(x,y). (D1) (DT Step 3) d_p_bb(x,y) \(\leftarrow\) d_p2_bb(x,y). (D2) (DT Step 3) d_p_bb(x,z) \(\leftarrow\) d_p2_bb(x,z). (D3) (DT Step 3) d_p2_bb(y,z) \(\leftarrow\) d_p2_bb(x,z), not p(x,z), e2(x,y). (D4) (DT Step 3)
The resulting rules are not stratified: d_p2_bb negatively depends on p, in (D4), and p transitively depends on d_p2_bb through d_p_bb, in (R1’) and (D2).
How does Topdown handle stratified negation without changes? The key is that Topdown has a sequential order for when each subquery is processed, whereas Bottomup infers facts of demand predicates, i.e., the predicates with prefix d_, without a sequential order. We illustrate this for the extended example above.
Example.
In the example above, Topdown would ask p2(1,2)?, which asks not p(1,2)? using rule (R4), and assuming it returns true, Topdown would find a value for y, say 3 and ask p2(3,2)?, then again using (R4) would ask not p(3,2)?, and so on.
In contrast, in the resulting rules from DT, a demand for p2 for the third hypothesis of (R4) depends on the truth value of p, whose demand depends on the demand for p2 as it appears in a rule whose conclusion’s predicate is p2, leading to a cyclic definition with negation.
It has been shown that wellfounded semantics of magicset transformed stratified rules is twovalued [DBLP:journals/tcs/KempSS95]; it can be shown easily that this applies to DT as well. This agrees with the understanding that Topdown computes wellfounded semantics [DBLP:journals/jacm/ChenW96]. Then, one could evaluate such rules after DT by computing wellfounded semantics. However, the best bottomup methods known for computing wellfounded semantics are quadratic time in the grounded rules [DBLP:journals/tplp/BrassDFZ01], which implies a prohibitive bound, where is the number of constants, and is the maximum number of variables in a rule.
A recent semantics, founded semantics [DBLP:conf/lfcs/LiuS18] can compute wellfounded semantics in linear time in the absence of what they call closed predicates. However, it can be shown that the example above requires closed predicates.
A number of serious efforts have been made to address the problem of nonstratified negation in the resulting rules from MST, as discussed in Section 7, but all of them are much more complicated than even MST, and none of them provide precise complexity guarantees, let alone matching Topdown.
4 Extending demand transformation and bottomup evaluation for stratified negation
We present simple extensions to DT and Bottomup so that the demand from the query is precisely captured, and answers to the query are inferred efficiently with precise time complexity guarantees. The resulting evaluation and complexity match Topdown exactly.
4.1 Extended demand transformation
We extend DT to handle stratified negation by introducing a new predicate n.p for each predicate p that appears in a negated hypothesis and using n.p in place of not p. Predicate n.p therefore stands for the complement of p. We use the demand for n.p to create demand for p. We then extend Bottomup in the next subsection to infer facts of n.p by exploiting stratification. The extended DT has three steps.

Replace each negated hypothesis not p(args) with n.p(args), where n.p is a new predicate.

Add a rule n.p(a,…,a) not p(a,…,a). for each n.p of arguments, where a,…,a are distinct variables.

Apply DT as before, treating demand for not p(args) as demand for p(args).
Note that the only rules that contain such negated hypotheses are rules added in Step 2.
Example.
For the extended running example, extended DT yields the following rules.
p(x,y) \(\leftarrow\) d_p_bb(x,y), e(x,y). (R1’) (Step 3, DT Step 1) p(x,z) \(\leftarrow\) d_p_bb(x,z), e(x,y), p(y,z). (R2’) (Step 3, DT Step 1) p2(x,y) \(\leftarrow\) d_p2_bb(x,y), n.p(x,y), e2(x,y). (R3”) (Steps 1,3, DT Step 1) p2(x,z) \(\leftarrow\) d_p2_bb(x,z), n.p(x,z), e2(x,y), p2(y,z). (R4”) (Steps 1,3, DT Step 1) n.p(x,y) \(\leftarrow\) d_n.p_bb(x,y), not p(x,y). (N1) (Steps 2,3, DT Step 1) d_p2_bb(1,2). (DF) (Step 3, DT Step 2) d_p_bb(y,z) \(\leftarrow\) d_p_bb(x,z), e(x,y). (D1) (Step 3, DT Step 3) d_n.p_bb(x,y) \(\leftarrow\) d_p2_bb(x,y). (D2’) (Step 3, DT Step 3) d_n.p_bb(x,z) \(\leftarrow\) d_p2_bb(x,z). (D3’) (Step 3, DT Step 3) d_p2_bb(y,z) \(\leftarrow\) d_p2_bb(x,z), n.p(x,z), e2(x,y). (D4’) (Steps 1,3, DT Step 3) d_p_bb(x,z) \(\leftarrow\) d_n.p_bb(x,z). (DN1) (Steps 2,3, DT Step 3)
Note the following observations:

Except for (N1), the resulting rules contain no negation, because not p(args) has been replaced with n.p(args) in all other rules.

The rule (DN1) is added in Step 3 of extended DT for the demand resulting from the second hypothesis of the rule (N1).

The new predicate n.p is still in a cycle containing negation with p, so the rules are not stratified.
Next, we extend bottomup evaluation to handle resulting rules that contain negation so that facts of new predicates n.p, and in turn facts of all predicates, can be inferred correctly.
4.2 Extended bottomup evaluation
We extend Bottomup so that facts for each predicate n.p are inferred and the rest of the evaluation proceeds as before with optimal time complexity.
We make the following observation that underlies the idea of the extension.
Lemma 1
Given a set of stratified rules and a query, let be the resulting rules after extended DT. After Bottomup of , consider predicate p in the lowest stratum in among predicates such thatd_n.p_s(args) has been inferred: if p(args) has not been inferred, then the generated rule n.p(a,…,a) d_n.p_s(a,…,a), not p(a,…,a). by extended DT can be used to infer n.p(args).
Proof. Since p is in the lowest stratum among all predicates for which there is a negated demand, there is no negated hypothesis that p transitively depends on. If d_n.p_s(args) has been inferred, then because of a rule introduced in Step 3 of extended DT, d_p_s(args) has been inferred as well. Since p only depends on d_p_s and other positive hypotheses, p(args) must be inferred by Bottomup if true, and not inferred otherwise. Therefore if it has not been inferred, it is false. Since n.p is the complement of p, n.p(args) is true.
Therefore, running Bottomup without rules containing negation, after Bottomup reaches a fixed point, new facts for a predicate n.p in the lowest stratum, for which the corresponding demand predicate d_n.p_s has a fact, can be inferred. After inferring facts for n.p, Bottomup may be applied again, and reach a new fixed point. After the new fixed point is reached, if there are new facts for a demand predicate d_n.p_s, then n.p facts can be inferred again for such facts for the predicate in the lowest stratum, and so on. Precisely, Bottomup after extended DT is extended as follows, and we refer to it as extended Bottomup after extended DT.
Repeat the following steps until fixed point.

Perform Bottomup without using rules that infer facts of predicates of the form n.p. Such rules are first added in Step 2 of extended DT and then transformed in Step 3 of extended DT, and are of the form n.p(a,…,a) dn.ps(a,…,a), not p(a,…,a). where s is all ’s.

Use a rule to infer a new fact n.p(args) if dn.ps is in the lowest stratum among predicates of the form dn.qs for some predicate q for which (i) dn.ps(args) has been inferred and (ii) p(args) has not been inferred.
We extend Lemma 1 to show correctness at any iteration of the extended evaluation.
Lemma 2
Suppose d_n.p_s(args) is inferred during extended Bottomup after extended DT. Ifp(args) is false for the original set of rules, then n.p(args) is inferred by extended Bottomup after extended DT, and not inferred otherwise.
Proof. (Case 1) p(args) is false for the original set of rules. Then, this fact cannot be inferred by the rules defining p in the rules obtained after extended DT since they are more restrictive than the original rules. Given that d_n.p_s(args) (say ) is inferred during extended Bottomup after extended DT, it is guaranteed that if the rule for n.p is ever used, then n.p(args) will be inferred. The rule for n.p is guaranteed to be used since at some point in the iteration will be the fact in the lowest stratum in Step 2.
(Case 2) p(args) is true for the original set of rules. Since d_n.p_s(args) is inferred during extended Bottomup after extended DT, then d_p_s(args) is inferred also due to the rule generated in Step 3 of extended DT. Hence, p(args) will be inferred during extended Bottomup after extended DT since there is a demand for it, and all of the hypotheses that define it must be in a lower stratum, and hence correctly inferred also. Since p(args) is inferred, n.p(args) cannot be inferred.
Theorem 1 (Extended Bottomup after extended DT matches Topdown)
A fact is inferred during extended Bottomup of a set of stratified rules and query after extended DT iff it is inferred during Topdown of the rules and query.
Proof Sketch. This relationship is already known for Bottomup after DT and Topdown for rules without stratified negation, therefore we only need to extend the argument to handle negated hypotheses.
If a demand fact for a negated hypothesis is inferred during extended Bottomup after extended DT, then all hypotheses to its left must be true, therefore a corresponding query must be evaluated during Topdown, and vice versa. So demand facts inferred correspond exactly to subqueries encountered during Topdown. By Lemma 2, if there is a demand for a negated hypothesis, then its corresponding fact is inferred correctly. Therefore, extended Bottomup after extended DT corresponds exactly to Topdown in terms of inferred facts.
5 Precise complexity guarantees
We give precise time complexity analysis using the following parameters as in [DBLP:journals/toplas/LiuS09].

#p: number of facts of predicate p, called size of p.

#p.i,…,i/j,…,j: maximum number of combinations of values taken by the i,…,ith arguments of facts of predicate p, given any fixed value of the j,…,jth arguments.
There are two forms of rules after decomposition of rules. When a rule has one hypothesis, it is of the form: p(x,…,x) q(y,…,y). The number of times this rule can fire is the number of facts of q, therefore the time complexity incurred by this rule is . In fact, we can omit the complexity of such rules, because (i) if q is an extensional predicate, then all its facts need to be read in, therefore the complexity is already incurred by the reading of the input, (ii) if q is an intensional predicate, then its size is bound by the complexity of the rules that infer its facts.
When a rule has two hypotheses, it is of the form: p(args) q(x,…,x), r(y,…,y). To calculate the number of firings, we can first consider facts of q and find matching facts of r such that common variables in the two hypotheses take the same value. Therefore, only variables in the second hypothesis but not in the first can take different values for each fact of q. Let be the set of indices in such that y is a common variable in the two hypotheses, then the complexity is bounded by . Symmetrically, we can consider facts of r and find matching facts of q. Let be symmetrically the set of indices in such that x is a common variable, then the complexity is also bounded by . Both are upper bounds, so the complexity is bounded by the minimum of the two:
Example.
For the running example, rule (R1) incurs the time complexity , and rule (R2) incurs the time complexity .
5.1 Complexity characteristics of extended Bottomup after extended DT
Theorem 2 (Extensions preserve complexity)
Extended Bottomup after extended DT preserves the complexity characteristics and optimality of Bottomup after DT.
Proof. The only difference in extended Bottomup is Step 2 where a demand fact for predicates of the form n.p in the lowest stratum is found. Such a fact can be found in constant time in data complexity, by searching for demand facts starting from the lowest stratum upwards. Therefore, the extension preserves the complexity characteristics and the optimality of Bottomup after DT.
The precise time and space complexity of Topdown, and its relationship with Bottomup after MST and DT, were open until [DBLP:conf/ppdp/TekleL10]. The results in [DBLP:conf/ppdp/TekleL10] include the following theorems—they establish that the time complexity of Topdown and that of Bottomup after DT are equal for rules with at most two hypotheses each, and for general Datalog rules, Bottomup after DT is equal to or faster than Topdown.
Theorem 3 (Bottomup after DT equals Topdown on decomposed rules [DBLP:conf/ppdp/TekleL10])
Let be a set of Datalog rules and a query, such that the rules have no singleton variables, and there are no more than two hypotheses per rule. Let be the set of rules and fact after demand transformation of . Let be the asymptotic time complexity of Topdown of , and be the asymptotic time complexity of Bottomup of . Then, .
Theorem 4 (Bottomup after DT beats Topdown [DBLP:conf/ppdp/TekleL10])
Let be a set of Datalog rules and a query. Let be the set of rules and fact after demand transformation of . Let be the asymptotic time complexity of Topdown of , and be the asymptotic time complexity of Bottomup of . Then, .
By Theorem 2, and the fact that Topdown behaves identically for stratified negation, these theorems can be extended as follows.
Corollary 1 (Extended Bottomup after extended DT equals Topdown on decomposed rules)
Let be a set of stratified rules and a nonfloundering query, such that the rules have no singleton variables, and there are no more than two hypotheses per rule. Let be the set of rules and fact after extended demand transformation of . Let be the asymptotic time complexity of Topdown of , and be the asymptotic time complexity of extended Bottomup of . Then, .
Corollary 2 (Extended Bottomup after extended DT beats Topdown)
Let be a set of stratified rules and a nonfloundering query. Let be the set of rules and fact after extended demand transformation of . Let be the asymptotic time complexity of Topdown of , and be the asymptotic time complexity of extended Bottomup of . Then, .
Example.
For the extended running example after extended DT, we calculate the time complexity by considering the most complex rule, copied below.
p2(x,z) \(\leftarrow\) d_p2_bb(x,z), n.p(x,z), e2(x,y), p2(y,z). (R4”)
Extended Bottomup decomposes this rule into three rules:
i1(x,z) \(\leftarrow\) d_p2_bb(x,z), n.p(x,z). (R4”1) i2(x,y,z) \(\leftarrow\) i1(x,z), e2(x,y). (R4”2) p2(x,z) \(\leftarrow\) i2(x,y,z), p2(y,z). (R4”3)
Rules (R4”1) and (R4”3) have a hypothesis containing all variables occurring in the rule, so they do not contribute any extra complexity. The time complexity incurred by (R4”2) is
Note that this complexity is precise, and can be very small because i1 is the intersection of d_p2_bb and n.p.
The complexity benefit of extended Bottomup after extended DT is even more apparent when contrasted to evaluating the resulting rules bottomup for wellfounded semantics whose worstcase time complexity is where is the number of constants, whereas the worstcase time complexity using our method is .
6 Applications and experiments
6.1 Experiments on the extended running example
To confirm the effectiveness of our method, we implemented our method to perform extended DT, and then created a Python program that performs extended Bottomup on the resulting rules. We compare our results with stateoftheart systems: clingo [DBLP:journals/aicom/GebserKKOSS11], an ASP solver, DLV2, the latest version of a deductive database with ASP and dynamic magic sets [alviano2012magic, alviano2017asp], and XSB [DBLP:journals/jacm/ChenW96], the dominant engine with tabling and wellfounded semantics.
We show experiments on the extended running example. For clingo, which does not support answering queries, we give it the transformed set of rules and facts. For XSB, we used a special construct, load_dynca to turn off indexing when loading facts, improving the performance more than 10 times. We use facts generated for predicates e and e2, by varying the number of nodes and edges. The running times were captured on an Intel Core i5 2.8 GHz with 8 GB RAM, with PyPy 7.1.0 to run Python programs, clingo 5.3.0, DLV2, and XSB 3.8, including the time for reading facts and producing output, and are averaged over 5 runs. The results are shown in Table 1, where K stands for 1000.
# Nodes  # Edges  EBU  clingo  clingoEBU  DLV2  DLV2/EBU  XSB  XSBEBU 

1K  200K  2.392  5.549  2.31  1.336  0.56  1.201  0.50 
1K  400K  4.597  9.845  2.14  2.599  0.57  2.471  0.54 
1K  600K  6.898  13.783  1.99  4.972  0.72  3.606  0.52 
2K  600K  7.871  18.129  2.30  5.573  0.71  3.676  0.47 
2K  800K  10.999  24.059  2.18  8.012  0.73  4.731  0.43 
2K  1000K  13.978  29.669  2.12  7.260  0.52  6.008  0.43 
One can see that the running times for extended Bottomup are tightly coupled with the number of edges as shown in the formula in Example 5.1. It is difficult to speculate on the time and space complexities for clingo and DLV2 because precise complexity analysis as shown for our method is not known. The strength of our method can be seen in contrast to these mature systems even though our Python program is not optimized for constant factors, and Python is known to be much slower than C and C++ used to implement XSB, DLV2, and Clingo.
6.2 Balbin’s notreachinreach2 problem
Balbin et al. [DBLP:journals/jlp/BalbinPRM91, Example 12] give the following rules to show the challenge of nonstratified negation resulting from applying MST on stratified rules (some predicates/variables have been renamed for readability):
r(x) \(\leftarrow\) s(x). (B1) r(x) \(\leftarrow\) e(x,y), r(y). (B2) r2(x) \(\leftarrow\) s2(x). (B3) r2(x) \(\leftarrow\) not r(x), e2(x,y), r2(y). (B4)
along with the query r2(1)?.
Using our method, the resulting rules from extended DT are:
r(x) \(\leftarrow\) d_r_b(x), s(x). (B1’) r(x) \(\leftarrow\) d_r_b(x), e(x,y), r(y). (B2’) r2(x) \(\leftarrow\) d_r2_b(x), s2(x). (B3’) r2(x) \(\leftarrow\) d_r2_b(x), n.r(x), e2(x,y), r2(y). (B4’) n.r(x) \(\leftarrow\) d_n.r_b(x), not r(x). (N1) d_r2_b(1). (DF) d_r_b(y) \(\leftarrow\) d_r_b(x), e(x,y). (D1) d_n.r_b(y) \(\leftarrow\) d_r2_b(x). (D2) d_r2_b(y) \(\leftarrow\) d_r2_b(x), n.r(x), e2(x,y). (D3) d_r_b(x) \(\leftarrow\) d_n.r_b(x). (D4)
These rules can be evaluated in linear time in the input size, because after decomposition into rules with two hypotheses, each rule has at least one hypothesis with extensional predicate s, e, s2, e2 that contains all variables in the rule. Contrast this to bottomup evaluation for wellfounded semantics, which would have only the bound of , where is the number of constants, since there is a rule with two variables.
6.3 MeskesNoack’s noextrajoinsinpath problem
Meskes and Noack [DBLP:journals/ipl/MeskesN93, Example 1] give the following rules to show that a negated predicate does not need to be in a positive cycle for nonstratified rules to occur after MST (some predicates/variables have been renamed for simplicity):
s(x) \(\leftarrow\) q(x,z), r(z,y). (M1) p(x,y) \(\leftarrow\) e(x,y), not s(y). (M2) p(x,z) \(\leftarrow\) e(x,y), p(y,z), not s(y). (M3)
along with the query p(1,y)?. It can be seen that the only negated predicate is s, and it is simply defined using extensional predicates, but after DT, nonstratified rules are obtained.
Extended DT yields the following rules:
s(x) \(\leftarrow\) d_s_b(x), q(x,z), r(z,y). (M1’) p(x,y) \(\leftarrow\) d_p_bf(x), e(x,y), n.s(y). (M2’) p(x,z) \(\leftarrow\) d_p_bf(x), e(x,y), p(y,z), n.s(y). (M3’) n.s(x) \(\leftarrow\) d_n.s_b(x), not s(x). (N1) d_p_bf(1). (DF) d_n.s_b(y) \(\leftarrow\) e(x,y). (D1) d_p_bf(y) \(\leftarrow\) d_p_bf(x), e(x,y). (D2) d_n.s_b(y) \(\leftarrow\) d_p_bf(x), e(x,y), p(y,z). (D3) d_s_b(x) \(\leftarrow\) d_n.s_b(x). (D4)
(M3’) after decomposition gives the highest time complexity: where i is an intermediate predicate for the leftmost two hypothesis, and is a subset of e restricted by d_p_bf for the first argument. Like our original running example, the time complexity is bounded by using extended Bottomup after extended DT versus for bottomup evaluation for wellfounded semantics.
7 Related work and conclusion
Datalog has been extensively studied, especially methods for efficient evaluation [Maier:2018:DCH:3191315.3191317, DBLP:books/aw/AbiteboulHV95].
Topdown evaluation with variant tabling was introduced in the 1980s [DBLP:conf/iclp/TamakiS86] and has been widely studied as well. It has been implemented in topdown evaluation engines such as XSB [DBLP:journals/jacm/ChenW96] and SWI [DBLP:journals/tplp/WielemakerSTL12]. Optimal bottomup evaluation was given in [DBLP:journals/toplas/LiuS09]. Incremental maintenance of Datalog with stratified negation was studied in [motik2019maintenance].
Transformations for demanddriven bottomup evaluation have been studied, in many forms of the wellknown magicset transformation (MST) [DBLP:conf/pods/BancilhonMSU86], as well as the more recent demand transformation (DT) [DBLP:conf/ppdp/TekleL10] that our work extends.
A method for precise calculation of time and space complexities of topdown evaluation with variant tabling for Datalog, as well as theorems establishing the precise relationship between topdown and bottomup evaluations are given in [DBLP:conf/ppdp/TekleL10, DBLP:conf/sigmod/TekleL11].
Although topdown evaluation handles stratified negation without changes, an analogous demanddriven bottomup evaluation has been challenging to obtain.
Balbin et al. [DBLP:journals/jlp/BalbinPRM91] introduce a transformation to a set of magicset transformed rules, that works for an unidentified class of rules, and does not apply to all rules with stratified negation. They then introduce a new method for bottomup evaluation with no complexity characterization.
Meskes and Noack [DBLP:journals/ipl/MeskesN93] extend generalized supplementary MST [DBLP:journals/jlp/BeeriR91] by ignoring negated hypotheses for rules that infer demand facts to keep the rules stratified, but the method overestimates the demand and hence does not correspond to topdown evaluation.
Ross [DBLP:journals/jacm/Ross94] presents a new transformation whose output is not Datalog, but a higherorder logic, complicating their evaluation and giving no complexity analysis.
Alviano et al. [alviano2012magic] extends magic sets for disjunctive Datalog, but its bottomup computation requires grounding and stable model search, not giving precise complexity guarantees as we do.
It has been shown that the wellfounded semantics of rules obtained by MST for stratified rules is twovalued [DBLP:journals/tcs/KempSS95], but the best known complexity for bottomup evaluation for wellfounded semantics is a highly prohibitive quadratic time in the size of the grounded rules.
Our work extends demand transformation and optimal bottomup evaluation and gives precise complexity guarantees for efficient demanddriven bottomup evaluation of stratified rules, preserving optimality, and relationships with topdown evaluation.
Future work includes methods for efficient bottomup evaluation and demanddriven evaluation for nonstratified negation with precise complexity guarantees.
Acknowledgments.
This work was supported in part by NSF under grants CCF1414078 and IIS1447549.
References
 [1] (200206) A LaTeX macro package for message sequence charts—maintenance document—describing version . Note: Included in MSC macro package distribution
 [2] (200206) A LaTeX macro package for message sequence charts—reference manual—describing version . Note: Included in MSC macro package distribution
 [3] (200206) A LaTeX macro package for message sequence charts—user manual—describing version . Note: Included in MSC macro package distribution
 [4] (1997) The LaTeX Graphics Companion. AddisonWesley.
 [5] (1997) ITUTS Recommendation Z.120: Message Sequence Chart (MSC). Geneva.
 [6] (1994) LaTeX—a document preparation system—user’s guide and reference manual. 2nd edition, AdsisonWesley. Note: Updated for LaTeX2e
 [7] (1996) Tutorial on message sequence charts (MSC’96). In FORTE,
Comments
There are no comments yet.