# Extended Magic for Negation: Efficient Demand-Driven Evaluation of Stratified Datalog with Precise Complexity Guarantees

Given a set of Datalog rules, facts, and a query, answers to the query can be inferred bottom-up starting from the facts or top-down starting from the query. For efficiency, top-down evaluation is extended with memoization of inferred facts, and bottom-up evaluation is performed after transformations to make rules driven by the demand from the query. Prior work has shown their precise complexity analysis and relationships. However, when Datalog is extended with even stratified negation, which has a simple and universally accepted semantics, transformations to make rules demand-driven may result in non-stratified negation, which has had many complex semantics and evaluation methods. This paper presents (1) a simple extension to demand transformation, a transformation to make rules demand-driven for Datalog without negation, to support stratified negation, and (2) a simple extension to an optimal bottom-up evaluation method for Datalog with stratified negation, to handle non-stratified negation in the resulting rules. We show that the method provides precise complexity guarantees. It is also optimal in that only facts needed for top-down evaluation of the query are inferred and each firing of a rule to infer such a fact takes worst-case constant time. We extend the precise relationship between top-down evaluation and demand-driven bottom-up evaluation to Datalog with stratified negation. Finally, we show experimental results for performance, as well as applications to previously challenging examples.

## Authors

• 1 publication
• 13 publications
• ### Acyclicity Notions for Existential Rules and Their Application to Query Answering in Ontologies

Answering conjunctive queries (CQs) over a set of facts extended with ex...
02/04/2014 ∙ by Bernardo Cuenca Grau, et al. ∙ 0

• ### Demand-driven Alias Analysis : Formalizing Bidirectional Analyses for Soundness and Precision

A demand-driven approach to program analysis have been viewed as efficie...
02/03/2018 ∙ by Swati Jaiswal, et al. ∙ 0

• ### Semantic Reasoning with Differentiable Graph Transformations

This paper introduces a differentiable semantic reasoner, where rules ar...
07/20/2021 ∙ by Alberto Cetoli, et al. ∙ 14

• ### Datalog Reasoning over Compressed RDF Knowledge Bases

Materialisation is often used in RDF systems to derive, in a preprocessi...
08/27/2019 ∙ by Pan Hu, et al. ∙ 9

• ### RuDaS: Synthetic Datasets for Rule Learning and Evaluation Tools

Logical rules are a popular knowledge representation language in many do...
09/16/2019 ∙ by Cristina Cornelio, et al. ∙ 0

• ### Worst-case Optimal Query Answering for Greedy Sets of Existential Rules and Their Subclasses

The need for an ontological layer on top of data, associated with advanc...
12/15/2014 ∙ by Sebastian Rudolph, et al. ∙ 0

• ### Dynamic Jointrees

It is well known that one can ignore parts of a belief network when comp...
01/30/2013 ∙ by Adnan Darwiche, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Datalog [Maier:2018:DCH:3191315.3191317] is a logic language for deductive databases [DBLP:books/aw/AbiteboulHV95], security [DBLP:conf/sp/DeTreville02], networking [DBLP:journals/cacm/LooCGGHMRRS09], semantic web [DBLP:journals/ws/CaliGL12], and many other applications [WarLiu17AppLP-arxiv].

Given a set of Datalog rules, facts, and a query, answers to the query can be inferred using bottom-up evaluation starting with the facts or top-down evaluation starting with the query. The dominant strategies for efficient evaluations are top-down evaluation with variant tabling [DBLP:conf/iclp/TamakiS86] to memoize and reuse answers to subqueries, and bottom-up evaluation with the magic set transformation (MST) [DBLP:conf/pods/BancilhonMSU86] so that evaluation of the transformed rules are driven by demand from the query.

The performance of Datalog engines is difficult to understand due to the multitude of factors involved. For example, the performance of different tabling strategies vary drastically [DBLP:conf/iclp/RaoRR96], and bottom-up evaluation after MST may be much slower than the bottom-up evaluation of the original rules [DBLP:conf/sigmod/SereniAM08]. Choosing the best evaluation method for a given set of rules requires precise complexity analysis of each evaluation method. Recently, demand transformation (DT), resulting in simpler rules with better space complexity than MST has been introduced, and the time and space complexities of top-down evaluation with tabling and bottom-up evaluation after DT has been precisely established.

Stratified negation is a simple kind of negation with a simple and universally accepted semantics, and it is handled gracefully by top-down evaluation with tabling requiring almost no change. However, MST applied to rules with stratified negation is known to result in rules with non-stratified negation, which has many complex semantics, such as well-founded semantics [DBLP:journals/jacm/GelderRS91] and stable model semantics [DBLP:conf/iclp/GelfondL88]. Under such semantics, precise complexity analysis is difficult, and worst-case behavior is prohibitive, as in the quadratic time complexity in the size of the grounded rules for well-founded semantics. A more recent semantics, founded semantics [DBLP:conf/lfcs/LiuS18], can evaluate a certain class of rules with arbitrary negation in linear time, but not for all rules. Other efforts to address this problem have been made, as discussed in Section 7, but they are more complicated than even MST, and none of them provide precise complexity guarantees, or proofs that they match top-down evaluation with tabling.

This paper presents a new method consisting of (1) a simple extension to demand transformation, a transformation to make rules demand-driven for Datalog without negation, to support stratified negation, and (2) a simple extension to an optimal bottom-up evaluation method for Datalog with stratified negation, to handle non-stratified negation in the resulting rules. We show that the method provides precise complexity guarantees, which is worst-case linear time in the size of the grounded rules, in contrast to existing quadratic time results. The method only infers facts needed for top-down evaluation of the query, and each firing of a rule to infer such a fact takes worst-case constant time. We extend the precise relationship between top-down evaluation and demand-driven bottom-up evaluation to Datalog with stratified negation. Finally, we show experimental results for performance, as well as applications to previously challenging examples.

## 2 Datalog with negation

Datalog [Maier:2018:DCH:3191315.3191317] is a language for defining rules, facts, and queries, where rules can be used with facts to answer queries.

A Datalog rule is of the form:

 p(a_1,...,a_k) ← p_1(a_11,...,a_1k_1),..., p_h(a_h1,...,a_hk_h).

where is a finite natural number, each (respectively ) is a predicate of finite number (respectively ) arguments, each and is either a constant or a variable, and each variable in the arguments of must also be in the arguments of some .

If , then each must be a constant, in which case is called a fact.

For the rest of the paper, rule refers only to the case where , in which case each is called a hypothesis, also known as premise, and is called the conclusion.

The meaning of a set of rules and facts is the set of facts that are given or can be inferred using the rules.

A query is of the form where each argument is either a constant or a variable. Answers to a query is the set of facts, given or inferred, that match the query, where a fact matches a query if there is a substitution of the variables in to constants such that under the substitution is the same as .

We call a predicate that appears in the conclusion of a rule an intensional predicate, and a predicate that does not appear in the conclusion of any rule but appears only in given facts an extensional predicate. We assume without loss of generality that intensional and extensional predicates are distinct.

In examples, we use letters for variables in arguments of predicates, and we use literal numbers for constants.

#### Example.

We use the following rules for transitive closure as a running example.

p(x,y) $$\leftarrow$$ e(x,y).          (R1)
p(x,z) $$\leftarrow$$ e(x,y), p(y,z).  (R2)


We consider the query p(1,x)?. It asks for all nodes reachable from 1 following the edges.

#### Stratified negation.

Datalog can be extended with stratified negation by extending rules to be of the form, where [not] indicates that not is optional:

 p(a_1,...,a_k) ← [\small not]p_1(a_11,...,a_1k_1),..., [% \small not]p_h(a_h1,...,a_hk_h).

and imposing the constraint that there is no cyclic dependency between any predicate and a negated predicate, i.e., a predicate preceded with a not. For example, replacing (R2) in the running example with the following rule, where q is an extensional predicate, results in Datalog with stratified negation:

p(x,z) $$\leftarrow$$ e(x,y), p(y,z), not q(x,z).  (R2’S)


because there is no cyclic dependency between p and q.

Formally, we say that predicate (negatively) depends on predicate if there is a rule whose conclusion’s predicate is and there is a (negative) hypothesis in the rule whose predicate is . A set of rules is stratified if for any two predicates and , if negatively depends on , then does not transitively depend on . Predicates in a stratified set of rules can be split into numbered strata such that for any two strata and , if a predicate in negatively depends on a predicate in , then has a higher number than .

Datalog with stratified negation has a simple and universally accepted semantics, where rules having predicates in a lower strata are evaluated first, and negated hypotheses under a substitution is considered false if the corresponding facts have not been inferred. This semantics coincides with all well-known semantics for negation, including perfect model semantics [DBLP:journals/jar/Przymusinski89], well-founded semantics [DBLP:journals/jacm/GelderRS91], and stable model semantics [DBLP:conf/iclp/GelfondL88].

#### Non-stratified negation.

Removing the constraint that there is no cyclic dependency between any predicate and a negated predicate may result in non-stratified negation, which has no universally accepted semantics. Consider the following rule:

t(x) $$\leftarrow$$ not t(x).


For a constant c, if t(c) is false, then it is also true by this rule, therefore standard semantics does not apply. Many sophisticated semantics have been developed, including well-founded semantics [DBLP:journals/jacm/GelderRS91] and stable model semantics [DBLP:conf/iclp/GelfondL88]. Also, under sophisticated semantics, precise complexity analysis is difficult, and worst-case behavior is prohibitive, e.g., quadratic time in the size of the grounded rules for well-founded semantics [berman1995computing]. Therefore, rules with non-stratified negation must be avoided whenever possible.

## 3 Demand-driven evaluation and challenge of stratified negation

This section describes top-down and bottom-up methods for answering queries, and challenge of stratified negation.

#### Top-down evaluation with tabling.

To answer a query, top-down evaluation starts with the query, generates subqueries from hypotheses of rules whose conclusions match the query, considering rules in the order given, and considering hypotheses from left to right, and does so repeatedly until the subqueries match given facts.

Straightforward top-down evaluation may lead to repeated subqueries, or even infinite recursion for recursive rules such as (R2) in the running example. To address this problem, tabling memoizes answers to queries encountered, and reuses the answers when a query is encountered again.

In this paper, we consider top-down evaluation using the dominant tabling strategy, variant tabling without early completion, which exploits all ways to infer the answers to a query modulo variable renaming [DBLP:journals/jacm/ChenW96]. We refer to this evaluation as Top-down in the rest of the paper.

Stratified negation is generally supported for queries that are non-floundering [DBLP:journals/jacm/ChenW96]. A query is non-floundering with respect to a set of rules if during Top-down of the query, all subqueries for negated hypotheses have all of their arguments bound.

In this paper, we only consider non-floundering queries of Datalog with stratified negation, because these queries can be evaluated by Top-down with a trivial extension to test the negation.

#### Optimal bottom-up evaluation and demand-driven transformations.

Bottom-up evaluation starts with given facts, infers new facts from conclusions of rules whose hypotheses match existing facts, and does so repeatedly until no more facts can be inferred.

An optimal method [DBLP:journals/toplas/LiuS09] transforms any given set of Datalog rules into an efficient specialized procedural implementation with guaranteed worst-case time and space complexities, and computes the complexities from the rules.

In particular, rules are first transformed to remove singleton variables (variables that occur in only one hypothesis) and multiple occurrences of the same variable in a single hypothesis. Next, rules with more than two hypotheses are decomposed into rules of two hypotheses. Then, the least fixed-point specification of the rules is transformed to a while-loop that considers given and inferred facts incrementally, one at a time, where auxiliary indices are used to find each matching fact in constant time.

The evaluation is optimal in that only combinations of facts that make the hypotheses of a rule simultaneously true are considered, and each such combination, which leads to a firing of the rule, is considered once in constant time. The time complexity is precisely the sum of the number of firings over all rules plus the size of extensional predicates for reading given facts.

In this paper, we only consider the decomposition of rules that takes the two leftmost hypotheses into a new rule at a time. We call this left-optimal bottom-up evaluation, because the time complexity of evaluation using this decomposition is optimal for the left-to-right ordering of the hypotheses in a rule. We refer to this evaluation as Bottom-up in the rest of the paper.

Note that the optimal bottom-up evaluation in [DBLP:journals/toplas/LiuS09] is not limited to the particular decomposition used by Bottom-up. It can perform time and space complexity calculation for all possible decompositions, and select an optimal one trading space for time, including space for intermediate predicates from the decompositions. We consider Bottom-up in this paper in order to establish correspondence with the left-to-right evaluation of hypotheses in Top-down.

Bottom-up infers all facts possible from the given facts, and thus may infer many facts not needed for answering the query. For efficient evaluation, the query can be used to limit the facts that can be inferred. This is achieved by transforming the rules to be restricted by the query, like in the well-known magic set transformation (MST) [DBLP:conf/pods/BancilhonMSU86].

Demand transformation (DT) [DBLP:conf/ppdp/TekleL10] is such a transformation that results in rules with the same time complexity and exponentially better space complexity in program size than MST. It transforms a set of rules and a query into a new set of rules and a fact, which can infer only facts that can be inferred during Top-down of the original rules. DT and MST make use of predicate annotations for argument binding patterns. DT differs from MST in that given intensional predicates are not annotated in transformed rules, therefore is much simpler, and much less space is used because a fact of a predicate is stored only once, rather than once for each possible annotation of the predicate.

DT has two stages: (I) compute demand patterns for intensional predicates, that is, a set of pairs indicating that under Top-down for the given query, there is a subquery for predicate p with pattern s, a string of characters for arguments of the subquery, where the th character is b if the th argument is a constant (i.e., bound), and f if it is a variable (i.e., free), and (II) transform given rules and query using the demand patterns computed, in three steps, as follows.

1. For each demand pattern , and each given rule p(args)  h,,h., generate following rule

p(args)  d_p_s(a,,a), h,,h.

where args denotes arguments in the conclusion, a,…,a are arguments in args that are bound by s, i.e., that correspond to character b in s.

2. For the given query of the form p(args)?, generate the following fact:

d_p_s(a,,a).

where s is the pattern for , and a,…,a are the constant arguments in args.

3. For each rule generated in Step 1, c  h,,h., and each h whose predicate is an intensional predicate p, generate the following rule:

d_p_s(a,,a)  h,,h.

where s is the pattern of h, and a,…,a are the bound arguments of h.

#### Example.

For the running example, the only demand pattern is , because the only type of query encountered during Top-down is one where the first argument of p is a constant. DT yields the following rules and fact:

p(x,y) $$\leftarrow$$ d_p_bf(x), e(x,y).          (R1’)   (DT Step 1)
p(x,z) $$\leftarrow$$ d_p_bf(x), e(x,y), p(y,z).  (R2’)   (DT Step 1)
d_p_bf(1).                            (DF)    (DT Step 2)
d_p_bf(y) $$\leftarrow$$ d_p_bf(x), e(x,y).       (DR)    (DT Step 3)


(R1’) and (R2’) restrict the original (R1) and (R2) using a new demand predicate d_p_bf indicating demand on the first of two arguments of predicate p. (DF) is a new fact that corresponds to demand by the given query where ther first of the two arguments of p is 1. (DR) propagates demand in (R2’) from d_p_bf(x) via e(x,y) to demand on the first argument y in p(y,z).

#### Challenge of stratified negation for MST and DT.

Not only can Top-down handle non-floundering queries of stratified Datalog with a trivial extension [DBLP:journals/jacm/ChenW96], which checks whether a negative hypothesis is true under the current substitution; but also can Bottom-up be trivially extended and give precise time and space complexity guarantees as for without negation [DBLP:journals/toplas/LiuS09], by checking whether a fully instantiated negative hypothesis is true in time.

It is therefore natural to think that MST and DT would apply to stratified negation as well, adding demands that mimic Top-down, so that Bottom-up of the resulting rules has the same optimal complexity as for without negation. Unfortunately, this is not true—applying MST to stratified rules, treating demand for a negated hypotheses as demand for the hypothesis without the negation, may result in non-stratified rules [DBLP:journals/jlp/BalbinPRM91]. We show that DT has the same problem as MST through the following example.

#### Example.

Consider extending the running example with two additional rules:

p2(x,y) $$\leftarrow$$ not p(x,y), e2(x,y).           (R3)
p2(x,z) $$\leftarrow$$ not p(x,z), e2(x,y), p2(y,z).  (R4)


p2 is the transitive closure of e2 whose computation does not use any pair in p. The four rules are stratified because only p2 depends negatively on p, and p does not transitively depend on p2.

Consider the query p2(1,2)?. The query is non-floundering because the only demand pattern for the negated hypotheses is . DT yields the following rules, treating demand for a negated hypothesis as demand for the hypothesis without the negation:

p(x,y) $$\leftarrow$$ d_p_bb(x,y), e(x,y).                          (R1’)    (DT Step 1)
p(x,z) $$\leftarrow$$ d_p_bb(x,z), e(x,y), p(y,z).                  (R2’)    (DT Step 1)
p2(x,y) $$\leftarrow$$ d_p2_bb(x,y), not p(x,y), e2(x,y).           (R3’)    (DT Step 1)
p2(x,z) $$\leftarrow$$ d_p2_bb(x,z), not p(x,z), e2(x,y), p2(y,z).  (R4’)    (DT Step 1)
d_p2_bb(1,2).                                           (DF)     (DT Step 2)
d_p_bb(y,z) $$\leftarrow$$ d_p_bb(x,z), e(x,y).                     (D1)    (DT Step 3)
d_p_bb(x,y) $$\leftarrow$$ d_p2_bb(x,y).                            (D2)    (DT Step 3)
d_p_bb(x,z) $$\leftarrow$$ d_p2_bb(x,z).                            (D3)    (DT Step 3)
d_p2_bb(y,z) $$\leftarrow$$ d_p2_bb(x,z), not p(x,z), e2(x,y).      (D4)    (DT Step 3)


The resulting rules are not stratified: d_p2_bb negatively depends on p, in (D4), and p transitively depends on d_p2_bb through d_p_bb, in (R1’) and (D2).

How does Top-down handle stratified negation without changes? The key is that Top-down has a sequential order for when each subquery is processed, whereas Bottom-up infers facts of demand predicates, i.e., the predicates with prefix d_, without a sequential order. We illustrate this for the extended example above.

#### Example.

In the example above, Top-down would ask p2(1,2)?, which asks not p(1,2)? using rule (R4), and assuming it returns true, Top-down would find a value for y, say 3 and ask p2(3,2)?, then again using (R4) would ask not p(3,2)?, and so on.

In contrast, in the resulting rules from DT, a demand for p2 for the third hypothesis of (R4) depends on the truth value of p, whose demand depends on the demand for p2 as it appears in a rule whose conclusion’s predicate is p2, leading to a cyclic definition with negation.

It has been shown that well-founded semantics of magic-set transformed stratified rules is two-valued [DBLP:journals/tcs/KempSS95]; it can be shown easily that this applies to DT as well. This agrees with the understanding that Top-down computes well-founded semantics [DBLP:journals/jacm/ChenW96]. Then, one could evaluate such rules after DT by computing well-founded semantics. However, the best bottom-up methods known for computing well-founded semantics are quadratic time in the grounded rules [DBLP:journals/tplp/BrassDFZ01], which implies a prohibitive bound, where is the number of constants, and is the maximum number of variables in a rule.

A recent semantics, founded semantics [DBLP:conf/lfcs/LiuS18] can compute well-founded semantics in linear time in the absence of what they call closed predicates. However, it can be shown that the example above requires closed predicates.

A number of serious efforts have been made to address the problem of non-stratified negation in the resulting rules from MST, as discussed in Section 7, but all of them are much more complicated than even MST, and none of them provide precise complexity guarantees, let alone matching Top-down.

## 4 Extending demand transformation and bottom-up evaluation for stratified negation

We present simple extensions to DT and Bottom-up so that the demand from the query is precisely captured, and answers to the query are inferred efficiently with precise time complexity guarantees. The resulting evaluation and complexity match Top-down exactly.

### 4.1 Extended demand transformation

We extend DT to handle stratified negation by introducing a new predicate n.p for each predicate p that appears in a negated hypothesis and using n.p in place of not p. Predicate n.p therefore stands for the complement of p. We use the demand for n.p to create demand for p. We then extend Bottom-up in the next subsection to infer facts of n.p by exploiting stratification. The extended DT has three steps.

1. Replace each negated hypothesis not p(args) with n.p(args), where n.p is a new predicate.

2. Add a rule n.p(a,,a) not p(a,,a). for each n.p of arguments, where a,…,a are distinct variables.

3. Apply DT as before, treating demand for not p(args) as demand for p(args).

Note that the only rules that contain such negated hypotheses are rules added in Step 2.

#### Example.

For the extended running example, extended DT yields the following rules.

p(x,y) $$\leftarrow$$ d_p_bb(x,y), e(x,y).                       (R1’)   (Step 3,    DT Step 1)
p(x,z) $$\leftarrow$$ d_p_bb(x,z), e(x,y), p(y,z).               (R2’)   (Step 3,    DT Step 1)
p2(x,y) $$\leftarrow$$ d_p2_bb(x,y), n.p(x,y), e2(x,y).          (R3”)   (Steps 1,3, DT Step 1)
p2(x,z) $$\leftarrow$$ d_p2_bb(x,z), n.p(x,z), e2(x,y), p2(y,z). (R4”)   (Steps 1,3, DT Step 1)
n.p(x,y) $$\leftarrow$$ d_n.p_bb(x,y), not p(x,y).               (N1)     (Steps 2,3, DT Step 1)
d_p2_bb(1,2).                                        (DF)    (Step 3,    DT Step 2)
d_p_bb(y,z) $$\leftarrow$$ d_p_bb(x,z), e(x,y).                  (D1)     (Step 3,    DT Step 3)
d_n.p_bb(x,y) $$\leftarrow$$ d_p2_bb(x,y).                       (D2’)   (Step 3,    DT Step 3)
d_n.p_bb(x,z) $$\leftarrow$$ d_p2_bb(x,z).                       (D3’)   (Step 3,    DT Step 3)
d_p2_bb(y,z) $$\leftarrow$$ d_p2_bb(x,z), n.p(x,z), e2(x,y).     (D4’)   (Steps 1,3, DT Step 3)
d_p_bb(x,z) $$\leftarrow$$ d_n.p_bb(x,z).                        (DN1)   (Steps 2,3, DT Step 3)


Note the following observations:

• Except for (N1), the resulting rules contain no negation, because not p(args) has been replaced with n.p(args) in all other rules.

• The rule (DN1) is added in Step 3 of extended DT for the demand resulting from the second hypothesis of the rule (N1).

• The new predicate n.p is still in a cycle containing negation with p, so the rules are not stratified.

Next, we extend bottom-up evaluation to handle resulting rules that contain negation so that facts of new predicates n.p, and in turn facts of all predicates, can be inferred correctly.

### 4.2 Extended bottom-up evaluation

We extend Bottom-up so that facts for each predicate n.p are inferred and the rest of the evaluation proceeds as before with optimal time complexity.

We make the following observation that underlies the idea of the extension.

###### Lemma 1

Given a set of stratified rules and a query, let be the resulting rules after extended DT. After Bottom-up of , consider predicate p in the lowest stratum in among predicates such thatd_n.p_s(args) has been inferred: if p(args) has not been inferred, then the generated rule n.p(a,,a) d_n.p_s(a,,a), not p(a,,a). by extended DT can be used to infer n.p(args).

Proof.  Since p is in the lowest stratum among all predicates for which there is a negated demand, there is no negated hypothesis that p transitively depends on. If d_n.p_s(args) has been inferred, then because of a rule introduced in Step 3 of extended DT, d_p_s(args) has been inferred as well. Since p only depends on d_p_s and other positive hypotheses, p(args) must be inferred by Bottom-up if true, and not inferred otherwise. Therefore if it has not been inferred, it is false. Since n.p is the complement of p, n.p(args) is true.

Therefore, running Bottom-up without rules containing negation, after Bottom-up reaches a fixed point, new facts for a predicate n.p in the lowest stratum, for which the corresponding demand predicate d_n.p_s has a fact, can be inferred. After inferring facts for n.p, Bottom-up may be applied again, and reach a new fixed point. After the new fixed point is reached, if there are new facts for a demand predicate d_n.p_s, then n.p facts can be inferred again for such facts for the predicate in the lowest stratum, and so on. Precisely, Bottom-up after extended DT is extended as follows, and we refer to it as extended Bottom-up after extended DT.

Repeat the following steps until fixed point.

1. Perform Bottom-up without using rules that infer facts of predicates of the form n.p. Such rules are first added in Step 2 of extended DT and then transformed in Step 3 of extended DT, and are of the form n.p(a,,a) dn.ps(a,,a), not p(a,,a). where s is all ’s.

2. Use a rule to infer a new fact n.p(args) if dn.ps is in the lowest stratum among predicates of the form dn.qs for some predicate q for which (i) dn.ps(args) has been inferred and (ii) p(args) has not been inferred.

We extend Lemma 1 to show correctness at any iteration of the extended evaluation.

###### Lemma 2

Suppose d_n.p_s(args) is inferred during extended Bottom-up after extended DT. Ifp(args) is false for the original set of rules, then n.p(args) is inferred by extended Bottom-up after extended DT, and not inferred otherwise.

Proof.  (Case 1) p(args) is false for the original set of rules. Then, this fact cannot be inferred by the rules defining p in the rules obtained after extended DT since they are more restrictive than the original rules. Given that d_n.p_s(args) (say ) is inferred during extended Bottom-up after extended DT, it is guaranteed that if the rule for n.p is ever used, then n.p(args) will be inferred. The rule for n.p is guaranteed to be used since at some point in the iteration will be the fact in the lowest stratum in Step 2.

(Case 2) p(args) is true for the original set of rules. Since d_n.p_s(args) is inferred during extended Bottom-up after extended DT, then d_p_s(args) is inferred also due to the rule generated in Step 3 of extended DT. Hence, p(args) will be inferred during extended Bottom-up after extended DT since there is a demand for it, and all of the hypotheses that define it must be in a lower stratum, and hence correctly inferred also. Since p(args) is inferred, n.p(args) cannot be inferred.

###### Theorem 1 (Extended Bottom-up after extended DT matches Top-down)

A fact is inferred during extended Bottom-up of a set of stratified rules and query after extended DT iff it is inferred during Top-down of the rules and query.

Proof Sketch.  This relationship is already known for Bottom-up after DT and Top-down for rules without stratified negation, therefore we only need to extend the argument to handle negated hypotheses.

If a demand fact for a negated hypothesis is inferred during extended Bottom-up after extended DT, then all hypotheses to its left must be true, therefore a corresponding query must be evaluated during Top-down, and vice versa. So demand facts inferred correspond exactly to subqueries encountered during Top-down. By Lemma 2, if there is a demand for a negated hypothesis, then its corresponding fact is inferred correctly. Therefore, extended Bottom-up after extended DT corresponds exactly to Top-down in terms of inferred facts.

## 5 Precise complexity guarantees

We give precise time complexity analysis using the following parameters as in [DBLP:journals/toplas/LiuS09].

• #p: number of facts of predicate p, called size of p.

• #p.i,,i/j,,j: maximum number of combinations of values taken by the i,…,i-th arguments of facts of predicate p, given any fixed value of the j,…,j-th arguments.

There are two forms of rules after decomposition of rules. When a rule has one hypothesis, it is of the form: p(x,,x) q(y,,y). The number of times this rule can fire is the number of facts of q, therefore the time complexity incurred by this rule is . In fact, we can omit the complexity of such rules, because (i) if q is an extensional predicate, then all its facts need to be read in, therefore the complexity is already incurred by the reading of the input, (ii) if q is an intensional predicate, then its size is bound by the complexity of the rules that infer its facts.

When a rule has two hypotheses, it is of the form: p(args) q(x,,x), r(y,,y). To calculate the number of firings, we can first consider facts of q and find matching facts of r such that common variables in the two hypotheses take the same value. Therefore, only variables in the second hypothesis but not in the first can take different values for each fact of q. Let be the set of indices in such that y is a common variable in the two hypotheses, then the complexity is bounded by . Symmetrically, we can consider facts of r and find matching facts of q. Let be symmetrically the set of indices in such that x is a common variable, then the complexity is also bounded by . Both are upper bounds, so the complexity is bounded by the minimum of the two:

 O(\small min(\small\#q×\small\#r.i∉C_12/j∈C_12, \small\#r×\small\#q.i∉C_21/j∈C_21))

#### Example.

For the running example, rule (R1) incurs the time complexity , and rule (R2) incurs the time complexity .

### 5.1 Complexity characteristics of extended Bottom-up after extended DT

###### Theorem 2 (Extensions preserve complexity)

Extended Bottom-up after extended DT preserves the complexity characteristics and optimality of Bottom-up after DT.

Proof.  The only difference in extended Bottom-up is Step 2 where a demand fact for predicates of the form n.p in the lowest stratum is found. Such a fact can be found in constant time in data complexity, by searching for demand facts starting from the lowest stratum upwards. Therefore, the extension preserves the complexity characteristics and the optimality of Bottom-up after DT.

The precise time and space complexity of Top-down, and its relationship with Bottom-up after MST and DT, were open until [DBLP:conf/ppdp/TekleL10]. The results in [DBLP:conf/ppdp/TekleL10] include the following theorems—they establish that the time complexity of Top-down and that of Bottom-up after DT are equal for rules with at most two hypotheses each, and for general Datalog rules, Bottom-up after DT is equal to or faster than Top-down.

###### Theorem 3 (Bottom-up after DT equals Top-down on decomposed rules [DBLP:conf/ppdp/TekleL10])

Let be a set of Datalog rules and a query, such that the rules have no singleton variables, and there are no more than two hypotheses per rule. Let be the set of rules and fact after demand transformation of . Let be the asymptotic time complexity of Top-down of , and be the asymptotic time complexity of Bottom-up of . Then, .

###### Theorem 4 (Bottom-up after DT beats Top-down[DBLP:conf/ppdp/TekleL10])

Let be a set of Datalog rules and a query. Let be the set of rules and fact after demand transformation of . Let be the asymptotic time complexity of Top-down of , and be the asymptotic time complexity of Bottom-up of . Then, .

By Theorem 2, and the fact that Top-down behaves identically for stratified negation, these theorems can be extended as follows.

###### Corollary 1 (Extended Bottom-up after extended DT equals Top-down on decomposed rules)

Let be a set of stratified rules and a non-floundering query, such that the rules have no singleton variables, and there are no more than two hypotheses per rule. Let be the set of rules and fact after extended demand transformation of . Let be the asymptotic time complexity of Top-down of , and be the asymptotic time complexity of extended Bottom-up of . Then, .

###### Corollary 2 (Extended Bottom-up after extended DT beats Top-down)

Let be a set of stratified rules and a non-floundering query. Let be the set of rules and fact after extended demand transformation of . Let be the asymptotic time complexity of Top-down of , and be the asymptotic time complexity of extended Bottom-up of . Then, .

#### Example.

For the extended running example after extended DT, we calculate the time complexity by considering the most complex rule, copied below.

p2(x,z) $$\leftarrow$$ d_p2_bb(x,z), n.p(x,z), e2(x,y), p2(y,z).  (R4”)


Extended Bottom-up decomposes this rule into three rules:

i1(x,z) $$\leftarrow$$ d_p2_bb(x,z), n.p(x,z).  (R4”1)
i2(x,y,z) $$\leftarrow$$ i1(x,z), e2(x,y).      (R4”2)
p2(x,z) $$\leftarrow$$ i2(x,y,z), p2(y,z).      (R4”3)


Rules (R4”1) and (R4”3) have a hypothesis containing all variables occurring in the rule, so they do not contribute any extra complexity. The time complexity incurred by (R4”2) is

 O(\small min(\small\#i1×\small\#e2.2/1, \small\#e2×\small\#i1.1/2)).

Note that this complexity is precise, and can be very small because i1 is the intersection of d_p2_bb and n.p.

The complexity benefit of extended Bottom-up after extended DT is even more apparent when contrasted to evaluating the resulting rules bottom-up for well-founded semantics whose worst-case time complexity is where is the number of constants, whereas the worst-case time complexity using our method is .

## 6 Applications and experiments

### 6.1 Experiments on the extended running example

To confirm the effectiveness of our method, we implemented our method to perform extended DT, and then created a Python program that performs extended Bottom-up on the resulting rules. We compare our results with state-of-the-art systems: clingo [DBLP:journals/aicom/GebserKKOSS11], an ASP solver, DLV2, the latest version of a deductive database with ASP and dynamic magic sets [alviano2012magic, alviano2017asp], and XSB [DBLP:journals/jacm/ChenW96], the dominant engine with tabling and well-founded semantics.

We show experiments on the extended running example. For clingo, which does not support answering queries, we give it the transformed set of rules and facts. For XSB, we used a special construct, load_dynca to turn off indexing when loading facts, improving the performance more than 10 times. We use facts generated for predicates e and e2, by varying the number of nodes and edges. The running times were captured on an Intel Core i5 2.8 GHz with 8 GB RAM, with PyPy 7.1.0 to run Python programs, clingo 5.3.0, DLV2, and XSB 3.8, including the time for reading facts and producing output, and are averaged over 5 runs. The results are shown in Table 1, where K stands for 1000.

One can see that the running times for extended Bottom-up are tightly coupled with the number of edges as shown in the formula in Example 5.1. It is difficult to speculate on the time and space complexities for clingo and DLV2 because precise complexity analysis as shown for our method is not known. The strength of our method can be seen in contrast to these mature systems even though our Python program is not optimized for constant factors, and Python is known to be much slower than C and C++ used to implement XSB, DLV2, and Clingo.

### 6.2 Balbin’s not-reach-in-reach2 problem

Balbin et al. [DBLP:journals/jlp/BalbinPRM91, Example 12] give the following rules to show the challenge of non-stratified negation resulting from applying MST on stratified rules (some predicates/variables have been renamed for readability):

r(x) $$\leftarrow$$ s(x).                       (B1)
r(x) $$\leftarrow$$ e(x,y), r(y).               (B2)
r2(x) $$\leftarrow$$ s2(x).                     (B3)
r2(x) $$\leftarrow$$ not r(x), e2(x,y), r2(y).  (B4)


along with the query r2(1)?.

Using our method, the resulting rules from extended DT are:

r(x) $$\leftarrow$$ d_r_b(x), s(x).                      (B1’)
r(x) $$\leftarrow$$ d_r_b(x), e(x,y), r(y).              (B2’)
r2(x) $$\leftarrow$$ d_r2_b(x), s2(x).                   (B3’)
r2(x) $$\leftarrow$$ d_r2_b(x), n.r(x), e2(x,y), r2(y).  (B4’)
n.r(x) $$\leftarrow$$ d_n.r_b(x), not r(x).              (N1)
d_r2_b(1).                                   (DF)
d_r_b(y) $$\leftarrow$$ d_r_b(x), e(x,y).                (D1)
d_n.r_b(y) $$\leftarrow$$ d_r2_b(x).                     (D2)
d_r2_b(y) $$\leftarrow$$ d_r2_b(x), n.r(x), e2(x,y).     (D3)
d_r_b(x) $$\leftarrow$$ d_n.r_b(x).                      (D4)


These rules can be evaluated in linear time in the input size, because after decomposition into rules with two hypotheses, each rule has at least one hypothesis with extensional predicate s, e, s2, e2 that contains all variables in the rule. Contrast this to bottom-up evaluation for well-founded semantics, which would have only the bound of , where is the number of constants, since there is a rule with two variables.

### 6.3 Meskes-Noack’s no-extra-joins-in-path problem

Meskes and Noack [DBLP:journals/ipl/MeskesN93, Example 1] give the following rules to show that a negated predicate does not need to be in a positive cycle for non-stratified rules to occur after MST (some predicates/variables have been renamed for simplicity):

s(x) $$\leftarrow$$ q(x,z), r(z,y).              (M1)
p(x,y) $$\leftarrow$$ e(x,y), not s(y).          (M2)
p(x,z) $$\leftarrow$$ e(x,y), p(y,z), not s(y).  (M3)


along with the query p(1,y)?. It can be seen that the only negated predicate is s, and it is simply defined using extensional predicates, but after DT, non-stratified rules are obtained.

Extended DT yields the following rules:

s(x) $$\leftarrow$$ d_s_b(x), q(x,z), r(z,y).             (M1’)
p(x,y) $$\leftarrow$$ d_p_bf(x), e(x,y), n.s(y).          (M2’)
p(x,z) $$\leftarrow$$ d_p_bf(x), e(x,y), p(y,z), n.s(y).  (M3’)
n.s(x) $$\leftarrow$$ d_n.s_b(x), not s(x).               (N1)
d_p_bf(1).                                    (DF)
d_n.s_b(y) $$\leftarrow$$ e(x,y).                         (D1)
d_p_bf(y) $$\leftarrow$$ d_p_bf(x), e(x,y).               (D2)
d_n.s_b(y) $$\leftarrow$$ d_p_bf(x), e(x,y), p(y,z).      (D3)
d_s_b(x) $$\leftarrow$$ d_n.s_b(x).                       (D4)


(M3’) after decomposition gives the highest time complexity: where i is an intermediate predicate for the leftmost two hypothesis, and is a subset of e restricted by d_p_bf for the first argument. Like our original running example, the time complexity is bounded by using extended Bottom-up after extended DT versus for bottom-up evaluation for well-founded semantics.

## 7 Related work and conclusion

Datalog has been extensively studied, especially methods for efficient evaluation [Maier:2018:DCH:3191315.3191317, DBLP:books/aw/AbiteboulHV95].

Top-down evaluation with variant tabling was introduced in the 1980s [DBLP:conf/iclp/TamakiS86] and has been widely studied as well. It has been implemented in top-down evaluation engines such as XSB [DBLP:journals/jacm/ChenW96] and SWI [DBLP:journals/tplp/WielemakerSTL12]. Optimal bottom-up evaluation was given in [DBLP:journals/toplas/LiuS09]. Incremental maintenance of Datalog with stratified negation was studied in [motik2019maintenance].

Transformations for demand-driven bottom-up evaluation have been studied, in many forms of the well-known magic-set transformation (MST) [DBLP:conf/pods/BancilhonMSU86], as well as the more recent demand transformation (DT) [DBLP:conf/ppdp/TekleL10] that our work extends.

A method for precise calculation of time and space complexities of top-down evaluation with variant tabling for Datalog, as well as theorems establishing the precise relationship between top-down and bottom-up evaluations are given in [DBLP:conf/ppdp/TekleL10, DBLP:conf/sigmod/TekleL11].

Although top-down evaluation handles stratified negation without changes, an analogous demand-driven bottom-up evaluation has been challenging to obtain.

Balbin et al. [DBLP:journals/jlp/BalbinPRM91] introduce a transformation to a set of magic-set transformed rules, that works for an unidentified class of rules, and does not apply to all rules with stratified negation. They then introduce a new method for bottom-up evaluation with no complexity characterization.

Meskes and Noack [DBLP:journals/ipl/MeskesN93] extend generalized supplementary MST [DBLP:journals/jlp/BeeriR91] by ignoring negated hypotheses for rules that infer demand facts to keep the rules stratified, but the method overestimates the demand and hence does not correspond to top-down evaluation.

Ross [DBLP:journals/jacm/Ross94] presents a new transformation whose output is not Datalog, but a higher-order logic, complicating their evaluation and giving no complexity analysis.

Alviano et al. [alviano2012magic] extends magic sets for disjunctive Datalog, but its bottom-up computation requires grounding and stable model search, not giving precise complexity guarantees as we do.

It has been shown that the well-founded semantics of rules obtained by MST for stratified rules is two-valued [DBLP:journals/tcs/KempSS95], but the best known complexity for bottom-up evaluation for well-founded semantics is a highly prohibitive quadratic time in the size of the grounded rules.

Our work extends demand transformation and optimal bottom-up evaluation and gives precise complexity guarantees for efficient demand-driven bottom-up evaluation of stratified rules, preserving optimality, and relationships with top-down evaluation.

Future work includes methods for efficient bottom-up evaluation and demand-driven evaluation for non-stratified negation with precise complexity guarantees.

#### Acknowledgments.

This work was supported in part by NSF under grants CCF-1414078 and IIS-1447549.

## References

• [1] V. Bos and S. Mauw (2002-06) A LaTeX macro package for message sequence charts—maintenance document—describing version . Note: Included in MSC macro package distribution
• [2] V. Bos and S. Mauw (2002-06) A LaTeX macro package for message sequence charts—reference manual—describing version . Note: Included in MSC macro package distribution
• [3] V. Bos and S. Mauw (2002-06) A LaTeX macro package for message sequence charts—user manual—describing version . Note: Included in MSC macro package distribution
• [4] M. Goossens, S. Rahtz, and F. Mittelbach (1997) The LaTeX Graphics Companion. Addison-Wesley.
• [5] ITU-TS (1997) ITU-TS Recommendation Z.120: Message Sequence Chart (MSC). Geneva.
• [6] L. Lamport (1994) LaTeX—a document preparation system—user’s guide and reference manual. 2nd edition, Adsison-Wesley. Note: Updated for LaTeX2e
• [7] E. Rudolph, P.Graubmann, and J. Grabowski (1996) Tutorial on message sequence charts (MSC’96). In FORTE,