The Problem of Correlation and Substitution in SPARQL -- Extended Version

01/13/2018 ∙ by Daniel Hernández, et al. ∙ 0

Implementations of a standard language are expected to give same outputs to identical queries. In this paper we study why different implementations of SPARQL (Fuseki, Virtuoso, Blazegraph and rdf4j) behave differently when evaluating queries with correlated variables. We show that at the core of this problem lies the historically troubling notion of logical substitution. We present a formal framework to study this issue based on Datalog that besides clarifying the problem, gives a solid base to define and implement nesting.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

A subquery is a query expression that occurs in the body of another query expression, called the outer query. A correlated subquery is one whose evaluation is dependent in some way on data being processed in the outer query. Informally the data got from the outer query should be replaced or substituted in the corresponding places in the inner query. Thus the notion of substitution comes to the heart of the problem at hand.

It is well known the complexities that this notion involves. As is well known, it has been always a troubling concept and source of error even to renowned logicians.111The story is recounted by Church in [5], pp. 289-90 and Cardone and Hindley [4], p.7. Russell and Whitehead, although used the notion, missed its formal statement in his Principia (1913). Hilbert and Ackermann gave an “inadequate” statement in his 1928’s Logic. Carnap in Logicshe Syntax der Sprache and Quine in System of Logistic definitions still contain problems. Hilbert and Ackermann in 1934 gave finally a correct statement. The query language SPARQL is not an exception: the notion of replacement (substitution) in the recommendation has an insufficient definition and even contradictory pieces 222See [11] where we report that the substitution notion presented on Sec. 18.6 of the current specification [10] contradicts the statements “Due to the bottom-up nature of SPARQL query evaluation, the subqueries are evaluated logically first, and the results are projected up to the outer query” and “Note that only variables projected out of the subquery will be visible, or in scope, to the outer query”, from Sec. 12..

We show in this paper that this notion is the source of problems that can be found in the evaluation of EXIST subqueries in SPARQL. The problem is highlighted when operators which incorporate possibly incomplete information are present. The following SPARQL query illustrates these problems. The query roughly asks for the id of persons and optionally their corporate email, subject to some conditions given by the expression in the FILTER EXISTS:

SELECT ?id ?email
WHERE { {{ ?id a :person } OPTIONAL { ?id :corpMail ?email }}
            {{ ?id a :person } OPTIONAL { ?id :privMail ?email }}
            FILTER (?email = *.com) } }

Surely the reader is facing the following problem: how to interpret this query? Well, you are not alone in your vacillation: The most popular implementations of SPARQL do not agree on it. For example, for the following database of persons

id 1 2 3 4 5 6
corpMail *.com *.net *.com *.net
privMail *.net *.com *.com *.net

Fuseki, Blazegraph, Virtuoso and rdf4j give almost all different results. Fuseki and Blazegraph give (1,*.com), (3,*.com) and (5,-); Virtuoso gives (1,*.com) and (3,*.com); finally rdf4j gives (3,*.com) and (5,-).333The engines studied in this paper are Fuseki 2.5.0, Blazegraph 2.1.1, Virtuoso, and rdf4j 2.2.1.

What is going on? Not simple to unravel. The main problem is how to assign the variables in order to evaluate the inner and outer expressions. Let us see why not all systems agree in showing up person 1. If we evaluate first the inner query then the variable ?email is bound to *.net for person 1, thus the filter ?email = *.com fails, and so the whole expression inside the first filter fails, hence person 1 is not shown in the output. Now, if the inner pattern is evaluated after bounding the variable ?email to *.com, then the OPTIONAL part does not match and thus the last filter pass, hence person 1 is outputted.

The intuition provided by the case of person 1 is that some systems evaluate the inner pattern before binding the ?email in the outer query, and others do it after the binding. Now this intuitive philosophy does not work to understand why Blazegraph and Fuseki outputs, (1,*.com) and (5,-). In defense of these systems, let us recall that the specification is not clear or precise enough about this evaluation.

We —the Semantic Web advocates— are in a problem: No query language with such uncertainties will gain wide adoption. Of course the example is highly unnatural but, as we will show, it codes the essence of the problems of substitution of correlated variables in SPARQL. Substituting is not trivial even when all values are constants, but when considering incomplete information, e.g. in the form of nulls or unbounds in SPARQL, the complexities rapidly scale up. In this setting, having a clean logical picture of what is going on is crucial.

Our aim is that the above problems can be modeled by using Datalog. Hence, we introduce Nested Datalog, an extension of classical Datalog to cope with nested expressions. Nested Datalog is used to describe two different substitution philosophies: syntactic substitution, which implies that the value to be substituted in a correlated variable comes from one source; and logical substitution, which implies that a variable is expecting values from two sources (from the valuation of the expression in the database and from the outer query).

The problem becomes more complex in the logical evaluation when one (or both) of the values is null. It turns out that it is not indifferent where we “compatibilize” both values (at the bottom of the program, at the top, in the middle). To model this it is needed one step further in the formalization in order to capture the nuances of incomplete information in the form of null values. To do this, we use Modal Datalog, a version of Datalog with modal features.

Once having the right setting, the subtleties of the notion of substitution in nested expressions having incomplete information (null values) becomes visible. Then, in Section 5, we show how SPARQL translates to this setting, and how the notion of substitution is expressed in it. Then we show the discrepancies of the different implementations and what they mean under the light of this formal framework. We conclude with our view of how to handle this problem.

In summary, the contributions of this paper are the following:

  1. Formalization of the problem (i.e. substitution in nested expressions), its study from a logical point of view, and analysis of the discrepancies of evaluation under different implementations of SPARQL.

  2. To provide a logical framework to understand and formalize the notion of substitution in nested expressions in the presence of incomplete information for SPARQL.

  3. Presentation of the logical (and consistent) alternatives defined and supported by our logical framework.

Related work.

Different problems related to the notion of substitution are listed in the SPARQL specification errata. In a previous work [11] we have reported some issues and presented three alternative solutions based on rewriting on the nested query before the substitution. After that, a W3C community group444 was created to address these issues. The community started defining queries and their expected outputs, for two alternative semantics. The first, proposed by Patel-Schneider and Martin [16], and the second proposed by Seaborne in the mailing list of the community group. None of these proposals study the problem in a formal framework as we do in this paper.

The idea of nested queries in Datalog is not new (see [9] for example). We introduce nesting in a different way in order to be more suitable for studying correlation. Similarly, null values have been already studied in deductive databases (e.g., [7, 8, 12]), but with a focus (on computing certain answers) that is not the goal of this paper.

2 Standard Datalog

We will briefly review notions of non recursive Datalog with equalities and safe negation. For further details see [15].

Datalog Syntax.

A term is either a variable or a constant. An atom is either a predicate formula where is a predicate name and each is a term, or an equality formula where and are terms. A literal is either an atom (a positive literal ) or the negation of an atom (a negative literal ). For readability we also write instead of . A rule is an expression of the form where is a predicate formula called the head of , and is a set of literals called the body of . A fact is a predicate formula with only constants. A program is a finite set of rules and facts. A query is a pair where is a predicate formula called the goal of , and is a program. We assume that all predicate formulas in a query with the same predicate name have the same number of arguments and that the terms in the head of each rule in and the goal of are all variables.

Semantics of Datalog.

Given a substitution from variables to constants, and a literal , denotes the literal resulting of substituting in each occurrence of by , for . Given a set of facts , a substitution , and a positive literal , we say that models with respect to , denoted , if either is an equality formula of the form where is a constant, or is predicate formula and . Similarly, given a set of facts , a substitution and a negative literal , , if . Given a rule , a fact is inferred in a set of facts if , for .

A variable occurs positively in a rule if occurs in a positive predicate formula in the body of . A rule is said to be safe if all its variables occur positively in . A program is safe if all its rules are safe. The safety restriction provides a syntactic restriction of programs which enforces the finiteness of derived predicates.

In this paper we do not consider equality formulas as in most works (e.g., [15]), but equality formulas of the form . They differ in the form of evaluation. If a rule has the literal in the body, then is said to be defined positively because the equality assigns the value to . On the contrary, here equality formulas require that all variables be assigned before being evaluated. Formally, here equality formulas are built-in.

The dependency graph of a Datalog program is a digraph where the set of nodes is the set of predicates that occur in the literals of , and there is an arc in for each rule in whose body contains predicate and whose head contains predicate . A Datalog program is said to be recursive if its dependency graph is cyclic, otherwise it is said to be non-recursive. On this report we consider only non-recursive programs.

Let be a program, be a subset of the Herbrand base of , and denote the set of facts occurring in or , intuitively the facts inferred in zero steps. Then, the meaning of is the result of adding to as many new facts as can be inferred from the rules of in . The inference process is applied repeatedly until a fixpoint, denoted , is reached. The answer of a query in an extensional database , denoted , is the subset of with facts having the same predicate as .

The fixpoint depends on the order used to evaluate rules. Here we assume the order of Stratified Datalog, where for every arc in the dependency graph of a program , a rule with head is not used in the inference process until every rule with head cannot be applied to infer another fact. Without this order a negative predicate formula can be wrongly evaluated as true if the evaluation is done before a fact matching is inferred.

3 Nested Datalog

We will extend Datalog in order to be able to compose queries, by introducing nested queries as a new type of atom that occurs as a filter device (i.e., a built-in).

Definition 1 (Syntax of Nested Datalog defined recursively)

1. A Datalog query is a Nested Datalog query. 2. A Nested Datalog query is a Datalog query where Nested Datalog queries are allowed as atoms.

The inference process of Nested Datalog differs from the standard one by the addition of the semantics for evaluating Nested queries. If query is an atom in a rule, its evaluation with respect to the substitution is true if and only if has at least one answer, where denotes the query resulting of “applying” to the substitution . This is the key notion we will study in what follows.

We will need the notion of assignment of a value to a variable. Consider a program with a single rule defined as . The assignment of the value to the variable can be done by adding a literal to , where is a fresh predicate, and the fact to . In the resulting program can only take the value . In what follows we will use the notation as a syntactical sugar to denote the result of assigning to in .

Substitution for a nested query turns out to be rather subtle. There are two main approaches that we will call syntactical and logical.

Syntactical substitution.

It works like standard replacement of a variable in an open sentence in logic or a free variable in a programming language. It occurs when a rule cannot be evaluated without having the value of a “free” variable occurring in it. Formally, a variable occurs free in a rule when it does not occur positively in and it occurs in an equality formula or in a nested query where recursively occurs free in a rule of .

Definition 2 (syntactic substitution)

Given a substitution , a program and a variable occurring in , the syntactical substitution of in with respect to is done by adding the literal to the body of each rule of where is free.

Logical substitution.

This is the problem of “substituting” in a program that has no “free” (i.e. all their rules are “logically closed”). Conceptually, in this case the semantics is one such that after finding a solution of , it checks its “compatibility” with .

The essential problem of logical substitution is “when” (at what point in the evaluation) we will test this compatibility. For example, consider the program with the rules and . The variable in the second rule is logically connected with the variable in the first rule. Thus, we can alternatively do the substitution in two places:

Top-down Bottom up

This example illustrates two extremes, (1) Top down: evaluate first, then, proceed to check the compatibility of the solution with . (2) Bottom up: Check compatibility of (with the database) before starting the evaluation of . However, there are also several valid substitutions in the middle. In fact, we can start with a top-down substitution and then move the literals down in the dependency graph of the program. The method used to move substitutions is equivalent to the standard method used to move selections in Relational Algebra because of optimization concerns. In the appendix we provide a detailed description of the process of moving substitutions. The following result follows from it.

Lemma 1

Given a substitution and a Nested Datalog program , moving down literals of the form in the dependency graph of does not change the semantics of .

Hitherto, we have defined logical substitution for a rule but not for a whole query. We now will define it using the top-down approach.555Other approaches can be obtained by moving the substitution point from upper levels to lower levels in the dependency graph of the query.

Definition 3 (Top-down logical substitution)

Given a substitution , a query and a variable occurring in the goal of and , the top-down logical substitution of in is the query resulting of replacing the goal of by , where is a fresh predicate, and adding the rule to .

Logical and syntactical substitutions are not arbitrary. They are motivated by the EXISTS operator of SPARQL as is shown in the following query.

WHERE { ?X :hasMail ?Y
                        WHERE { ?X :hasMail ?Z FILTER (?Y <> ?Z) } } }

Intuitively this query finds people ?X with multiple emails. The variable ?X cannot be substituted by a constant in the nested query, because that substitution breaks the syntax of the SELECT clause (where only variables are allowed). Thus, ?X has to be substituted logically. On the other hand, ?Y is a free variable in the nested query, so ?Y has to be substituted syntactically. We claim that these substitutions are better understood if we rewrite this SPARQL query as the Nested Datalog query where is the following program:

Hence, the result of applying a substitution in produces a program with the rule .

There is a third form of substitution, namely improper. To gain some intuition, consider the following SPARQL query:


The inner pattern of this query can be modeled as a query where contains the rule . The variable cannot be substituted logically, because it does not occur in the goal nor in the head of the unique rule of . Furthermore, cannot be substituted syntactically, because it is not free. However, some systems assume that is correlated. The behavior of these systems coincides with applying improper substitution. Here, substitution is done by replacing the unique rule of by the rules and , where is a fresh predicate.

Essentially, an improper substitution is a logical substitution that starts in some point of the nested query instead of the goal of the query, so that there is a gap in the logical chain. After that point, substitution is similar to logical substitution in the sense that it moves the value assigned to from the head of a rule to the body, and thus to other rules below in the dependency graph.

In the example, the substitution is improper, because the logical chain start on the second rule so there is a gap between the goal of the nested query and the point where the logical chain starts.

Improper substitution breaks the design of Datalog where the scope of variables is the rule where they occur. It disallows renaming variables because its scope could be extended beyond the nested query, so breaking the design of logic and compositional languages where non free variables are renamed without changing the semantics of the expression, and free variables cannot be renamed. In our opinion, this compromises the compositionality of queries.

Now we have a parametrical definition of for a query (i.e., without the values of a particular substitution ), independently of the kind of substitution used (syntactical, logical or improper) substitutions are used. A consequence, is the following result.

Lemma 2

Given a Nested Datalog query and a simple extensional database for , there exists a first order formula such that each .


It is well known that without nesting each Datalog query corresponds to a such formula . In fact, each literal is translated into a first order literal . Each rule is translated as a first order formula where are the non free variables of the rule. And the whole program is translated as the conjunction of the formulas corresponding to each rule.

The proof is done by translating each the atom that we added to Datalog into first order formulas, i.e., Nested queries. Suffices giving a recursive translation for them. We define it as follows. The formula of a nested query is the first order formula , where and is the formula of after renaming each predicate in with a fresh predicate (this ensures that the evaluation is isolated from the outside of the nested query), and are the variables that occur positively in the rule where is nested. The equalities model the logical substitution. Also, free variables in rules of , are not added in quantifiers of rules of , so they are syntactically substituted. Improper substitution of a variable in a rule can be simulated by removing the in the formula of . ∎

A corollary of this lemma is that nesting does not add expressive power to Datalog. Hence, if we chose using one form of substitution but not the other (e.g., using logical substitution but not syntactical), we will have the same expressive power than if we had chosen the contrary, or no substitution.

4 Modal Datalog

Modal Datalog is a version of Datalog where each rule is labeled with a modal logical operator. In what follows we develop it.

Most real world information includes incomplete data. The main device to codify incompleteness has been the null values. A null, denoted , represents either that the value is missing or non applicable. A relation containing null values is said incomplete, while one without them is said complete. The semantics of an incomplete relation is the set of all possible complete relations resulting from replacing consequently each null by a constant or a symbol , denoting a non-applicable value. We follow the semantics of and by Lerat and Lipski [14]. However, in this paper we only consider nulls, because the evaluation process of Modal Datalog never generates non applicable values if they are not present in the database.

The answer of a query in a complete database is characterized by the set where is a first order formula whose free variables are instantiated by . On the other hand, an incomplete database is interpreted as a set of complete databases , the possible worlds. In this context, we can give modal characterizations to an answer : is sure, denoted , if for all . Similarly, is a maybe answer, denoted , if there exists a database such that .

Modal Datalog essentially introduces a mode for each rule in a Datalog program. If the mode of is , then is said to be sure and infers facts that are valid for all instances of the null values occurring in in the current database. Otherwise, if the mode of is , then is said to be a maybe rule and infers facts that are valid for at least one instance of the null values.

Definition 4 (Modal Datalog Syntax)

A Modal Datalog Program is a set of rules of the form where is either or , where the symbol can occur in the body of the rule in the same places than terms.

A modal Datalog query is one built with Modal Datalog programs.

Next we present how modal predicates are derived from sets of facts and substitutions. We write if is derived from and with the label sure or maybe. We say that a Modal Datalog predicate formula is less informative than another , denoted , if every instance of is an instance of . Let be a literal, be a set of facts, and be a substitution. Then:

  • if one of the following conditions holds:

    • is a positive predicate formula and .

    • is a negative predicate formula and there does not exist a fact in such that , for .

    • is and is .

    • is or and

    • is and has at least one answer.

    • is and has no answers.

  • if one of the following conditions holds:

    • is a positive predicate and there exists a fact such that .

    • is a negative predicate formula and there does not exist a fact in such that , for .

    • is and is .

    • is or and .

    • is and has at least one answer.

    • is and has no answers.

Definition 5 (Semantics of Modal Datalog)

Given a Modal Datalog query , a database , and a set of already inferred facts , a fact is inferred from a rule in ( is either or ) if and only if there exists a substitution such that for all literals in , is the less informative substitution such that for all positive predicate formulas in , and .

Example 1

Let be a set of facts and be the query where the program has the rules and . Then, let us evaluate . The literal in is only true with the substitution , because is the only available fact. To infer we need first to infer from rule . Despite rule finds possible answers, inferring is not possible because infers the less informative fact, i.e., . Hence, is empty.

Now we are ready to define the promised notion of substitution in Modal Datalog. As before, we will make a distinction between syntactical substitution and logical substitution.

In the case of syntactical substitution, the approach is the same. We add a literal to the rule where occurs free. As this literal is syntactic sugar for introducing a predicate formula , then is indeed a positive predicate formula, so occurs positively in it (see Def. 5). Then, is not free anymore, and takes the value of , that can be a null or a constant.

The logical substitution is more subtle. As we saw, in logical substitutions the value of a variable comes from more than one source. For example, if one simply adds the literal to the body of the rule ( is either or ), then the value of will be provided by two sources, namely and the literal . The problem is what to do if in one source is null and in the other a constant. We solve this problem by splitting the rule in two, so one preserves the mode of the original rule and in the other we use the mode to merge the values coming from both sources.

The logical substitution of in a rule is the replacement of by the rules and , where is a fresh predicate. The mode in the first rule checks the compatibility and follows the SPARQL design, where a null value is considered compatible with a constant.

We showed that in Nested Datalog logical substitutions can be moved down in the dependency graph of a program without changing its results (Lemma 1). The following Lemma states that this feature does not hold in Modal Datalog.

Lemma 3

Given a substitution and a Modal Datalog program , moving down literals of the form in the dependency graph of could change the semantics of .


It suffices to show an example that witnesses this. Consider the query where is the unique rule .

Program A Program B

In a database containing the facts and , and , program A will have no solutions and program B will have the solution . ∎

The example in the previous proof shows:

Corollary 1

In Modal Datalog bottom-up and top-down evaluations do not behave in the same manner.

5 Substitution in FILTER EXISTS expressions

By using the machinery of Modal Datalog, we will model the evaluation of FILTER EXISTS expressions in SPARQL. Our objective is to provide a framework for safe semantics for correlated subqueries.

5.1 SPARQL codified as Modal Datalog

First we show that SPARQL can be coded in Modal Datalog using a translation inspired by [1], that is, via relational algebra. The syntax and semantics of the SPARQL fragment studied here are defined using Relational Algebra with set semantics (as is done in [6] and [2]).

We write to denote a relation with schema (attributes) , that is, a set of mappings from to constants or null values. We extend standard relational algebra to handle null values by using modal evaluation when needed:

where are two relations; is the truncation of the tuple to the set of attributes ; is the renaming of the attribute by ; is the concatenation of tuples, where takes the most informative value for each common attribute , or the available value if is not common; and is the relation with a single tuple filled with null values.

Here we study the fragment of SPARQL composed by the operators , , , , , . Note that we have selected one mode for each modal operator. The difference corresponds to the operator MINUS in [17], referred as DIFF in [3].666The standard MINUS is slightly different in the case when the subtracting mapping has no attributes, but both can be mutually simulated (see [13, 3]). The fragment where the operands of have the same attributes precludes the emergence of nulls when evaluating databases without nulls, thus coincides with Relational Algebra. Otherwise, the extended algebra is required. In this context, the operator OPTIONAL, denoted here as , is defined in [17] as .777This definition of is slightly different with such stated by the standard. However, it is well known that the standard SPARQL operators are definable in the algebra presented here (e.g., see [13, 3]).

Definition 6 (From algebra to Modal Datalog)

Given two relations and , the Modal Datalog rules for each algebraic operator are defined as follows:

The translation of each algebraic operator into a set of Modal Datalog rules by Def. 6 allows translating algebraic expressions into Modal Datalog queries.

Example 2

Given the relations , and , the expression is translated as the query where the program has the rules , , and .

Lemma 4

Let be an algebra expression and be the Modal Datalog query obtained from according to Def. 6. For every database , it holds that and are equivalent, i.e. their evaluation results in the same answers.888Note that answers of have the form while answers of have the form . In this lemma we assume that these answers are the same as they have the same components.

5.2 Modal Datalog semantics of FILTER EXISTS

Hitherto, we have a semantics for a SPARQL fragment and a translation to Modal Datalog, except for expressions of the forms and , where and are called respectively the outer and the inner patterns.999In the standard syntax these operators correspond to ( FILTER EXISTS ) and ( FILTER NOT EXISTS ), respectively.

The philosophy of these operators is the following. Given a relation and an algebraic expression , we have that and return the set of tuples where has a solution or no solutions, respectively. According to the SPARQL specification, is the result of replacing in each variable in the domain of by . As we indicated in the introduction, this definition is ambiguous and contradictory with other parts of the specification, and (as expected) systems have different interpretations for it.

We will unveil this problem by showing how the definition of is viewed in Modal Datalog where the notion of substitution shows up in a clean logical manner.

Definition 7 (Filter Exists)

Given a relation and a SPARQL query , the expressions and are translated to the Modal Datalog rules and respectively.

Now we are ready to enumerate three sources of discrepancy in the interpretation of a rule :101010Due to space limitations, here we consider only the positive case . It is not difficult to extend the results for the negative case .

  1. Free variables. A free variable in is in some cases assumed uncorrelated. The lack of correlation of a variable is simulated by replacing by the rule , where is a fresh variable and is the result of replacing each occurrence of in by .

  2. Improper substitution. Some engines implement improper substitution.

  3. Substitution level. A variable in the goal of the nested query can be substituted logically in different places, ranging from top-down to bottom-up logical substitution. These substitutions are not equivalent (see Lemma 3).

Next we present example queries to show the ways the systems address the above sources of discrepancy. We will use the dataset presented in the introduction. We assume that is a pattern giving all persons in the database, and and are patterns returning respectively the corporate and personal emails of a person .

The first two discrepancies.

Consider the queries , and , where is . These queries can be simulated with the same Modal Datalog query having goal and a single rule , where is either , , or . Now, we have the following cases:

  • Case : It has a unique alternative which is substituting logically, so returning only person 1. All systems agree with this answer.

  • Case : It has two interpretations, namely allowing or not allowing improper substitution. In the first, answers include only person 1. In the second answers are persons 1 and 3. Blazegraph and Fuseki agree with the first interpretation, while rdf4j and Virtuoso with the second.

  • Case : It has two interpretations, depending whether is assumed correlated or not. In the first interpretation, person 1 is the unique answer. In the second interpretation, there are no answers because is when evaluating . Blazegraph, rdf4j and Virtuoso agree with the first interpretation, while Fuseki with the second.

Level 1

Level 2

Level 3








Figure 1: Dependency graph of the Nested Datalog query for the inner pattern .

The last discrepancy.

To check the application of logical substitution, consider the query where is , corresponding to the one presented in the introduction. Fig. 1 depicts the dependency graph of . Some edges are labeled to refer alternative places where a substitution can be applied. For instance, the top-down approach consists in inserting the rule in the edge 1, and replacing in the rule below by , where is a fresh predicate. Hence, only person 5 succeeds. No system agrees with this evaluation.

If substitutions are done in Level 2, then persons 3 and 5 succeed. Only rdf4j agrees with this interpretation.

Blazegraph and Fuseki agree with substitution in Level 3, i.e., just after variables instantiation. Strictly, this approach is not logical as it has some improper substitutions. Indeed, the logical chain of variable below the edge 7 does not start in the goal of the nested query. Here persons 1, 3 and 5 succeed.

Virtuoso, Blazegraph and Fuseki follow a similar approach but they also append the literals and to the rule below edge 1. Hence, person 5 is discarded by the filter on this rule.

6 Conclusions

This work shows that the notion of substitution continues to haunt researchers and developers. We showed that it is at the core of the subtle problems that the specification and implementations of SPARQL face regarding subqueries of the form FILTER EXISTS.

We think the lasting contribution of this paper is the finding of several types of substitution in the presence of nested expression and incomplete information, that are playing some roles and seems that had passed unnoticed until now.

Although we consciously did not advance any proposal to fix the problems of substitution in nested expressions in SPARQL, the paper leaves a chart with the possible avenues to solve them. We think that the Working Groups of the W3C have the last word on this issue.


  • [1] R. Angles and C. Gutierrez. The Expressive Power of SPARQL. In Proc. of the International Semantic Web Conference (ISWC), volume 5318 of LNCS, pages 114–129. Springer, 2008.
  • [2] R. Angles and C. Gutierrez. The Multiset Semantics of SPARQL Patterns. In Proc. of the International Semantic Web Conference (ISWC), volume 9981 of LNCS, pages 20–36. Springer, 2016.
  • [3] R. Angles and C. Gutierrez. Negation in SPARQL. In Proc. of Alberto Mendelzon International Workshop on Foundations of Data Management (AMW), volume 1644 of CEUR Workshop Proceedings, 2016.
  • [4] F. Cardone and J. R. Hindley. History of lambda-calculus and combinatory logic. Handbook of the History of Logic, 5:723–817, 2006.
  • [5] A. Church. Introduction to mathematical logic, volume 13. Princeton University Press, 1996.
  • [6] R. Cyganiak. A Relational Algebra for SPARQL. Digital Media Systems Laboratory HP Laboratories Bristol. HPL-2005-170, page 35, 2005.
  • [7]

    F. Dong and L. V. S. Lakshmanan. Deductive databases with incomplete infor- mation. In Proc. of the Joint International Conference and Symposium on Logic Programming (JICSLP), pages 303–317. MIT Press, 1992.

  • [8] F. Dong and L. V. S. Lakshmanan. Intuitionistic Interpretation of Deductive Databases with Incomplete Information. Theor. Comput. Sci., 133(2):267–306, 1994.
  • [9] L. Giordano and A. Martelli. Structuring logic programs: A modal approach. J. Log. Program., 21(2):59–94, 1994.
  • [10] S. Harris and A. Seaborne. SPARQL 1.1 Query Language - W3C Recommendation, March 21 2013.
  • [11] D. Hernandez, C. Gutierrez, and R. Angles. Correlation and substitution in SPARQL. CoRR, abs/1606.01441, 2016.
  • [12] Q. Kong and G. Chen. On Deductive Database with Incomplete Information. ACM Trans. Inf. Syst., 13(3):355–369, 1995.
  • [13] R. Kontchakov and E. V. Kostylev. On expressibility of non-monotone operators in SPARQL. In Proc. of the International Conference on Principles of Knowledge Representation and Reasoning (KR), pages 369–379. AAAI Press, 2016.
  • [14] N. Lerat and W. Lipski Jr. Nonapplicable Nulls. Theor. Comput. Sci., 46(3):67–82, 1986.
  • [15] M. Levene and G. Loizou. A guided tour of relational databases and beyond. Springer, 1999.
  • [16] P. F. Patel-Schneider and D. Martin. Existstential aspects of SPARQL. In Proc. of the International Semantic Web Conference (ISWC), Posters & Demonstrations Track, 2016.
  • [17] J. Perez, M. Arenas, and C. Gutierrez. Semantics and Complexity of SPARQL. ACM Trans. Database Syst., 34(3):16:1–16:45, 2009.

Appendix 0.A Moving logical substitutions to lower levels

In this appendix we will describe how a substitution literal can be moved to lower levels of the dependency graph without changing the semantics of a Nested Datalog query. Because Nested Datalog programs are non recursive, the structure of the dependency graph of is a tree, so substitution literals can be moved down repeatedly until the leaf predicates are reached.

We need another structure. Given a program , then we call the tree of to the rooted tree whose nodes are labeled with the rules of and where a node labeled rule is the child of a node labeled by a rule if the head of is in the body of .

Now, we will describe how to move down an literal in a rule of a program where this literal occurs. Without loss of generality, consider that is the following rule:

where are positive predicates formulas with intensional predicates where occurs and are the rest of the literals in the body of .

For , let be and be the set of positions of such that (i.e., are the same variable). Then, rename in with a fresh predicate name and do the following for each rule in of the form :

  1. Let be a copy of the thread whose root is in the tree of , and the result of renaming consequently all intentional predicates occurring in by a fresh predicate, except that is renamed as .

  2. Append the literal to the body of the root of , for each position in .

  3. Copy the rules of into .

Example 3

Consider the following program:

where is an EDB-predicate. Then, moving the literal one level below results in the following program:

Definition 8 (Bottom-up logical substitution)

Let be a substitution and be a Nested Datalog query. Then the bottom-up substitution is the query resulting after applying the top-down substitution of in and then moving down the added literals until they reached the leafs of the tree of .

Is not difficult to see that the process of moving a literal down once level results in an equivalent program. Hence, we got the following result.

Lemma 5

Let be a Nested Datalog query. Let and be the respective evaluation procedures using top-down and bottom up substitution. Then, for every extensional database for holds .

Appendix 0.B Comments on ESWC 2018 review

This document is an extended version of a paper submitted to ESWC 2018. In this appendix, we will present some answers to questions formulated by the reviewers. Some of them could be useful to understand the notions defined here. We are currently working to improve the readability of the paper. When we achieve the desired clarity, we will remove this appendix.

It is important to recall that the first motivation of our work is that the evaluation of correlated (sub)queries differs among the current SPARQL engines (we presented examples of this fact). Second, the origin of such differences comes from an ambiguous semantics defined by the SPARQL 1.1 specification. Third, there is no standard foundation or reference to explain the notion of correlated queries (even for well-known studied languages like SQL, relational algebra, and Datalog). Therefore, our goal was to provide a logical framework to study possible interpretations and semantics for correlated queries, and to understand the implementations of actual engines.

The importance of a logical foundation for query languages was noticed by Reiter. He presented a variety of arguments in favor of doing so. One of these arguments was that an evaluation argument could be proved to be sound and complete with respect to the logical semantics of the data model and the query. In this work we follow the motivations of Reiter when providing a logical framework to understand the notion of correlation in nested queries.

Answers to questions.

Reviewer 3 is right: we missed the paper of Kaminski et al. (ACM 2017), and it should be incorporated into the related work. We reviewed the WWW’2016 version, but it does not mention the issues of substitution. On the other side, the ACM’2017 version differs from our work as it proposes a concrete correlation method. In our case, we aim to provide a logical framework to try to understand alternative semantics. For example, according to their proposal, the result of the example query in the introduction of our paper is . This result is different to all solutions provided by the studied engines.

Reviewer 1: Concerning the modes of the operators join and minus. Let us explain the case of the join (the case of the minus is similar). Given two relations and , the SPARQL operation corresponds to the set of concatenations of compatible tuples in and in . That is, and holds that , is unbound or is unbound, for each variable (Here we represent unbound values with the symbol .) This is equivalent to say that there exists an instance of the tuples and nu such that (that we wrote in modal notation). A tuple is usually called an instance (or possible world as we mentioned previously in the paper) of . The same applies to literals (reviewer 4 asked this). Reviewer 3 is also confused with the “magic” behavior of in the modal evaluation of equalities. The is false because there is a instance (e.g., ) where the literal is not true. On the other hand, is true because there is a instance (e.g., ) where the literal is true.

In several parts of the document we use the term “built-in”. It seems that this notion is not clear to some reviewers. The built-in si a formal notion in Datalog, e.g. see the survey of Ceri, Gottlob, and Tanca, “What You Always Wanted to Known About Datalog (And Never Dared to Ask).” A built-in is a literal that occurs in the body of a rule and is evaluated after instantiating all variables occurring positively in that rule. In this work nested queries and equality formulas are built-ins.

Reviewer 2: Concerning the analysis of the two semantics discussed in the related work. The answer is yes. The proposed by Patel-Schneider and Martin evaluates the inner pattern first, so it corresponds to the top-down logical substitution. On the other hand, the proposal of Seaborne applies substitutions in the leaves of the syntactical tree of an expression, so it corresponds to bottom-up logical substitution.

Reviewer 3: Concerning the expression “equality formulas require that all variables be assigned before being evaluated.” The reviewer is right in that the explanation is poor. To explain better the notion that we are introducing, consider the rule . According to Levene and Loizou (“A guided tour of relational databases and beyond”), if the database contains a fact q(a) then the rule infers the fact . Intuitively, the equality “pass” the value from to . We do not use such notion because it brings problems when is bound in the outer query. With our built-in equality, the rule is unsafe because does not occur positively. In other words, is not assigned.

Reviewer 3: Concerning to Stratified Datalog. When Datalog is extended with negation, two approaches could be used, Stratified or Inflationary (see the survey of Ceri, Gottlob and Tanca of 1989). Datalog programs with negation satisfy several minimal Herbrand models. This entails difficulties in defining the semantics of Datalog programs. Stratified Datalog permit us choosing a distinguished minimal Herbrand model by approximating the CWA. To the best of our knowledge, all translation from SQL and SPARQL to Datalog consider stratified programs.

Reviewer 3: About the notion of “logically connected.” It was intended as an informal intuition just to say that two variables are not logically independent (that is, the values they get are related somewhat). For instance, consider the rules and . Then the variables and are logically connected. To be more precise, the connection is not between the variables, but between the position where variables occur in each rule. We can rename the variables consequently without breaking the logical connection.

The notions of “logical chain”, “logical substitution” and “improper substitution” are related to the notion of “logically connected”. A logical chain is a sequence of connections between variables. We can apply a logical substitution in a variable occurring in a rule of a nested query if there is a logical chain from an occurrence of in the goal of the nested query to the variable in such rule. Similarly, the substitution is improper when there is no such connection between the variable substituted Y and an occurrence of in the goal of the nested query. One of our contributions is showing that some engines do an improper substitution.

What is the problem with improper substitution? In Sec. 3 we argue that improper substitution compromises the compositionality of nesting. As we explain in this appendix, one of the reasons for a logical foundation is to provide a logical interpretation for queries. In the proof of Lemma 2, we extend the usual translation from Datalog programs to first order formulas by providing a translation for the new type of atom: nested queries. In an ideal setting, each nested query can be translated into a first order query, independently of the outer query where it is nested. Intuitively, compositionality is achieved when the meaning of the nested query does not depend on the query where it is nested. For instance, let be a query nested in a rule of an outer query, be a variable that occurs in a rule in , and be not logically connected with the goal of . Then, allowing improper substitution, we have two possible translations for as a first order formula. If occurs positively in , then we remove the universal quantifier to the rule . Otherwise, we do not remove it.

From reviewer 3: concerning the meaning of . We write to denote the program . Thus, the program has a rule and a fact.

From reviewers 3 and 4: The notice that in the table of page 6, when is moved down, then it changes to . Reviewers suggest that it could be a typo. However, this is correct. The variables on the left side of the equality correspond to the variable that is connected with X, that may change depending on the rule. Also does not change because it is a constant in the current substitution.

Reviewer 4 complains that the semantics of was not explained for a program . We do not define , because it is not necessary to define the semantics of a program. It suffices to define what facts are inferred by a rule. The procedure for doing that is standard in the Datalog literature. First, a true value is defined for literals in the body of a rule with respect to a substitution. We write to denote that a literal is evaluated as true with respect to a substitution and a set of facts . Second, a fact is inferred in a rule if is the head and for each literal in the body of the rule, is true with respect to . This process ends when no more facts can be inferred from all the rules of the program. Thus, the semantics of the program is defined by this inference process.

For first order formulas and , the notation has the standard meaning. That is, is a logical consequence of .

Reviewer 4: concerning to how variables in a negated subquery can be positively defined. Variables are defined positively in the scope of a rule. As subqueries are built-ins, a subquery does not matter for counting the variables occurring positively in the rule where occurs. Also, the variables that are defined positively in each rule of the subquery do not depend on if is negated or not.

Reviewer 4: concerning to what “evaluate ” means. It means computing all the facts that can be inferred from P in the current database.

Reviewer 4: concerning to what “compatible” means. Compatibility is an standard SPARQL notion. Two mappings mu and nu are said to be compatible if for each variable in the domain of mu holds or is unbound in .

Reviewer 4: concerning what “moving down literals of the form ” mean. It essentially means if is logically connected with another variable that occurs in a rule below in dependency graph of the program, then an equivalent literal is appended to such rule, and then removed from the original rule. This process is detailed in Appendix A.

Reviewer 4: concerning what “if the logical substitution in a query uses the definition of substitution for a rule.” The answer is yes. We first defined how to substitute a variable by a value in a rule. Then we defined how to substitute by in a program. It is not simply substituting by in a rule where occurs because in Datalog the scope of variables are the rules where they occur. Thus, we need to identify the variables that are logically connected and then apply a substitution of them, using the previous definition of substitution in a rule. We denote the substitution of in a program as .

Reviewer 4: concerning to “extensional database.” This is a standard concept of Datalog that can be revised in the Datalog survey of Ceri et al. mentioned before. Essentially, an extensional database is the set of facts where the query is evaluated, and the intentional database is such that includes the inferred facts.

Reviewer 4: about how the corollary follows from Lemma 2. First order queries and standard Datalog have the same expressive power (see the survey referred above). Lemma 2 says that Nested Datalog also has the same expressive power than first order queries. Then, we conclude that nesting does not add expressive power to Datalog. This applies to all the presented methods of substitution because all of them were checked in the proof of Lemma 2.

Reviewer 4: concerning the word EXISTStential in the bibliography. This word is correct. It was part of the title of the cited paper. The authors create this new world to make emphasize of the keyword EXISTS.