G-CORE: A Core for Future Graph Query Languages

12/05/2017 ∙ by Renzo Angles, et al. ∙ 0

We report on a community effort between industry and academia to shape the future of graph query languages. We argue that existing graph database management systems should consider supporting a query language with two key characteristics. First, it should be composable, meaning, that graphs are the input and the output of queries. Second, the graph query language should treat paths as first-class citizens. Our result is G-CORE, a powerful graph query language design that fulfills these goals, and strikes a careful balance between path query expressivity and evaluation complexity.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


G-CORE is a design by the LDBC Graph Query Language Task Force, consisting of members from industry and academia, intending to bring the best of both worlds to graph practitioners.

LDBC is not a standards body and rather than proposing a new standard, we hope that the design and features of G-CORE will guide the evolution of both existing and future graph query languages, towards making them more useful, powerful and expressive.

1. Introduction

In the last decade there has been increased interest in graph data management. In industry, numerous systems that store and query or analyze such data have been developed. In academia, manifold functionalities for graph databases have been proposed, studied and experimented with.

Graphs are the ultimate abstraction for many real world processes and today the computer infrastructure exists to collect, store and handle them as such. There are several models for representing graphs. Among the most popular is the property graph data model, which is a directed graph with labels on both nodes and edges, as well as property,value pairs associated with both. It has gained adoption with systems such as AgensGraph (agens), Amazon Neptune (neptune), ArangoDB (arangodb), Blazegraph (blazegraph), CosmosDB (cosmos), DataStax Enterprise Graph (dsegraph), HANA Graph (RPBL13), JanusGraph (janus), Neo4j (N17), Oracle PGX (sevenich2016using), OrientDB (orientdb), Sparksee (sparksee), Stardog (stardog), TigerGraph (tiger), Titan (titan), etc. These systems have their own storage models, functionalities, libraries and APIs and many have query languages. This wide range of systems and functionalities poses important interoperability challenges to the graph database industry. In order for the graph database industry to cooperate, community efforts such as Apache Tinkerpop, openCypher(openCypher) and the Linked Data Benchmark Council (LDBC) are providing vendor agnostic graph frameworks, query languages and benchmarks.

LDBC was founded by academia and industry in 2012 (ABLFNENMKVT14) in order to establish standard benchmarks for such new graph data management systems. LDBC has since developed a number of graph data management benchmarks (OALCGPPB15; KEKFA17; IHNHPMCCSATXNB16) to contribute to more objective comparison among systems, informing prospective users of some of the strong- and weak-points of the various systems before even doing a Proof-Of-Concept study, while providing system engineers and architects clear targets for performance testing and improvement. LDBC regularly organizes Technical User Community (TUC) meetings, where not only members report on progress of LDBC task forces but also gather requirements and feedback from data practitioners, who are also present. There have been over 40 graph use-case presentations by data practitioners in these TUC meetings, who often are users of the graph data management software of LDBC members, such as IBM, Neo4j, Ontotext, Oracle and SAP. The topics and contents of these collected TUC presentations show that graph databases are being adopted over a wide range of application fields, as summarized in Figure 1

. This further shows that the desired graph query language features are graph pattern matching (e.g., identification of communities in social networks), graph reachability (e.g., fraud detection in financial transactions or insurance), weighted path finding (e.g., route optimization in logistics, or bottleneck detection in telecommunications), graph construction (e.g., data integration in Bioinformatics or specialized publishing domains such as legal) and graph clustering (e.g., on social networks for customer relationship management).

Application Fields Used Features
healthcare / pharma 14 graph reachability 36
publishing 10 graph construction 34
finance / insurance 6 pattern matching 32
cultural heritage 6 shortest path search 19
e-commerce 5 graph clustering 14
social media 4
telecommunications 4
Figure 1. Graph database usage characteristics derived from the use-case presentations in LDBC TUC Meetings 2012-2017 (source: https://github.com/ldbc/tuc_presentations).

1.1. Three Main Challenges

The following issues are observed about existing graph query languages. These observations are based on the LDBC TUC use-case analysis and feedback from industry practitioners:

Composability. The ability to plug and play is an essential step in standardization. Having the ability to plug outputs and inputs in a query language incentivizes its adoption (modularity, interoperability); simplify abstractions, users do not have to think about multiple data models during the query process; and increases its productivity, by facilitating reuse and decomposition of queries. Current query languages do not provide full composability because they output tables of values, nodes or edges.

Path as first-class citizens. The notion of Path is fundamental for graph databases, because it introduces an intermediate abstraction level that allows to represents how elements in a graph are related. The facilities provided by a graph query language to manipulate paths (i.e. describe, search, filter, count, annotate, return, etc.) increase the expressivity of the language. Particularly, the ability to return paths enables the user to post-process paths within the query language rather that in an ad-hoc manner (dries).

Capture the core of available languages. Both the desirability of a standard query language and the difficulty of achieving this, is well-established. This is particularly true for graph data languages due to the diversity of models and the rich properties of the graph model. This motivates our approach to take the successful functionalities of current languages as a base from where to develop the next generation of languages.

1.2. Contributions

Since the lack of a common graph query language kept coming up in LDBC benchmark discussions, it was decided in 2014 to create a task force to work on a common direction for property graph query languages. The authors are members of this task-force.

This paper presents G-CORE, a closed query language on Property Graphs. It is a coherent, comprehensive and consistent integration of industry desiderata and the leading functionalities found in industry practices and theoretical research.

The paper presents the following contributions:

Path Property Graph model. G-CORE treats paths as first-class citizens. This means that paths are outputs of certain queries. The fact that the language must be closed implies that paths must be part of the graph data model. This leads to a principled change of the data model: it extends property graphs with paths. That is, in a graph, there is also a (possibly empty) collection of paths; where a path is a concatenation of existing, adjacent, edges. Further, given that nodes, edges and paths are all first-class citizens, paths have identity and can also have labels and property,value pairs associated with them. This extended property graph model, called the Path Property Graph model, is backwards-compatible with the property graph model.

Syntax and Semantics of G-CORE. A key contribution is the formal definition of G-CORE. This formal definition prevents any ambiguity about the functionality of the language, thus enabling the development of correct implementations. In particular, an open source grammar for G-CORE is available111https://github.com/ldbc/ldbc_gcore_parser.

Complexity results. To ensure that the query language is practically usable on large data, the design of G-CORE was built on previous complexity results. Features were carefully restricted in such ways that G-CORE is tractable (each query in the language can be evaluated efficiently). Thus, G-CORE provides the most powerful path query functionalities proposed so far, while carefully avoiding intractable complexity.

Organization of the paper. This paper first defines the Extended Property Graph model in Section 2. Then it explains G-CORE in Section 3 via a guided tour, using examples on the LDBC Social Network Benchmark dataset (OALCGPPB15), which demonstrate its main features. We summarize our formal contributions, comprising syntax, semantics and complexity analysis of G-CORE in Section 4, while the details of these are described in Appendix A. In Section 5, we show how G-CORE can be easily extended to handle tabular data. We discuss related work in Section 6, before concluding in Section 7.

Figure 2. A small social network. A Path Property Graph (PPG) is a Property Graph that can have “Stored Paths”.

2. Path Property Graphs

We first define the data model of G-CORE, which is an extension of the Property Graph data model (RN10; AABHRV16; V17; N17; LPGM17). We call this model the Path Property Graph model, or PPG model for short. Let L be an infinite set of label names for nodes, edges and paths, K an infinite set of property names, and V an infinite set of literals (actual values such as integer and real numbers, strings, dates, truth values and , that represent true and false, respectively, etc.). Moreover, given a set , let denote the set of all finite subsets of (including the empty set), and denote the set of all finite lists of elements from (including the empty list).

Definition 2.1 ().

A PPG is a tuple , , where:

  1. is a finite set of node identifiers, is a finite set of edge identifiers and is a finite set of path identifiers, where , and are pairwise disjoint.

  2. is a total function.

  3. is a total function such that for every , it holds that , where: (i) , (ii) for every , and (iii) or for every

  4. is a total function.

  5. is a total function for which there exists a finite set of tuples such that

Given an edge in a PPG , if , then is the starting node of and is the ending node of . The function allows us to have several edges between the same pairs of nodes. Function assigns to each path identifier an actual path in : this is a list satisfying condition (3) in Definition 2.1. Function is used to specify the set of labels of each node, edge, and path, while function is used to specify the values of a property for every node, edge, and path. To be precise, if and is a property name, then is the set of values of the property for the identifier . Observe that if , then we implicitly assume that property is not defined for identifier , as there is no value of this property for this object. Note that although K is an infinite set of property names, in only a finite number of properties are assigned values as we assume that there exists a finite set of tuples such that .

Example 2.2 ().

As a simple example of a PPG, consider the small social network graph given in Figure 2. Here we have

as node, edge, and path identifiers, respectively;

as edge and path assignments, respectively; and,


as label and property value assignments, respectively.

Paths. It is worth remarking that paths are included as a first-class citizens in this data model (at the level of nodes and edges). In particular, paths can have labels and properties, where the latter can be used to describe built-in properties like the length of the path. In our example above, the path with identifier 301 has label “toWagner” and value 0.95 on property “trust”.

For convenience, we use and to denote the list of all nodes and edges of a path bound to a variable , respectively. Formally, if then and . In our example above, and .

3. A Guided Tour of G-CORE

Figure 3. Social Network Benchmark schema (simplified).

We will now demonstrate and explain the main features of the G-CORE language. The concrete setting is the LDBC Social Network Benchmark (SNB), as illustrated in the simple social network from Figure 2, whose (simplified) schema is depicted in Figure 3. Figure 4 depicts the toy instance (which we refer to as social_graph) on which our example queries are evaluated. The use-cases in these examples are data integration and expert finding in a social network.

Always returning a graph. Let us start with what is possibly one of the simplest G-CORE queries:

2  MATCH   (n:Person)
3    ON    social_graph
4    WHERE n.employer = ’Acme’ 

In G-CORE every query returns a graph, as embodied by the CONSTRUCT clause which is at the start of every query body. This example query constructs a new graph with no edges and only nodes, namely those persons who work at Acme – all the labels and properties that these person nodes had in social_graph are preserved in the returned result graph.

Match and Filter. The MATCH..ON..WHERE clause matches one or more (comma separated) graph patterns on a named graph, using the homomorphic semantics (AABHRV16).

Systems may omit ON if there is a default graph – let us assume in the sequel that social_graph is the default graph. Parenthesis demarcate a node, where n is a variable bound to the identity of a node, :Person a label, and n.employer a property. The G-CORE builds on the ASCII-art syntax from Cypher (Cypher18) and the regular path expression syntax from PGQL (RHKMC16), which has proven intuitive, effective and popular among property graph database users.

The previous example contains a WHERE filter with the obvious semantics: it eliminates all matches where the employer is not Acme.

Figure 4. Initial graph database (social_graph). Note that the knows edges are drawn bi-directionally – this means there are two edges: one in each direction.

Multi-Graph Queries and Joins. A more useful query would be a simple data integration query, where we might have loaded (unconnected) company nodes into a temporary graph company_graph, but now want to create a unified graph where employees and companies are connected with an edge labeled worksAt. Let us assume that company_graph contains nodes for Acme, HAL, CWI and MIT. As an aside, the real SNB dataset already contains such Company nodes with :worksAt edges to the employees (which in reality do not have an employer property).

The below query has a MATCH clause with two graph patterns, matching these on two different input graphs. Graph patterns that do not have variables in common lead to the Cartesian product of variable bindings, but this query also has a WHERE clause that turns it into an equi-join:

4CONSTRUCT (c)<-[:worksAt]-(n)
5  MATCH   (c:Company) ON company_graph, 
6          (n:Person)  ON social_graph
7    WHERE c.name = n.employer  
8UNION social_graph 

The UNION operator takes it intuitive meaning, and will be touched upon later when we talk about node and edge identity.

Generally speaking, MATCH produces a set of bindings which alternatively may be viewed as a table having a column for each variable and one row for each binding. Bindings typically contain node, edge and path identities, whose shape is opaque, but we use intuitive names prefixed # here:

c n
#Acme #Alice
#HAL #Celine
#Acme #John

Dealing with Multi-Valued properties. In the previous query there is the complication that n.employer is multi-valued for Frank Gold: he works for both MIT and CWI. Therefore, his person node fails to match with both companies. To explain this, we show the bindings and values of c.name and n.employer if WHERE c.name = n.employer were omitted, and the query would be a Cartesian product:

c c.name n n.employer
#MIT ”MIT” #Peter
#CWI ”CWI” #Peter
#Acme ”Acme” #Peter
#HAL ”HAL” #Peter
#MIT ”MIT” #Frank {”CWI”, ”MIT”}
#CWI ”CWI” #Frank {”CWI”, ”MIT”}
#Acme ”Acme” #Frank {”CWI”, ”MIT”}
#HAL ”HAL” #Frank {”CWI”, ”MIT”}
#MIT ”MIT” #Alice ”Acme”
#CWI ”CWI” #Alice ”Acme”
#Acme ”Acme” #Alice ”Acme”
#HAL ”HAL” #Alice ”Acme”
#MIT ”MIT” #Celine ”HAL”
#CWI ”CWI” #Celine ”HAL”
#Acme ”Acme” #Celine ”HAL”
#HAL ”HAL” #Celine ”HAL”
#MIT ”MIT” #John ”Acme”
#CWI ”CWI” #John ”Acme”
#Acme ”Acme” #John ”Acme”
#HAL ”HAL” #John ”Acme”

Notice that according to the definition of our data model, the value of c.name is a set. But in the case c.name is a singleton set, we omit curly braces, so we simply write "MIT" instead of {"MIT"}. In the table above, the rows in bold would be the ones that earlier led to bindings surviving the join. Essentially, "MIT"={"CWI","MIT"} and "CWI"={"CWI","MIT"} evaluate to FALSE.

Note that Peter is unemployed, so his n.employer value is unbound. More precisely, its Person node does not have an employer property at all. In case of an absent property, its evaluation results in the empty set, which a length test can detect. G-CORE provides CASE expressions to coalesce such missing data into other values.

One way to resolve the failing join for Frank, would be to use IN instead of =, so the comparisons mentioned earlier resolve to TRUE:

8CONSTRUCT (c)<-[:worksAt]-(n)
9  MATCH   (c:Company) ON company_graph,  
10          (n:Person)  ON social_graph
11    WHERE c.name IN n.employer 
12UNION social_graph 

Notice that the IN operator can be used when c.name is a singleton set, as in this case it is natural to ask whether the value in c.name is an element of n.employer. If we need to compare c.name with n.employer as sets, then the operator SUBSET can be used.

Another way to deal with this in G-CORE is to bind a variable ({name:e}) to the employer property, which unrolls multi-valued properties into individual bindings:

12CONSTRUCT (c)<-[:worksAt]-(n)
13  MATCH   (c:Company)             ON company_graph,
14          (n:Person {employer=e}) ON social_graph
15    WHERE c.name = e 
16UNION social_graph 

Inside the MATCH expression that binds a node, curly braces can be used to bind variables to property values. The set of bindings for this MATCH (which includes the join) now has three variables and the following bindings:

c n e
#MIT #Frank ”MIT”
#CWI #Frank ”CWI”
#Acme #Alice ”Acme”
#HAL #Celine ”HAL”
#Acme #John ”Acme”

Construction that respects identities. The CONSTRUCT operation fills a graph pattern (used as template) for each binding in the set of bindings produced by the MATCH clause. Edges are denoted with square brackets, and can be pointed towards either direction; in this case there is no edge variable, but there is an edge label :worksAt. Note, to be precise, that CONSTRUCT by default groups bindings when creating elements. Nodes are grouped by node identity, and edges by the combination of source and destination node. While five new edges are created here, they are between four existing persons and four existing companies due to this grouping. For instance, the person #Frank, who works for both MIT and CWI, gets two :worksAt edges, to respectively company #MIT and company #CWI.

In the last line of this example query, we UNION-ed these new edges with the original graph, resulting in an enriched graph: the original graph plus five edges. The “full graph” query operators like union and difference are defined in terms of node, edge and path identities. These identities are taken from the input graph(s) of the query. G-CORE is a query language, not an update language. Even though CONSTRUCT allows with SET prop:=val and REMOVE prop to change properties and values (a later example will demonstrate SET), this does not modify the graph database, it just changes the result of that particular query. The practice of returning a graph that shares (parts of) nodes, edges and paths with its inputs, using this concept of identity, provides opportunities for systems to share memory and storage resources between query inputs and outputs.

A shorthand form for the union operation is to include a graph name directly in the comma separated list of CONSTRUCT patterns, as depicted in the next query:

16CONSTRUCT  social_graph,
17    (x GROUP e :Company {name:=e})<-[y:worksAt]-(n) 
18  MATCH   (n:Person {employer=e})  

Graph Aggregation. The above query demonstrates graph aggregation. Supposing there would not have been any company nodes in the graph, we might also have created them with this excerpt:

CONSTRUCT (n)-[y:worksAt]->(x:Company{name:=n.employer})

However, this unbound destination node x would create a company node for each binding222In addition, it would create a company with the name property with the values .. This is not what we want: we want only one company per unique name. Graph aggregation therefore allows an explicit GROUP clause in each graph pattern element. Thus, in the above query with GROUP e, we create only one company node for each unique value of e in the binding set. Here the curly brace notation is used inside CONSTRUCT to instantiate the Company.name property in the newly created nodes.

The set of bindings of our graph aggregation query example has the same variables n and e variables of the previous binding set. The CONSTRUCT for node expression (n) groups by node identity so instantiates the nodes with identity #Frank, #Alice, #Celine and #John in the query result. These nodes were already part of social_graph, so given that the CONSTRUCT is UNION-ed with that, no extra nodes result.

For the (x GROUP e..) node expression, CONSTRUCT groups by e into bindings ”CWI”, ”MIT”, ”Acme”, and ”HAL” and because x is unbound, it will create four new nodes with, say, identities #CWI, #MIT, #Acme and #HAL. For the edges to be constructed, G-CORE performs by default grouping of the bindings on the combination of source and destination node, and this results in again five new edges.

When using bound variables in a CONSTRUCT, they must be of the right sort: it would be illegal to use n (a node) in the place of y (an edge) here. In case an edge variable (here: y) would have been bound (in the MATCH), CONSTRUCT imposes the restriction that its node variables must also be bound, and be bound to exactly its source and destination nodes, because changing the source and destination of an edge violates its identity. However, it can be useful to bind edges in MATCH and use these to construct edges with a new identity, which are copies of these existing edges in terms of labels and property-values. For this purpose, G-CORE supports the -[=y]- syntax which makes a copy of the bound y edges (as well as the (=n) syntax for nodes). Then, the above restriction does not apply. With the copy syntax, it is even possible to copy all labels and properties of a node to an edge (or a path) and vice versa.

In this example, x and y were unbound and could have been omitted. In the preceding examples, they were in fact omitted. Unbound variables in a CONSTRUCT are useful if they occur multiple times in the construct patterns, in order to ensure that the same identities will be used (i.e., to connect newly created graph elements, rather than generate independent nodes and edges).

Storing Paths with @p. G-CORE is unique in its treatment of paths, namely as first-class citizens. The below query demonstrates finding the three shortest paths from John Doe towards each other person who lives at his location, reachable over knows edges, using Kleene star notation <:knows*>:

18CONSTRUCT (n)-/@p:localPeople{distance:=c}/->(m) 
19  MATCH   (n)-/3 SHORTEST p<:knows*> COST c/->(m)
20    WHERE (n:Person) AND (m:Person)
21      AND n.firstName = ’John’ AND n.lastName = ’Doe’ 
22      AND (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m) 

In G-CORE, paths are demarcated with slashes -/ /-. In the above example p <:knows*> binds the shortest path between the single node n (i.e. John Doe) and every possible person m, under the restriction that this target person lives in the same place. By writing e.g., -/3 SHORTEST p <:knows*>/-> we obtain multiple shortest paths (at most 3, in this case) for every source–destination combination; if the number 3 would be omitted, it would default to 1. In case there are multiple shortest paths with equal cost between two nodes, G-CORE delivers just any one of them. By writing p <:knows*> COST c/-> we bind the shortest path cost to variable c. By default, the cost of a path is its hop-count (length). We will define weighted shortest paths later. If we would not be interested in the length, COST c could be omitted.

Figure 5. Graph view social_graph1, which adds nr\_message properties to the original social_graph (social\_graph2 is social_graph1 plus the Stored Paths in the grey box).

In CONSTRUCT (n)-/@p:localPeople{distance:c}, we see the bound path variable @p. The @ prefix indicates a stored path, that is, this query is delivering a graph with paths. Each path is stored attaching the label :localPeople, and its cost as property distance.

The graph returned by this query – which lacks a UNION with the original social_graph – is a projection of all nodes and edges involved in these stored paths. We omitted a figure of this for brevity.

Reachability and All Paths. In a similar query where we just return m, and do not store paths, the <:knows*> path expression semantics is a reachability test:

23  MATCH   (n:Person)-/<:knows*>/->(m:Person) 
24    WHERE n.firstName = ’John’ AND n.lastName = ’Doe’ 
25      AND (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m) 

In this case we use -/<:knows*>/-> without the SHORTEST keyword. Using ALL instead of SHORTEST: asking for all paths, is not allowed if a path variable is bound to it and used somehwere, as this would be intractable or impossible due to an infinite amount of results. However, G-CORE can support it in the case where the path variable is only used to return a graph projection of all paths:

25CONSTRUCT (n)-/p/->(m)
26  MATCH   (n:Person)-/ALL p<:knows*>/->(m:Person)
27    WHERE n.firstName = ’John’ AND n.lastName = ’Doe’ 
28      AND (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m) 

The method (BLLW12) shows how the materialization of all paths can be avoided by summarizing these paths in a graph projection; hence this functionality is tractable.

Existential Subqueries. In the SNB graph, isLocatedIn is not a simple string attribute, but an edge to a city, and the three previous query examples used pattern matching directly in the WHERE clause: (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m). G-CORE allows this and uses implicit existential quantification, which here is equivalent to:

30    MATCH (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m) )

This constructs one new node (unbound anonymous node variable ()) for each match of n and m coinciding in the city where they are located – where that city is represented by a () again. Whenever such a subquery evaluates to the empty graph, the automatic existential semantics of WHERE evaluates to FALSE; otherwise to TRUE.

Views and Optionals. The fact that G-CORE is closed on the PPG data model means that subqueries and views are possible. In the following example we create such a view:

30GRAPH VIEW social_graph1 AS ( 
31  CONSTRUCT social_graph,
32        (n)-[e]->(m) SET e.nr_messages := COUNT(*) 
33    MATCH (n)-[e:knows]->(m)
34      WHERE (n:Person) AND (m:Person)
35      OPTIONAL (n)<-[c1]-(msg1:Post|Comment), 
36               (msg1)-[:reply_of]-(msg2),
37               (msg2:Post|Comment)-[c2]->(m)
38        WHERE (c1:has_creator) AND (c2:has_creator) )

The result of this graph view can be seen in Figure 5. To each :knows edge, this view adds a nr_messages property, using the SET .. := sub-clause. This sub-clause of CONSTRUCT can be used to modify properties of nodes, edges and paths that are being constructed. This particular nr_messages property contains the amount of messages that the two persons n and m have actually exchanged, and is a reliable indicator of the intensity of the bond between two persons.

The edge construction (n)-[e]->(m) adds nothing new, but as described before, performs implicit graph aggregation, where bindings are grouped on n,m,e, and COUNT(*) evaluates to the amount of occurrences of each combination.

This example also demonstrates OPTIONAL matches, such that people who know each other but never exchanged a message still get a property e.nr_messages=. All patterns separated by comma in an OPTIONAL block must match. Technically, the set of bindings from the main MATCH is left outer-joined with the one coming out of the OPTIONAL block (and there may be more than one OPTIONAL blocks, in which case this repeats). There can be multiple OPTIONAL blocks, and each OPTIONAL block can have its own WHERE; we demonstrate this here by moving some label tests to WHERE clauses (on :Person and :has_creator). This query also demonstrates the use of disjunctive label tests (msg1:Post|Comment).

If a query contains multiple OPTIONAL blocks, they have to be evaluated from the top to the bottom. For example, to evaluate the following pattern:

38MATCH (n:Person)
39   OPTIONAL (n)-[:worksAt]->(c)
40   OPTIONAL (n)-[:livesIn]->(a)

we need to perform the following steps: evaluate (n:Person) to generate a binding set , evaluate (n)-[:worksAt]->(c) to generate a binding set , compute the left-outer join of with to generate a binding set , evaluate (n)-[:livesIn]->(a) to generate a binding set , and compute the left-outer join of with to generate a binding set that is the result of evaluating the entire pattern. Obviously, in this case the order of evaluation is not relevant, and the previous pattern is equivalent to:

40MATCH (n:Person)
41   OPTIONAL (n)-[:livesIn]->(a)
42   OPTIONAL (n)-[:worksAt]->(c)

However, the order of evaluation can be relevant if the optional blocks of a pattern shared some variables that are not mentioned in the first pattern. For example, in the following expression the variable a is mentioned in the optional blocks but not in the first pattern (n:Person):

42MATCH (n:Person)
43   OPTIONAL (n)-[:worksAt]->(a)
44   OPTIONAL (n)-[:livesIn]->(a)

Arguably, such a pattern is not natural, and it should not be allowed in practice. By imposing the simple syntactic restriction that variables shared by optional blocks have to be present in their enclosing pattern, one can ensure that the semantics of a pattern with multiple OPTIONAL blocks is independent of the evaluation order (PAG09).

Weighted Shortest Paths. The finale of this section describes an example of expert finding: let us suppose that John Doe wants to go to a Wagner Opera, but none of his friends likes Wagner. He thus wants to know which friend to ask to introduce him to a true Wagner lover who lives in his city (or to someone who can recursively introduce him). To optimize his chances for success, he prefers to try “friends” who actually communicate with each other. Therefore we look for the weighted shortest path over the wKnows (“weighted knows”) path pattern towards people who like Wagner, where the weight is the inverse of the number of messages exchanged: the more messages exchanged, the lower the cost (though we add one to the divisor to avoid overflow). For each Wagner lover, we want a shortest path.

In G-CORE, weighted shortest paths are specified over basic path patterns, defined by a PATH .. WHERE .. COST clause, because this allows to specify a cost value for each traversed path pattern. The specified cost must be numerical, and larger than zero (otherwise a run-time error will be raised), where the full cost of a path (to be minimized) is the sum of the costs of all path segments. If the COST is omitted, it defaults to 1 (hop count).

44GRAPH VIEW social_graph2 AS ( 
45PATH wKnows = (x)-[e:knows]->(y)  
46  WHERE NOT ’Acme’ IN y.employer  
47  COST 1 / (1 + e.nr_messages) 
48CONSTRUCT social_graph1, (n)-/@p:toWagner/->(m)
49  MATCH   (n:Person)-/p<~wKnows*>/->(m:Person)
50    ON    social_graph1
51    WHERE (m)-[:hasInterest]->(:Tag {name=’Wagner’}) 
52      AND (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m)
53      AND n.firstName = ’John’ AND n.lastName = ’Doe’)
Matching all patterns (Homomorphism) *
Matching literal values 3, 3
Matching shortest paths 3
Matching all shortest paths 3
Matching weighted shortest paths 3
(multi-segment) optional matching 3
Querying multiple graphs 3
Queries on paths 3
Filtering matches 3,3,3,3,3,3,3,3,3,3
Filtering path expressions 3
Value joins 3
Cartesian product 3
List membership 3
Set operations on graphs 3, 3, 3
Existential subqueries
- Implicit 3, 3, 3
- Explicit 3
Graph construction *
Graph aggregation 3
Graph projection 3
Graph views 3, 3
Property addition 3
Table 1. Overview of G-CORE features and their line occurrences in the example queries in Section 3.

The result of this graph view (social_graph2) was already depicted in Figure 5: it adds to social_graph1 two stored paths. Apart from GRAPH VIEW name AS (query), and similar to CREATE VIEW in SQL which introduces a global name for a query expression, G-CORE also supports a GRAPH name AS (query1) query2 clause which, similar to WITH in SQL, introduces a name that is only visible inside query2.

Powerful Path Patterns. Basic PATH patterns are a powerful building block that allow complex path expressions as concatenations of these patterns (RHKMC16) using a Kleene star, yet still allow for fast Dijkstra-based evaluation. In G-CORE, these path patterns can even be non-linear shapes, as PATH can take a comma-separated list of multiple graph patterns. But, the path pattern must contain a start and end node (a path segment), which is taken to be the first and last node in its first graph pattern. This ensures path patterns can be stitched together to form paths – a path pattern always contains a path segment between its start and end nodes. These basic path patterns can also contain WHERE conditions, without restrictions on their complexity. As John Doe wants his preference for Wagner to remain unknown at his work, we exclude employees of Acme from occurring on the path333Note that non-linear path patterns, such as PATH (a)-[]-(b),(b)-(c) add power over linear patterns with existential filters: PATH (a)-[]-(b) WHERE (b)-(c), because the latter cannot bind variables. In G-CORE, variable c can be used in a COST expression.. The result of this query is a view social_graph2 in which all these shortest paths from John Doe to Wagner lovers have been materialized (the @p in (n)-/@p:toWagner/->(m)).

A unique capability of G-CORE is to query and analyze databases of potentially many stored paths. We demonstrate this in the final query, where we score John’s friends for their aptitude:

53CONSTRUCT (n)-[e:wagnerFriend {score:=COUNT(*)}]->(m)
54          WHEN e.score > 0
55  MATCH   (n:Person)-/@p:toWagner/->(), (m:Person) 
56    ON    social_graph2
57    WHERE n = nodes(p)[1] 

For the :toWagner paths, we use nodes(p)[1] to look at the second node in each path, i.e. a direct friend of John Doe. G-CORE starts counting at 0 so nodes(my_path)[n] returns the item from the list returned by the function nodes(), which returns all nodes on a path. For these direct friends we count how often they occur as the start of :toWagner paths. These scores has been attached as a score property to new :wagnerFriend edges. Since in the toy example there are only two Wagner lovers and thus two shortest paths to them, both via Peter, the result of this query is a single :wagnerFriend edge between John and Peter with score 2.

4. Formalizing and Analyzing G-CORE

One of the main goals of this paper is to provide a formal definition of the syntax and semantics of a graph query language including the features shown in the previous sections. Formally, a G-CORE query is defined by the following top-down grammar:

Thus, a G-CORE query consists of a sequence of Path and Graph clauses, followed by a full graph query, i.e., a combination of basic graph queries under the set operations of union, intersection and difference. A basic graph query consists of a single Construct clause followed by one Match clause. We have seen examples of all these features in Section 3.

In Appendix A, we provide formal definitions of the syntax and semantics of G-CORE. The basic idea of the language is, given a PPG , to create a new PPG using the Construct clause. This is achieved, in turn, by applying an intermediate step provided by the Match clause. The application of such a clause creates a set of bindings , based on a graph pattern that is evaluated over . The interaction between the Match and the Construct clause is explained in more detail below:

  • The result of evaluating the graph pattern that defines the content of a Match clause over a PPG always corresponds to a set of bindings, which is denoted by . The bindings in can then be filtered by using boolean conditions specified in the Where clause.

  • A Construct clause then takes as input both the PPG and the set of bindings , and produces a new PPG , which is denoted by . Note that is also an input in the evaluation of , as the set of bindings can make reference to objects whose labels and properties are defined in .

The role of the Path clause is to define complex path expressions, as well as the cost associated with them, that can in turn be used in graph patterns in the Match clause. In this way, it is possible to define rich navigational patterns on graphs that capture expressive query languages that have been studied in depth in the theoretical community (e.g., the class of regular queries (RRV17)).

Complexity analysis. The G-CORE query language has been carefully designed to ensure that G-CORE queries can be evaluated efficiently in data complexity. Formally, this means that for each fixed G-CORE query , the result of evaluating over an input PPG can be computed in polynomial time. The main reasons that explain this fact are given below.

First of all, graph patterns correspond (essentially) to conjunctions of atoms expressing that two nodes are linked by a path satisfying a certain regular expression over the alphabet of node and edge labels. The set of all bindings of a fixed graph pattern over the input PPG can then be easily computed in polynomial time: we simply look for all possible ways of replacing node and edge variables in by node and edge identifiers in , respectively, and then for each path variable representing a path in from node to node whose label must conform to a regular expression , we replace by the shortest/cheapest path in from to that satisfies (if it exists). This can be done in polynomial time by applying standard automata-theoretic techniques in conjunction with Dijkstra-style algorithms. (Notice that the latter would not be true if our semantics was based on simple paths; in fact, checking if there is a simple path in an extended property graph whose label satisfies a fixed regular expression is an NP-complete problem (MW95)).

Suppose, now, that we are given a fixed G-CORE query that corresponds to a sequence of clauses followed by a full graph query . Each clause is defined by a graph pattern whose evaluation corresponds to a binary relation over the nodes of the input PPG . By construction, the graph pattern might mention binary patterns which are defined in previous clauses. Therefore, it is possible to iteratively evaluate in polynomial time all graph patterns that are mentioned in the clauses of . Once this process is finished, we proceed to evaluate (which is defined in terms of the ’s).

By definition, is a boolean combination of full graph queries . It is thus sufficient to explain how to evaluate each such a full graph query in polynomial time. We can assume by construction that consists of a Construct clause applied over a Match clause. We first explain how the set of bindings that satisfy the Match clause can be computed in polynomial time. Since one or more Optional clauses could be applied over the Match clause, the semantics is based on the set of maximal bindings for the whole expression, i.e., those that satisfy the primary graph pattern expressed in the Match clause, and as many atoms as possible from the basic graph patterns that define the Optional clauses. The computation of can be carried out in polynomial time by a straightforward extension of the aforementioned techniques for efficiently evaluating basic graph patterns. Finally, filtering in accordance with the boolean conditions expressed in the Where clause can easily be done in polynomial time (under the reasonable assumption that such conditions can be evaluated efficiently). Recall that a possible such a condition is Exists , for a subquery. We then need to check whether the evaluation of over yields an empty graph. We inductively assume the existence of an efficient algorithm for checking this.

Finally, the application of the Construct clause on top and the set of bindings generated by the Match clause can be carried out in polynomial time. Intuitively, this is because the operations allowed in the Construct clause are defined by applying some simple aggregation and grouping functions on top of bindings generated by relational algebra operations.

Given that all evaluation steps of G-CORE have polynomial complexity in data size, we conclude that G-CORE is tractable.

5. Extensions of G-Core

Practical use of graphs often requires handling tabular data. This suggests that extending G-CORE with additional functionality for projecting tabular results or constructing graphs from imported tabular data may be useful.

Projecting tabular results. In order to integrate the results of graph matching into another system, it would be necessary or at least convenient to be able to produce a tabular projection from a query. It is quite straightforward to imagine the set of bindings produced by MATCH as a table, and use that to return a tabular projection. To achieve this, G-CORE could be extended with a SELECT clause for projecting expressions into a table. Such a tabular projection clause would also allow the introduction of other common relational operations for slicing, sorting, and aggregation, similar to Cypher’s RETURN clause or the SELECT clauses of SQL or SPARQL. Furthermore, SELECT could be used for adding various forms of expression-level subqueries, such as scalar subqueries or list subqueries.

Consider this example of a query that uses tabular projection:

57SELECT m.lastName + ’,  + m.firstName AS friendName
58MATCH (n:Person)-/<:knows*>/->(m:Person)
59    WHERE n.firstName = ’John’ AND n.lastName = ’Doe’
60      AND (n)-[:isLocatedIn]->()<-[:isLocatedIn]-(m)

This query matches persons with the name “John Doe” together with indirect friends that live in the same city and returns a table with the names of these friends.

It should be noted that the introduction of tabular projection into G-CORE changes the language to a multi-sorted language that is capable of either producing a table or a graph. Such a language would no longer be fully closed under graphs in a strict sense, which is one reason why this extension has been left to the future.

Importing tabular data. Conversely, integration of G-CORE with existing systems raises the question of how pre-existing tabular data could be processed in a pure graph query language. Next we present two different alternative proposals for how tabular data could be brought in to the graph world of G-CORE.

Binding table inputs. One way to import tabular data would be through the introduction of a new FROM <table> clause that would import sets of scalar bindings from a table, which could be used for defining a graph using the CONSTRUCT clause such as in this example:

61  (cust GROUP custName :Customer {name:=custName}),
62  (prod GROUP prodCode :Product {code:=prodCode}),
63  (cust)-[:bought]->(prod)
64FROM orders

This will construct a new graph from an input table of customer names custName and product codes prodCode by connecting per-customer and per-product nodes as given by the table.

Interpreting tables as graphs. Another alternative is to allow the MATCH .. ON .. to treat a tabular input following ON as a graph consisting of only isolated nodes that correspond to each row in the table. The properties of these nodes are the columns of the table and the values are the fields of the corresponding row.

If we express the previous example using this syntax, it would now look as follows:

65  (cust GROUP o.custName :Customer {name:=o.custName}),
66  (prod GROUP o.prodCode :Product {code:=o.prodCode}),
67  (cust)-[:bought]->(prod)
68MATCH (o) ON orders

6. Discussion and Related Work

Graph query languages have been extensively researched in the past decades, and comprehensive surveys are available. Angles and Gutierrez (90004) surveyed GQLs proposed during the eighties and nineties, before the emergence of current (practical) graph database systems. Wood (90524) studied GQLs focusing on their expressive power and computational complexity. Angles (90525) compares graph database systems in terms of their support for essential graph queries. Barceló (90852) studies the expressiveness and complexity of several navigational query languages. Recently, Angles et al. (AABHRV16) presented a study on fundamental graph querying functionalities (mainly graph patterns and navigational queries) and their implementation in modern graph query languages.

The extensive research on querying graph databases has not give rise yet to a standard query language for property graphs (like SQL for the relational model). Nevertheless, there are several industrial graph database products on the market. Gremlin (91037) is a graph-based programming language for property graphs which makes extensive use of XPath to support complex graph traversals. Cypher (91040), originally introduced by Neo4j and now implemented by a number of vendors, is a declarative query language for property graphs that has graph patterns and path queries as basic constructs. We primarily consider version 9 of Cypher as outlined by (Cypher18; Cypher9), while recognizing that Cypher is an evolving language where several advancements compared to Cypher 9 have already been made. Oracle has developed PGQL (RHKMC16), a graph query languages that is closely aligned to SQL and that supports powerful regular path expressions. Several implementations of PGQL, both for non-distributed (sevenich2016using) and distributed systems (roth2017pgx), exist. Here, we consider PGQL 1.1 (pgql11), which is the most recent version that is commercially available (obdsg).

G-CORE has been designed to support most of the main and relevant features provided by Cypher, PGQL, and Gremlin. Next we describe the main differences among G-CORE, Cypher, PGQL, and Gremlin based on the query features described in Section 3. Some features (e.g. aggregate operators) will not be discussed here as there are not substantial differences from one language to other.

Graph pattern queries. The notion of basic graph pattern, i.e. the conjunction of node-edge-node patterns with filter conditions over them, is intrinsically supported by Cypher, PGQL and G-CORE. Some differences arise regarding the support for complex graph patterns (i.e. union, difference, optional). Both Cypher and G-CORE define the UNION operator to merge the results of two graph patterns. The absence of graph patterns (negation) is mainly supported via existential subqueries. It is expressed in G-CORE, Cypher and PGQL with the WHERE NOT (EXISTS) clause. Optional graph patterns can be explicitly declared in G-CORE and Cypher with the OPTIONAL clause. PGQL does not support optional graph patterns, although they can be roughly simulated with length-restricted path expressions (see below). Although Gremlin is focused on navigational queries, it supports complex graph patterns (including branches and cycles) as the combination of traversal patterns.

Path queries. G-CORE, Cypher and PGQL support path queries in terms of regular path expressions (i.e. edges can be labeled with regular expressions). The main difference between Cypher 9 and PGQL is that the closure operator is restricted to a single repeated label / value. Both Cypher and PGQL support path length restrictions, a feature that although can be simulated using regular expressions, improves the succinctness of the language. Gremlin supports arbitrary or fixed iteration of any graph traversal (i.e. it is more expressive than regular path queries). Similar to Cypher, Gremlin allows specifying the number of times a traversal should be performed.

Query output. The general approach followed by Cypher 9 and PGQL is to return tables with atomic values (e.g. property values). This approach can be extended such that a result table can contain complex values. The extension in Cypher 9 allows returning nodes, edges, and paths. Recent implementations of Cypher have the ability to return graphs alongside this table (CAPS; CypherMultiGraph). Gremlin also supports returning complete paths as results. In contrast, G-CORE has been designed to return graphs with paths as first class citizens.

Query composition. With the output of a query in G-CORE being a graph, it follows naturally that queries can be composed by querying the output of one query by means of another query. Neither Cypher 9, PGQL or SPARQL supports this capability. Gremlin supports creating graphs and then populate them before querying the new graph. A notable parallel to G-CORE is the evolution of Cypher 10, where queries are composed through the means of “table-graphs”. Cypher 10 expresses queries with multiple graphs and a driving table as input, and produces a set of graphs along with a table as output. This allows Cypher 10 queries to compose both linearly and through correlated subqueries (Cypher18).

Evaluation semantics. There are several variations among the languages regarding the semantics for evaluating graph and path expressions. In the context of graph pattern matching semantics, G-CORE, PGQL, and Gremlin follow the homomorphism-based semantics (i.e. no restrictions are imposed during matching), and Cypher 9 follows a no-repeated-edge semantics (i.e. two variables cannot be bound to the same term in a given match) to prevent matching of potentially infinite result sets when enumerating all paths of a pattern. With respect to the evaluation of path expressions, G-CORE uses shortest-path semantics (i.e. paths of minimal length are returned), Cypher 9 implements no-repeated-edge semantics (i.e. each edge occurs at most once in the path), and Gremlin follows arbitrary path semantics (i.e. all paths are considered). Additionally, Cypher 9 and PGQL allow changing the default semantics by using built-in functions (e.g. allShortestPaths).

Expressive power versus efficiency. A balance between expressiveness and efficiency (complexity of evaluation) means a balance between practice and theory. Currently no industrial graph query language has a theoretical analysis of its complexity and, conversely, theoretical results have not been systematically translated into a design. One of the main virtues of G-CORE is that its design is the integration of both sources of knowledge and experience.

SPARQL and RDF. In this paper we concentrated on property graphs, but there are other data models and query languages available. A well-known alternative is the Resource Description Framework (RDF), a W3C recommendation that defines a graph-based data model for describing and publishing Web metadata. RDF has a standard query language, SPARQL (sparql10), which was designed to support several types of complex graph patterns (including union and optional). Its latest version, SPARQL 1.1 (sparql11), adds support for negation, regular path queries (called property paths), subqueries and aggregate operators. The path queries support reachability tests, but paths cannot be returned, nor can the cost of paths be computed. The evaluation of SPARQL graph patterns follows a homomorphism-based bag semantics, whereas property paths are evaluated using an arbitrary paths semantics (AABHRV16). SPARQL allows queries that return RDF graphs, however creating graphs consisting of multiple types of nodes (e.g., belonging to different RDF schema classes; having different properties) in one query is not possible as SPARQL lacks flexible graph aggregation: its CONSTRUCT directly instantiates a single binding table without reshaping. Such constructed RDF graphs can not be reused as subqueries, that is, for composing queries; nor does the language offer “full graph” operations to union or diff at the graph level. We think the ideas outlined in G-CORE could also inspire further development of SPARQL.

7. Conclusions

Graph databases have come of age. The number of systems, databases and query languages for graphs, both commercial and open source, indicates that these technologies are gaining wide acceptance (agens; neptune; arangodb; blazegraph; cosmos; dsegraph; RPBL13; janus; N17; sevenich2016using; orientdb; sparksee; stardog; tiger; titan).

At this stage, it is relevant to begin making efforts towards interoperability of these systems. A language like G-CORE could work as a base for integrating the manifold models and approaches towards querying graphs.

We defend here two principles we think should be at the foundations of the future graph query languages: composability, that is, having graphs and their mental model as departure and ending point and treat the most popular feature of graphs, namely paths, as first class citizens.

The language we present, G-CORE, which builds on the experiences with working systems, as well as theoretical results, show these desiderata are not only possible, but computationally feasible and approachable for graph users. This paper is a call to action for the stakeholders driving the graph database industry.


Appendix A A formal definition of G-CORE

In this section we formally present the semantics of G-CORE queries. For ease of presentation, we focus on a simplified syntax equal in expressive power to the full syntax of the language.

a.1. Basic notions

G-CORE queries are recursively defined by the top-down grammar presented in Section 4. In this section we define in detail the different components of the G-CORE grammar. But before doing so, it is important to introduce some basic notions that are used in the definition of their semantics.

Recall the domains used by the PPG data model as defined in Section 2. Let L be an infinite set of label names for nodes, edges and paths, K an infinite set of property names and V an infinite set of literals (i.e. actual values like integers).

Paths conforming to regular expressions. In graph query languages, one is typically interested in checking if two nodes are linked by a path whose label satisfies a regular expression , and computing one (or more) of such paths if needed. Some graph query languages. e.g., Cypher 9 (Cypher18), define the semantics of path expressions based on simple paths only (those without repetition of nodes and/or edges), which is known to easily lead to intractability in data complexity (MW95). For this reason, in G-CORE we follow a long tradition of graph query languages introduced in the last 30 years and define the semantics of path expressions based on arbitrary paths (see, e.g., (AABHRV16)). In addition, for the problem of computing a path from node to node that satisfies a given regular expression , we choose to compute the shortest such a path according to a fixed lexicographical order on nodes.444We acknowledge that using a fixed lexicographical order could be too restrictive when choosing a single path, so a system implementing G-CORE could use a different criterion based, for instance, on whether it can be evaluated more efficiently. The reason for this is that checking for the existence of an arbitrary path in a PPG from to that conforms to a regular expression , and computing the shortest path that witnesses this fact, can be done in polynomial time by applying standard automata techniques in combination with depth-first search. We define the notion of (shortest) paths conforming to regular expressions below.

We start by defining the notion of regular expression used in G-CORE. A regular expression is specified by the grammar:

where . Intuitively, an expression of the form either or , with , refers to an edge label, while refers to a node label. The expression is used as a wildcard that stands for “any label”. Notice that under this definition, the alphabet of every regular expression is a finite subset of .

Let be a PPG and be a path over . Then given a regular expression , we say that is a path from to conforming to if there exists a string in the regular language defined by such that:

  • For each , either or for some .

  • For each , either

    • , or

    • and for some , or

    • and for some .

The length of a path , written , is . Then is a shortest path from a node to a node conforming to a regular expression , if for every path from to that conforms to , it holds that .

Bindings. From now on, we assume that , and are countably infinite sets of node, edge and path variables, respectively, which are pairwise disjoint. Let be a PPG. A binding over is a partial function such that if , if , and if . The domain of a binding is denoted by , and it is assumed to be finite. Two bindings and are said to be compatible, denoted by , if for every variable , it holds that . Notice that if and are compatible bindings, then is a well-defined function.

Let , be finite sets of bindings over . The following four basic operations will be extensively used in this article:

Given a countably infinite set of value variables disjoint from , and , it is straightforward to extend our definition of bindings (and our presentation of the semantics of G-CORE) to capture assignments of literals to variables, i.e., such an extended binding would be a partial function such that if . For ease of presentation, however, we do not further consider such extended bindings in the sequel.

Expressions. We next define expressions according to the following grammar:

where is a variable, is property key, is a label, is a unary operator, is a binary operator, is a built-in function whose value only depends on its inputs and is an aggregation function. Unary operators are boolean negation Not, arithmetic negation , etc. Binary operators are comparisons such as , boolean operations such as And and Or, arithmetic operations on numbers and strings such as , etc. Built-in functions include the standard ones for type casting, string, date and collection handling (e.g. Size) known from other query languages, as well as graph-specific function such as Labels, which returns the set of labels, and Nodes, and Edges, which return the list of nodes and edges of , respectively, when applied to an object identifier . Aggregation functions include the standard ones inherited from relational query languages, Count, Min, Max, Sum, Avg, etc. and Collect to collect all values of the group into a collection.

The semantics of expressions inherited from other query languages is defined in analogy with them, so we do not repeat it here. We denote the evaluation of any such an expression  over a PPG and a set of bindings as . In any such a case, (recall that V includes truth values and ). The semantics of the G-CORE specific expressions, on the other hand, is defined as follows (where we denote by the singleton binding set ):

  • If is a variable , then .

  • If , then .

  • If , then iff iff .

  • If , then .

  • If , then .

  • If , then .

  • If , then iff , assuming that .

a.2. The MATCH clause

Basic graph patterns. A basic graph pattern is specified by the following grammar:

where , , , and is a regular expression.

Let be a basic graph pattern. The evaluation of over a PPG , denoted by , is inductively defined:

  • If is a node pattern , then .

  • If is an edge pattern , then .

  • If is a path pattern , then , , and is the shortest path from to