1 Introduction
An important area of complex systems research is that of temporal networks [10]. Temporal networks differ from static networks in that edges are timestamped according to when the link occurred in the network. One example is a cyber network, in which nodes may be IP addresses and the edges represent packets of data that transferred between them. Another common example are social networks, where the edges can represent communications or interactions that occur between people at different points in time. Such networks are typically treated as multidigraphs, where edges are directed and multiple edges can exist between pairs of nodes. By including the temporal information, an analyst can hope to have a more informed understanding of the network than they would if they ignored the temporal ordering of edges.
One popular method for trying to analyze both static and temporal networks is subgraph pattern matching
[18]. Given a graph and a smaller graph of interest , we attempt to find all matching subgraphs that are isomorphic to . By doing so, we hope to find a set of interactions that match a particular pattern of interest. An example might be a graph that represents a specific molecular pattern in a chemical network, or a graph that represents the behavior of a piece of malware in a cyber network. Such subgraph matching often involves datasets with attributes on either the nodes or the edges, which can reduce the number of matches.In some use cases, only the total count of matching subgraphs is needed. In this usage, the subgraph patterns we seek to find are typically referred to as network motifs, and the total count of each pattern is used as a way of characterizing the network as a whole [11]. It is also possible to use the count of matching subgraphs as a way of characterizing individual nodes in a network, based on the number of matching subgraphs each node lies on. Network motif analysis has found a wide range of uses including analyzing biological networks [16], social networks [4] and computational graphs [19].
The subgraph isomorphism problem has long been known to be NPcomplete [1], however it can often be computed relatively efficiently in practice. The exponential performance of the subgraph matching algorithm is usually only exponential in regards to the number of nodes or edges in . This means searching for a small subgraph can actually be quite efficient in many cases, particularly if node and edge attributes are used to restrict the number of potential matches.
While a number of techniques exist for finding matches in static networks, most do not take the temporal order of the edges into account when finding instances of in . However, the ordering of these edges be essential to the meaning of the query graph or motif. A simple example is that of a sequential cycle, in which each edge in the cycle occurs before the one following it. If time is not taken into account, nonsequential cycles may be found instead, which could have completely different meaning given the use case. For an example see Figure 1, in which we are looking for a sequentially ordered 3cycle in our graph. While a number of 3cycles exist, only one matches this ordering criteria correctly.
In addition to a simple temporal ordering restriction, it can also be useful to add a restriction on the length of time permitted between edges in the subgraph. This is referred to as a temporal motif [13]. In a temporal motif approach, all edges in the matching subgraph must occur within a period of time units. This added restriction prevents us from including edges that may have occurred long after the initial edges in our subgraph. This can improve both the usefulness of the results as well as the performance time required to find the matching subgraphs. See Figure 2 as an example of a sequential 3cycle, with hour. In this case the cycle between A, B and C would not be included, despite having the correct ordering, due to the edge between C and A occurring several hours later.
In this paper we present a new algorithm for finding matching subgraphs in a temporal network that inherently takes the ordering of the edges into account. We do this through a chronologically sorted edgedriven approach that only searches through edges that occur in the correct order. By doing so, we can have a significantly more efficient approach for general temporal subgraph matching than was previously possible.
2 Related Work
Subgraph matching algorithms have existed since at least the time of the publication of Ullmann’s algorithm in 1976 [18]. Since then, many modifications have been made to this approach to improve its performance. One of the most commonly used is known as the VF2 algorithm [3], itself an extension of the VF algorithm [2] (named for two of its authors, Vento and Foggia). While this approach can be used to find patterns in static networks, by default it ignores the chronological ordering of edges, and only expects them to match a particular topological configuration. While it may be possible to add edge restrictions to enforce some temporal requirements in how the subgraphs are matched, it is not a natural part of the algorithm.
Motif analysis on temporal networks is a much more recent area of research. Kovanen et al. provide an algorithm for finding temporal subgraphs, but have the added restriction that edges in the motif must represent consecutive events for the node [7]. While this allows for fast subgraph counting, it does not allow subgraph matching on graphs that have other edges occurring in between the edge events of a motif. Another approach was published by Gurukar et al. [6]
. The authors present a heuristic for counting temporal motifs, but provide only an estimated count with their approach, and do not exhaustively find all exact temporal subgraph matches.
Another closely related work is that of Redmond and Cunningham [14] [15]. Their definition of temporal subgraph isomorphism has some similarities to our own, but also has some distinct differences. The most significant difference is that the query graphs do not have their edges timestamped. Instead, a subgraph is considered to be a temporal match if all of the edges occur within a particular and incoming edges occur before outgoing edges. Since their query graphs do not have any particular ordering to their edges (beyond that incoming edges must occur first), there is no way to directly compare the results of their algorithm to our own.
Closer in spirit to our work is the recent publication by Parnajape, Benson and Leskovec [13]. Their research uses an equivalent definition of temporal subgraph matching to our own. Their algorithmic approach, however, is quite different. When it comes to the general subgraph matching use case, their approach is to first perform a static subgraph match to find potential temporal matches. After these potential matches are found, a separate algorithm is used to find all temporal subgraph matches from these potential matches. While the running time of this secondary algorithm is linear, it requires all potential matches to be found first. This means that in some cases significantly more subgraphs will be initially found than needed, slowing down the overall performance of the algorithm. In theory, this difference could be exponential, requiring exponentially more time than an approach that finds only the true temporal matches would. A particularly bad example would be looking for a temporal motif in a complete graph, where only one temporal match existed. Without any additional restrictions, an exponential number of subgraphs would be returned by the intial static subgraph matching phase, despite there be only one actual temporal subgraph match.
In addition to their general temporal subgraph matching algorithm, the authors also present a faster algorithms for counting 2node, 3edge star and triangle temporal motifs [13]. These approaches are significantly faster than their approach for general temporal subgraph matching, but have the downside of working for only a particular set of graph patterns. While these differ significantly from both our approach and earlier static approaches, it is similar to ours in regards of being chronologically edgedriven, with the edges iterated over in chronological order to find the matching subgraphs. Their timing experiments show them to be very efficient at finding 2node, 3edge star and triangle temporal motifs. While it is common to perform motif analysis on small graphs such as these, there are many known cases where larger motifs are needed or useful [19, 20, 21]. The need to support efficient temporal subgraph matching on larger query graphs was a main factor driving our own algorithm research.
3 Example Application
To give an example of the usefulness of temporal graph isomorphism, we will demonstrate how a targeted temporal subgraph query can produce practical results from a publicly available dataset, and how the results vary significantly if the temporal ordering is ignored.
The data used was collected from the Carnegie Mellon CERT Insider Threat Tools dataset, a synthetically generated insider cyber test dataset [5]. The data represents a fictitious company, with the nodes representing the employees, their computers, files, removable media and websites they visited. Edges represent actions connecting these nodes together (e.g., download a file, log in to a PC, etc.). For our analysis, we have selected data taking place in the month of April 2011, which represents 315K nodes with 21.8M temporal edges.
The particular insider threat we have focused on is the case where an employee is repeatedly stealing information from their coworkers by logging into their machines, opening files, and sending them to his home email address. A motif that may be indicative of such a threat is shown in Fig 3. While not all actions matching this query would necessarily indicate malicious behavior, observing an employee with an unusually high rate of matching subgraphs may indicate closer observation may be needed. For our use case, we have selected a = 1 hour, as we are expecting this pattern to represent actions that occur in a relatively short period of time. When searching for this temporal query graph in our dataset, we find the exact employee we were expecting (CDE1846), who was in fact the perpetrator in the dataset.
We also compared how searching for this same motif with an unordered static subgraph isomorphism algorithm would effect the results. The results of our experiments can be seen in Table 1. The first column represents the subgraph isomorphism technique being used. The second column has the value of (where appropriate). The third column lists the number of employee nodes that were found on matching subgraphs. The fourth column tells the rank of the perpetrator node in our search. A value of 1 means the perpetrator was found on the largest number of matching subgraphs.
Isomorphism  # Matches  Rank  

Temporal  1 hr  1  1 
Temporal  2 hr  8  1 
Temporal  4 hr  68  1 
Temporal  8 hr  298  1 
Temporal  1 day  600  4 
Temporal  2 days  721  47 
Temporal (Alt. Order)  1 hr  2  1 
Temporal (Alt. Order)  2 hr  28  1 
Temporal (Alt. Order)  4 hr  343  1 
Temporal (Alt. Order)  8 hr  1,053  7 
Temporal (Alt. Order)  1 day  2,247  24 
Temporal (Alt. Order)  2 days  2,248  132 
Static  NA  2,297  410 
While our targeted temporal search gave us the exact correct answer, a static search gave back a much larger number of employees lying on matching subgraphs. A total of 2,297 matching nodes were found, with the perpetrator having a rank of only 410 in the list. There are actually a number of reasons why temporal order is so important in this use case. One important factor is the use of the value. By restricting the time to one hour, we are describing a particular set of actions: one that occurs in a relatively short amount of time. If we make this value too high, we can get inferior results, even with the correctly temporal ordering. When this same pattern occurs over a period of several hours, it may represent a normal employee going about his normal daily business, instead of the pattern we are actually looking for, which is someone logging into multiple PCs quickly to pilfer files from their coworkers. As can be seen in Table 1, as the value of increases, we get increasingly more employees found in matching subgraphs, despite the fact these employees were innocent. When increases to the size of 1 day, the ranking of the perpetrator drops as well.
Another important factor is that there is a different meaning depending on the order of these events/edges. A user who is opening a file after sending it suggests he is already familiar with the contents. A user who opens it first suggests someone who is unsure of the contents, and may be looking for something in particular before sending it. While the difference between these actions may be subtle, they do represent actual differences in behavior. As an example, if we flip the order of these events, we get significantly worse results, as can be seen in the rows marked Temporal (Alt. Order). Not only is there an increasing number of matching employees, but we also begin to see the actual perpetrator drop in rank as increases.
4 Approach
While our approach could be modified to do simple subgraph counting, our desire was to return a set of all instances of subgraphs that were isomorphic to . We also wanted an algorithm that was not limited to any particular size or structure of query graph or motif. The heart of the algorithm is described in the function TemporalMatch in Algorithm 1 as well as the related subroutine FindNextMatch listed in Algorithm 2. Some important bookkeeping is necessary to make sure the nodes and edges of the graph are being matched correctly:

: A list of edges , sorted chronologically by time stamp.

: A list of edges , sorted chronologically by time stamp.

map: A mapping of nodes in to nodes in . In practice, this is an array of length . If a node is not assigned to node in , it will have a value of 1.

map: A mapping of nodes in to nodes in . In practice, this is an array of length . If a node is not yet assigned to node in , it will have a value of 1.

edgeCount: A mapping of nodes in to the number of adjacent edges that have been mapped to them for our motif . When this value drops to zero, that vertex is free to be reassigned to another node in if necessary.

eStack: A stack used to keep track of what edges in are currently mapped to edges in . They are always mapped in chronological order, so that the first one will map to the first edge in .

: The latest time an edge can have in our matching subgraph, given the time for the first edge and . This is initially infinity until the first edge is added. It gets reset when eStack is empty.

results: A set of matching subgraphs to return at the completion of the algorithm.
The algorithm works by iterating over each edge index in in chronological order. Before the next edge in can be reached, a matching edge in must be found. (Note: and are integer indices into the sorted list of edges and , and not actual edge structures). If an appropriate is found, we map the source and destination nodes accordingly, and continue on to find a match for the next edge in . Once the remaining edges in are accounted for, we can add that subgraph to our results list, and continue on to find an alternative edge in that can also map to the last edge . If we iterate over all remaining edges before all edges in are accounted for, we must pop the last matched edge from our stack, and try again from that point in the list of edges.
To determine which edge in we can try next, we have the FindNextMatch subroutine specified in Algorithm 2. This takes into account whether or not either the source or destination node have already been mapped, as well as the sequential ordering of the most recently mapped edge. If these are not taken into account, the performance of the algorithm may drastically slow down. Other attribute based approaches may also be used to help improve this narrowing process as well.
4.1 Time Complexity
A worst case computational time complexity can be calculated to be . This would occur when each edge in could be matched to each edge in , and the value of exceeds that of the time range for . An example would be a complete graph with an exponential number of subgraphs isomorphic to the query graph. For every edge in we could expect it to match to every edge in subsequent in time to the selected edge. Even with the temporal restriction that the edges must be subsequent to each other, we can still have an exponential asymptotic complexity, since the sum of a linearly decreasing series is still exponential (e.g., ).
In practice, we can expect the runtime to be typically less than this. If the number of edges expected within is a constant , we can calculate the time complexity to be . The proof for this is straight forward: In a worst case scenario (where each edge in is a match for every edge in ) at most edges need to be visited for every edge in . In realworld scenarios, we can actually expect performance to be even better than this, since in most situations not every edge should be a match. This also helps to shed light on how the value of can have such a large impact in the overall performance time.
In comparison to existing approaches, using VF2 to perform static subgraph matching has a worst case computational time complexity of and a best case of [3]. To perform the additional steps needed to perform the general temporal subgraph matching of Paranjape et al., there is an additional time of , where is the static subgraph induced by and is the temporal edges between pairs of nodes in [13]. As the values of and tend to be very small, the authors consider this value to be essentially a constant, and consider the performance of this secondary step linear in regards to . As long as the number of potential temporal edge matches () does not exceed , we can consider the overall time complexity of the approach of Paranjape et al. to be asymptotically equivalent to VF2.
5 Timing Experiments
Determining which method should work faster analytically is a challenge, as the performance of both approaches is exponential in a worst case scenario, and many factors can come into play in determining the exact time complexity. Instead, we have opted to do a number of timing experiments comparing the performance time of our algorithm against that of VF2. As this is actually just the first step in the previously published approach, in practice the actual time required should be longer. However, we believe this should give us a decent baseline to compare our performance against, as the secondary step of Paranjape et al. is performed in linear time, and the implementation of their general temporal subgraph isomorphism algorithm is currently unavailable to us.
Graph  Nodes  Static Edges  Edges  Time Span 

CollegeMsg  1.9K  20.3K  60K  193 days 
EmailEu  986  2.5K  332K  2.2 years 
MathOverflow  24.8K  228K  507K  6.5 years 
Enron  82.2K  322K  1.15M  4 years 
For our experiments we have selected four temporal network datasets from the SNAP graph library [9] and Koblenz Network Collection (KONECT) [8]. Information about each of the datasets can be seen in Table 2. CollegeMsg represents a set of private messages between users of an online social network at University of California, Irvine [12]. EmailEu represents a collection of emails between members of an anonymous European research institution [13]. MathOverflow represents comments on a popular online messageboard on mathematics, where an edge represents a comment between users [13]. Enron contains edges representing the email traffic between employees of the Enron corporation between 1999 and 2003 [8]. Each networks has its own unique topological characteristics, which can be seen to some extent by the varying results from the temporal motif counting in Table 3.
Static  = 1 hour  = 1 day  

Graph  Motif  Subgraphs  Time  Subgraphs  Time  Speed Up  Subgraphs  Time  Speed Up 
CollegeMSG  876  18.8  3.46K  0.546  34.5x  63.2K  6.24  3.01x  
CollegeMSG  238K  43.2  75.4K  2.04  21.1x  1.46M  34.3  1.26x  
CollegeMSG  1.57K  19.3  1.06K  0.499  38.6x  53.5K  6.16  3.12x  
CollegeMSG  48  2.18  0  0.296  7.38x  173K  1.28  1.71x  
CollegeMSG  202M  9.88K  1.28M  35.1  281x  49.7M  1.34K  7.36x  
CollegeMSG  10.9M  2.02K  227K  6.74  300x  16.2M  435  4.66x  
EmailEu  3.96K  33.5  258  1.14  29.4x  129K  12.9  2.60x  
EmailEu  367K  91.4  30.1K  1.80  50.9x  2.40M  59.4  5.43x  
EmailEu  12.0K  40.1  493  1.15  34.7x  276K  16.8  6.22x  
EmailEu  554  4.99  23  1.09  4.57x  49.2K  6.44  0.775x  
EmailEu  57.7M  7.18K  4.71M  131  54.9x  144M  1.75K  4.10x  
EmailEu  7.08M  1.54K  31.1K  2.45  627x  6.89M  200  7.68x  
Static  = 1 day  = 1 week  
Graph  Motif  Subgraphs  Time  Subgraphs  Time  Speed Up  Subgraphs  Time  Speed Up 
MathOverflow  1.42M  2.94K  2  0.001  2.94Mx  531  0.047  62.5Kx  
MathOverflow  218M  11.4K  248  0.015  760Kx  9.06K  0.218  52.3Kx  
MathOverflow  5.53M  4.02K  4  0.001  4.02Mx  684  0.047  85.5Kx  
MathOverflow  285K  2.34K  0  0.015  156Kx  180  0.015  156Kx  
MathOverflow  —  hr  805  0.031  2.79Mx  214K  5.897  14.7Kx  
MathOverflow  —  hr  1.70K  0.062  1.39Mx  72.0K  2.059  42.0Kx  
Enron  246K  1.21K  2  0.046  26.3Kx  125  0.156  7.77Kx  
Enron  119M  4.51K  1.21K  0.078  57.9Kx  22.7K  0.656  6.88Kx  
Enron  1.71M  1.62K  2  0.047  34.4Kx  274  0.156  10.4Kx  
Enron  33.3 K  55.6  0  0.047  1.18Kx  14  0.109  510x  
Enron  —  hr  12.6M  333  259x  217M  6.05K  14.3x  
Enron  —  hr  2.33K  0.28  309Kx  112K  5.57  15.5Kx 
The motifs used can be seen in Fig 4. These represent six different temporal subgraph patterns, between 4 and 5 nodes in size. These were selected in part as they cannot currently be found by the more efficient algorithms of Paranjape et al., which only work for 2node, triangle and 3edge star motif patterns [13]. We chose query graphs which we believed would give a wide range of results on our test datasets. We performed our experiments using 1 hour and 1 day for CollegeMSG and EmailEu datasets, and 1 day and 1 week for the MathOverflow and Enron datasets. The latter was necessary due to the longer periods of time between interactions in these networks.
To give an idea of the efficiency of our approach over that of using a static subgraph matching algorithm, we have calculated the following for each of our motifs:

The number of matching static subgraphs.

The time in seconds to find the matching static subgraphs with the Boost implementation of VF2. [17]

The number of matching temporal subgraphs.

The time in seconds to find the matching temporal subgraphs with our chronological edgedriven algorithm.
All algorithms were implemented in C++ and executed on a single thread of a 1.80 GHz Intel Xeon E52603 CPU with 64 GB of memory. The Boost graph library implementation of the VF2 algorithm was used to perform the static subgraph matching in our experiments [17]. Timings include only the time to perform subgraph searches, and not input or output times. The results of our experiments can be seen in Table 3. In some cases the time taken for VF2 to complete took over 24 hours. In these situations we stopped the algorithm, and simply recorded the time as taking 24 hours.
Both the subgraph counts and performance times lead to some intriguing results. In some cases, the difference between the number of static subgraphs and temporal subgraphs can be very dramatic. The most extreme case was that of the MathOverflow graph, which had around 285,000 static subgraphs matching , but none matching that pattern when time was taken into account for values of 1 day.
In other cases, the number of temporal subgraphs was actually higher than the number of static subgraphs. This is due to the fact that we were merged parallel edges when computing static subgraph isomorphism with VF2. In some of these cases our approach was still significantly faster. Examples include the CollegeMSG graph, when 1 day, for motifs , , , , and . In each case, the number of temporal isomorphic subgraphs was significantly higher than the number of static isomorphic subgraphs, and yet there was still speed up between 1.26x and 4.66x. We see a similar pattern with the EmailEu datasets with day as well. In all but one case, our approach was faster than that of VF2, despite having more isomorphic subgraphs.
Generally speaking, increasing does decrease the performance of our approach. In fact, when searching for the motif with day in the EmailEu graph, we actually found our algorithm to be slower than VF2. We suspect this would continue to be the case for many of the datasets as continues to increase, although it can depend on many factors about the graph, such as the number of parallel edges between nodes, and the range of time represented in the network.
For the most part, the performance of our algorithm was vastly faster than that of VF2 for the given motifs. This was especially true for the largest of the graphs, the MathOverflow and Enron datasets. For these graphs, when 1 day, we had speed up between 1000 and over 4 million times that of the VF2 algorithm, with 4 of the motifs for the MathOverflow producing over 1 million times speed up. While the speed up decreased when = 1 week, we still saw speed ups between around 14x to over 100,000x that of VF2, with most motifs giving a speed up in the thousands.
To gain a better understanding of how the number of matching subgraphs effects the speed up of the algorithm, we created a chart plotting the ratio of static subgraphs over temporal subgraphs on the xaxis, with the amount of speed up on the yaxis. The resulting loglog plot can be seen in Fig 5. While there is some fluctuation in the results, there is a clear positive correlation between the ratio of matching subgraphs and the speed up of the algorithm. While there are undoubtedly a number of factors that impact the overall performance, the reduction in number of matching subgraphs seems to be one of the biggest factors in our algorithm’s performance improvement over using VF2.
6 Conclusion and Future Work
As can be seen in from the results of our experiments, a significant speed up can be had in many cases by applying a chronological edgebased matching approach to the temporal subgraph isomorphism problem. Not surprisingly, the data also shows that the choice of can have a significant impact on the performance as well. In situations where grows large, other techniques for temporal subgraph matching may prove to be more efficient. This ultimately seems to depend heavily on the total number of matching subgraphs. If the number of temporal isomorphic subgraphs greatly exceeds the number of static isomorphic subgraphs, it may end up being more efficient to follow the techniques that apply the static subgraph matching first, and filter for temporal matches second. Further study will be needed to see if the secondary filtering technique of Paranjape et al [13] is efficient enough to outperform our algorithm in these situations.
One of the fascinating differences between temporal and static subgraph matching is that the number of matching temporal graphs can be either greater or lesser than the number of static graphs. Despite the fact that a number of the experiments produced more matching subgraphs for the temporal approach, the performance was still usually faster than the traditional static matching method. It was only when the difference became two orders of magnitude greater that we saw a slow down of our approach over that of VF2. We believe this advantage of our algorithm may in part be due to the fact that less of the search space is required to be traversed in the temporal matching process, even if it means more subgraphs are matched overall. By only visiting edges in chronological order, we can often ignore a large portion of the edges during the matching process, improving the overall time complexity.
While smaller query graphs may suffice for many uses of motif analysis, we believe the ability of our approach to generalize to larger subgraphs could be useful to many. It also shows that a more efficient general approach to the temporal subgraph matching problem is possible. By performing the matching against the edges and not the nodes, we can take the ordering of the edges into account and match only those that are in the correct chronological sequence. By making these algorithmic improvements, we can enable a speed up of performance by the thousands or even millions in some cases.
Subgraph matching and motif analysis have become an increasingly powerful tools for those working in data mining and graph analytics. While the problem remains NPcomplete, great gains can still be made in practice for many realworld datasets. When it comes to query graphs of four or more nodes, we believe our approach to general temporal subgraph matching should be more efficient in most cases than previously published techniques. By taking advantage of the chronological ordering of edges, larger temporal motifs and query graphs should no longer be off limits from being used in temporal network analysis.
7 Acknowledgements
The research described is part of the Analysis in Motion (AIM) Initiative at Pacific Northwest National Laboratory (PNNL). It was conducted under the Laboratory Directed Research and Development Program at PNNL, a multiprogram national laboratory operated by Battelle for the U.S. Department of Energy under Contract DEAC0576RL01830.
References

[1]
S. A. Cook, The complexity of theoremproving procedures
, in Proceedings of the third annual ACM symposium on Theory of computing, ACM, 1971, pp. 151–158.
 [2] L. P. Cordella, P. Foggia, C. Sansone, and M. Vento, Performance evaluation of the vf graph matching algorithm, in Image Analysis and Processing, 1999. Proceedings. International Conference on, IEEE, 1999, pp. 1172–1177.
 [3] , A (sub) graph isomorphism algorithm for matching large graphs, IEEE transactions on pattern analysis and machine intelligence, 26 (2004), pp. 1367–1372.
 [4] K. Faust and J. Skvoretz, Comparing networks across space and time, size and species, Sociological methodology, 32 (2002), pp. 267–299.
 [5] J. Glasser and B. Lindauer, Bridging the gap: A pragmatic approach to generating insider threat data, in Security and Privacy Workshops (SPW), 2013 IEEE, IEEE, 2013, pp. 98–104.
 [6] S. Gurukar, S. Ranu, and B. Ravindran, Commit: A scalable approach to mining communication motifs from dynamic networks, in Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, ACM, 2015, pp. 475–489.
 [7] L. Kovanen, M. Karsai, K. Kaski, J. Kertész, and J. Saramäki, Temporal motifs in timedependent networks, Journal of Statistical Mechanics: Theory and Experiment, 2011 (2011), p. P11005.
 [8] J. Kunegis, KONECT: the Koblenz network collection, in Proceedings of the 22nd International Conference on World Wide Web, ACM, 2013, pp. 1343–1350.
 [9] J. Leskovec and A. Krevl, SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, June 2014. Accessed: 20170801.
 [10] N. Masuda and R. Lambiotte, A guide to temporal networks, vol. 4, World Scientific, 2016.
 [11] R. Milo, S. ShenOrr, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon, Network motifs: simple building blocks of complex networks, Science, 298 (2002), pp. 824–827.
 [12] P. Panzarasa, T. Opsahl, and K. M. Carley, Patterns and dynamics of users’ behavior and interaction: Network analysis of an online community, Journal of the Association for Information Science and Technology, 60 (2009), pp. 911–932.
 [13] A. Paranjape, A. R. Benson, and J. Leskovec, Motifs in temporal networks, in Proceedings of the Tenth ACM International Conference on Web Search and Data Mining, ACM, 2017, pp. 601–610.
 [14] U. Redmond and P. Cunningham, Temporal subgraph isomorphism, in Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ACM, 2013, pp. 1451–1452.
 [15] U. Redmond and P. Cunningham, Subgraph isomorphism in temporal networks, arXiv preprint arXiv:1605.02174, (2016).
 [16] S. S. ShenOrr, R. Milo, S. Mangan, and U. Alon, Network motifs in the transcriptional regulation network of escherichia coli, Nature genetics, 31 (2002), p. 64.
 [17] J. Siek, L.Q. Lee, A. Lumsdaine, et al., Boost graph library, Pearson India, 2002.
 [18] J. R. Ullmann, An algorithm for subgraph isomorphism, Journal of the ACM (JACM), 23 (1976), pp. 31–42.
 [19] S. Valverde and R. V. Solé, Network motifs in computational graphs: A case study in software architecture, Physical Review E, 72 (2005), p. 026107.
 [20] E. YegerLotem, S. Sattath, N. Kashtan, S. Itzkovitz, R. Milo, R. Y. Pinter, U. Alon, and H. Margalit, Network motifs in integrated cellular networks of transcription–regulation and protein–protein interaction, Proceedings of the National Academy of Sciences of the United States of America, 101 (2004), pp. 5934–5939.
 [21] L. V. Zhang, O. D. King, S. L. Wong, D. S. Goldberg, A. H. Tong, G. Lesage, B. Andrews, H. Bussey, C. Boone, and F. P. Roth, Motifs, themes and thematic maps of an integrated saccharomyces cerevisiae interaction network, Journal of biology, 4 (2005), p. 6.