Methods for handling both uncertainty and complex relational structure have received much attention in machine learning. For instance, in composite event recognition(Cugola and Margara, 2012; Artikis et al, 2012; Alevizos et al, 2017), relations are defined over entities of actors and objects involved in an event. Such applications are typically characterised by uncertainty, and in many cases data of significant volume and velocity. Manual derivation of relational dependencies is a time-consuming process and, in the presence of large data streams, unrealistic.
One of the logic-based representations that handles uncertainty is Markov Logic Networks (MLNs) (Richardson and Domingos, 2006) that combine first-order logic and probabilistic graphical models. Online structure learning approaches for MLNs have been effectively applied to a variety of tasks (Michelioudakis et al, 2016b; Huynh and Mooney, 2011). Although these approaches facilitate the automated discovery of multi-relational dependencies in noisy environments, they assume a fully labelled training sequence, which is unrealistic in most real-world applications.
We propose a novel method for completing the supervision, using the graph-cut minimisation technique (Zhu et al, 2009) and a distance function for first-order logic. Graph-cut minimisation essentially derives labels for unlabelled data, by computing their distance to their labelled counterparts. In particular, we adapt the graph-cut minimisation approach proposed by Zhu et al (2003) to first-order logic, in order to operate over logical structures instead of numerical data. To do so, we use a structural measure (Nienhuys-Cheng, 1997), designed to compute the distance between logical atoms, and modify it using the Kuhn-Munkres algorithm (Kuhn, 1955), to accurately calculate the distance over sets of logical atoms that represent the training examples.
The proposed supervision completion method operates in an online fashion (single-pass), by means of a caching mechanism that stores previously seen labels for future usage. The Hoeffding bound, a statistical tool that enables approximate globally-optimal decisions from locally-optimal ones, is used to filter out contradicting labels that may compromise the labelling accuracy. The completed training data can be subsequently used by any supervised structure learner. To demonstrate the benefits of SPLICE, our proposed method to semi-supervised online structure learning, we focus on composite event recognition (CER), by employing the OSL (Michelioudakis et al, 2016b) and OLED (Katzouris et al, 2016) online structure learners. Both of these learners can construct Event Calculus theories (Kowalski and Sergot, 1986; Mueller, 2008; Skarlatidis et al, 2015) for CER applications.
In CER, the goal is to recognise composite events (CEs) of interest, given an input stream of simple derived events (SDEs). CEs can be defined as relational structures over sub-events, either CEs or SDEs, and capture the knowledge of a target application. The proposed method (SPLICE) is evaluated on the task of activity recognition from surveillance video footage, as well as in maritime monitoring. In the former case, the goal is to recognise activities taking place between persons, e.g., people meeting or moving together, by exploiting information about observed activities of individuals. In maritime monitoring, the goal is to recognise vessel activities, by exploiting information such as vessel speed, location and communication gaps. Our empirical analysis suggests that our approach is capable of completing the supervision, even in the presence of little given annotation, and eventually enhances the accuracy of the underlying structure learner.
In summary, the main contributions of this paper are:
An online supervision completion method using a caching mechanism to store labelled examples for future usage, and the Hoeffding bound to filter out contradicting examples that may compromise the overall accuracy.
An adaptation of the graph-cut minimisation technique to first-order logic, using a structural distance for comparing logical atoms, and the Kuhn-Munkres algorithm for improving the accuracy of the distance calculation.
The first system for semi-supervised online structure learning combining online supervision completion and two state-of-the-art structure learners, in order to learn Event Calculus definitions for CER.
An evaluation of the combined system on two real (non-synthetic) datasets concerning activity recognition and maritime monitoring. In the former case, we use a benchmark video surveillance dataset that includes manually constructed ground truth. In the latter case, we use a dataset comprising vessel position signals from the area of Brest, France, spanning one month.
The remainder of the paper is organised as follows. Section 2 provides the required background for the proposed method and Section 3 describes our approach to semi-supervised online structure learning. Section 4 reports the experimental results on both datasets. Section 5 discusses related work on semi-supervised structure learning and alternative distance measures for logical representations, while Section 6 concludes and proposes directions for future research.
We present existing methods that are employed in the rest of the paper. We begin by briefly presenting the Event Calculus, as well as the basic functionality of OSL and OLED structure learners. Then, in Section 2.2 we describe the ideas behind graph-cut minimisation and a variation based on the harmonic function. Finally, in Section 2.3 we discuss a distance function used for comparing sets of logical atoms and one of its drawbacks that we overcome using the Kuhn-Munkres algorithm.
2.1 Event Calculus and Structure Learning
One way of performing CER is by using the discrete Event Calculus (DEC) (Mueller, 2008). The ontology of DEC consists of time-points, events and fluents. The underlying time model is linear and represented by integers. A fluent is a property whose value may change over time by the occurrence of particular events. DEC includes the core domain-independent axioms of the Event Calculus, which determine whether a fluent holds or not at a specific time-point. This axiomatisation incorporates the common sense law of inertia, according to which fluents persist over time, unless they are affected by an event occurrence. Event occurrences are denoted by the predicates, while predicates denote whether a fluent holds. The and predicates express the conditions in which a fluent is initiated or terminated, and are triggered by predicates. The core DEC axioms are defined as follows:
Variables and functions start with a lower-case letter, while predicates start with an upper-case letter. Axioms (1) and (2) express when a fluent holds, while axioms (3) and (4) denote the conditions in which a fluent does not hold. In CER, as we have formulated it here, the truth values of the composite events (CE)s of interest — the ‘query atoms’ — are expressed by means of the predicate. The incoming ’simple, derived events’ (SDE)s are represented by means of , while any additional contextual information is represented by domain-dependent predicates. The SDEs and such contextual information constitute the ‘evidence atoms’. This way, CEs may be defined by means of and predicates, stating the conditions in which a CE is initiated and terminated.
In order to learn Event Calculus theories, online structure learning methods may be employed in order to efficiently learn in the presence of data streams. OSL (Michelioudakis et al, 2016b) is an online structure learner, based on Markov Logic Networks (MLNs) (Richardson and Domingos, 2006), that can learn (Skarlatidis et al, 2015) definitions — a probabilistic variant of DEC — by adapting the procedure of OSL (Huynh and Mooney, 2011) and exploiting a given background knowledge. In particular, OSL exploits the axioms to constrain the space of possible structures during search. Each axiom contains predicates, that consist the supervision, and , predicates, that form the target CE definitions that we want to learn. OSL creates mappings from atoms to , atoms and searches only for explanations of the latter. Upon doing so, OSL only needs to find appropriate bodies over the current time-point to form clauses. Each incoming training example is used along the already learned clauses to predict the truth values of the . Then, OSL constructs a hypergraph that represents the space of possible structures as graph paths. For all incorrectly predicted CEs the hypergraph is searched, using relational path-finding (Richards and Mooney, 1992), for clauses supporting the recognition of these incorrectly predicted CEs. The paths discovered during the search correspond to conjunctions of true ground evidence atoms (SDEs and contextual information) and are used to form clauses. The weights of the clauses that pass the evaluation stage are optimised using the AdaGrad online learner (Duchi et al, 2011).
OLED (Katzouris et al, 2016)
is based on Inductive Logic Programming, constructing CE definitions in the Event Calculus, in a single pass over the data stream. OLED constructs definitions by encoding each positive example, arriving in the input stream, into a so-calledbottom rule, i.e., a most specific rule of the form , where is an or atom and are relational features (e.g., SDEs). A bottom clause is typically too restrictive to be useful, thus, OLED searches the space of all possible rules that -subsume the bottom rule. To that end, OLED starts from the most-general rule and gradually specialises that rule, in a top-down fashion, by adding
’s to its body and using a rule evaluation function to assess the quality of each specialisation. OLED’s single-pass strategy draws inspiration from the VFDT (Very Fast Decision Trees) algorithm(Domingos and Hulten, 2000) which is based on the Hoeffding bound, a statistical tool that allows to approximate the quality of a rule on the entire input using only a subset of the data. Thus, in order to decide between specialisations, OLED accumulates observations from the input stream until the difference between the best and the second-best specialisation satisfies the Hoeffding bound.
Both OSL and OLED have shortcomings. OLED is a crisp learner and therefore it cannot learn models that yield probabilistic inference capabilities. On the other hand, OSL
is based on MLNs and thus inherits their probabilistic properties, but its structure learning component is sub-optimal, i.e., it tends to generate large sets of clauses, many of which have low heuristic value. An in-depth comparison of these systems can be found in(Katzouris et al, 2018). More importantly, both OSL and OLED are supervised learners and in the presence of unlabelled training examples they impose closed-world assumption, that is, they assume everything not known is false, i.e., negative examples. This assumption can seriously compromise the learning task or even worse render it impossible if very little supervision is available, which is a common scenario in real-world applications.
2.2 Harmonic Function Graph-Cut Minimisation
Graph-based semi-supervised learning techniques (Zhu et al, 2009) construct a graph, whose vertices represent the labelled and unlabelled examples in the dataset and the edges reflect the similarity of these examples. Using such a graph, the learning task can be formulated as a graph-cut minimisation problem. The idea is to remove a minimal set of edges, so that the graph is cut into two disjoint sets of vertices; one holding positive examples and one holding negative ones.
Formally, let a training sequence consisting of labelled instances and unlabelled ones . The labelled instances are pairs of a label and a
-dimensional numerical feature vectorof input values, while the unlabelled ones are feature vectors with unknown label. Each of these instances represents either a labelled or an unlabelled vertex of the graph. These vertices are then connected by undirected weighted edges that encode their similarity according to a given distance function. Consequently, the labelled vertices can be used to determine the labels of the unlabelled ones. Once the graph is built, the task reduces into assigning values to the unlabelled vertices. Thus, the goal is to find a function over the vertices, where is a negative label and a positive one, such that for labelled instances, and the cut size is minimised in order for the unlabelled ones to be assigned optimal values.
The minimum graph-cut can be represented as a regularised risk minimisation problem (Blum and Chawla, 2001)
, by using an appropriate loss function, forcing the labelled vertices to retain their values and a regularisation factor controlling the cut size. The cut size is the sum of the weightscorresponding to connected vertices and having different labels, and is computed as follows:
Equation (5) is an appropriate measure of the cut size, since it is affected only by edges for which . Note that if and are not connected, then by definition, while if the edge exists and is not cut, then . Thus, the cut size is well-defined even when summing over all vertex pairs. Assuming that the maximal loss per edge is , the loss for labelled instances should be zero if and otherwise. Thus, the loss function is defined as follows:
Consequently, by combining the loss function, as expressed by eq. (6) and the cut size, as expressed by eq. (5), as a regularisation factor, the minimum graph-cut regularised risk problem is formulated as follows:
Note that eq. (7) is an integer programming problem because is constrained to produce discrete values. Although efficient polynomial–time algorithms exist to solve the minimum graph-cut problem, still the formulation has a particular defect. There could be multiple equally good solutions; a label may be positive in one of the solutions, and negative in the rest. An alternative formulation proposed by (Zhu et al, 2003) for the graph-cut minimisation problem, that overcomes these issues, is based on the harmonic function. The proposed approach is based on harmonic energy minimisation of a Gaussian field and it has been shown to respect the harmonic property, i.e., the value of at each unlabelled vertex is the average of of the neighbouring vertices. In the context of semi-supervised learning, a harmonic function is a function that retains the values of the labelled data and satisfies the weighted average property on the unlabelled data:
The former formula enforces that the labelled vertices retain their values, while the latter averages the labels of all neighbouring vertices of a given vertex, according to the weights of their edges. Therefore, the value assigned to each unlabelled vertex is the weighted average of its neighbours. The harmonic function leads to the same solution of the problem as defined in eq. (7), except that is relaxed to produce real values. The main benefit of the continuous relaxation is that a unique optimal closed–form solution exists for that can be computed using matrix techniques. The drawback of the relaxation is that the solution is a real value in and does not directly correspond to a label. This issue can be addressed by thresholding at zero (harmonic threshold) to produce discrete labels.
2.3 Distance over Herbrand Interpretations
Distance functions constitute essential components of graph-based methods to semi-supervised learning and control the quality of the solution. In the case of numerical data, the Euclidean distance, the Gaussian kernel or Radial Basis Functions are common choices, as are matching distances for categorical data. However, in the presence of relational data there is a need for structure-based distances.
A technique proposed by Nienhuys-Cheng (1997) derives a distance for tree structure formalisms and thus provides a generic and natural approach for syntactic comparison of ground logical atoms. The distance function is defined on a set of expressions (namely ground atoms and ground terms), motivated by the structure and complexity of the expression, as well as the symbols used therein. Let be the set of all expressions in a first-order language and the set of real numbers. The distance over expressions , bounded by , is defined as follows:
The first formula states that the distance of an expression to itself is zero. The second one states that if predicates and are not identical, either in terms of symbol or arity, then their distance is one because they refer to different concepts. We assume that the negation of a predicate has always distance from , and thus, it can be seen as a special case of the second formula, where . In case and are identical, then their distance is computed recursively by the distance of the terms therein. The distance is also used by Nienhuys-Cheng (1997) over subsets of , i.e., sets of ground atoms, by means of the Hausdorff metric (Hausdorff, 1962). Informally, the Hausdorff metric is the greatest distance you can be forced to travel from a given point in one of two sets to the closest point in the other set.
A drawback of the Hausdorff metric (Raedt, 2008; Ramon and Bruynooghe, 1998) is that it does not capture much information about the two sets as it is completely determined by the distance of the most distant elements of the sets to the nearest neighbour in the other set. Thus, it may not be representative of the dissimilarity of the two sets. Formally, given the sets and , their Hausdorff distance is computed as follows:
The overall distance for these sets would be represented by one of the pairwise distances, namely the maximum distance among the minimum ones. Moreover, this type of approach allows one element in one set to match with multiple elements in the other set, which is undesirable because some elements may have no match and thus may be ignored in the resulting distance value. As stated by Raedt (2008), these limitations motivate the introduction of a different notion of matching between two sets, which associate one element in a set to at most one other element. To that end, we employ the Kuhn-Munkres algorithm (Hungarian method) (Kuhn, 1955), which computes the optimal one-to-one assignment given some cost function, in our case the structural distance expressed by eq. (9). The goal is to find the assignment of ground atoms among the sets that minimises the total cost, i.e., the total structural distance.
3 Semi-Supervised Online Structure Learning
Our goal is to effectively apply online structure learning in the presence of incomplete supervision. To do so, we take advantage of the structural dependencies underlying a logic-based representation and exploit regularities in the relational data, in order to correlate given labelled instances to unlabelled ones and reason about the actual truth values of the latter. Structure learning methods attempt to discover multi-relational dependencies in the input data, by combining appropriate evidence predicates, that possibly explain the given supervision, that is, the labelled ground query atoms of interest. The underlying assumption is that sets of ground evidence atoms that explain particular labelled query atoms are also contiguous to sets of ground evidence atoms that relate to unlabelled instances. One promising approach to model such similarities for partially supervised data is to use graph-based techniques. Graph-based methods attempt to formulate the task of semi-supervised learning as a graph-cut optimisation problem (see Section 2.2) and then find the optimal assignment of values for the unlabelled instances given a similarity measure.
Figure 1 presents the components and procedure of our proposed graph-based approach, using, for illustration purposes, the activity recognition domain as formalised in the Event Calculus. In order to address the online processing requirement, we assume that the training sequence arrives in micro-batches. At each step of the online procedure, a training example (micro-batch) arrives containing a sequence of ground evidence atoms, e.g. two persons walking individually, their distance being less than pixel positions and having the same orientation. Each micro-batch may be fully labelled, partially labelled, or contain no labels at all. Labelling is given in terms of the Event Calculus query atoms. Unlabelled query atoms are prefixed by ‘?’. For instance, in micro-batch there is no labelling for time-point , while time-point expresses a positive label for the activity. Micro-batch is passed onto the data partitioning component that groups the training sequence into examples. Each unique labelled example present in the micro-batch is stored in a cache, in order to be reused in subsequent micro-batches that may have missing labels.
Labelled and unlabelled examples are converted into graph vertices, linked by edges that represent their structural similarity. The resulting graph is then used to label all unlabelled ground query atoms. Given the fully labelled training sequence, an online structure learning step refines or enhances the current hypothesis — and the whole procedure is repeated for the next training micro-batch . For the online structure learning component we may use OSL or OLED (see Section 2.1).
Henceforth, we refer to our proposed approach as SPLICE (semi-supervised online structure learning). The components of our method are detailed in the following subsections. To aid the presentation, we use examples from human activity recognition.
3.1 Data Partitioning
In a typical semi-supervised learning setting, the training sequence consists of both labelled instances and unlabelled ones where each label corresponds to a -dimensional feature vector of input values. Given a logic-based representation of instances, our approach begins by partitioning the given input data (micro-batch ) into sets of ground evidence atoms, each one connected to a supervision (query) ground atom. The resulting sets are treated as training examples. Let be the set of all true evidence ground atoms and the set of all ground query atoms of interest in micro-batch . Each example should contain exactly one ground query atom and a proper subset of evidence atoms corresponding to . Given the sets and , we construct an example for each ground query atom in , regardless whether it is labelled or not. To do so, we partition the evidence atoms in into non-disjoint subsets, by grouping them over the constants they share directly to the ground query atom of each example. A constant is shared if and only if it appears in both atoms and its position on both atoms has the same type. Note that the position of a constant in some evidence atom may differ from that in . We refrained from including longer range dependencies, such as considering evidence atoms that can be reached through several shared constants, to favour run-time performance.
Figure 2 illustrates the presented procedure. As usual, express query atoms, while all other predicates express evidence atoms. Unlabelled query atoms are denoted by the prefix ‘?’. Data partitioning takes into account only true evidence atoms and concerns only a specific query predicate. Note that each resulting example has a set of evidence atoms that comprise only constants relevant to the query atom. For instance, the ground evidence atom appearing only in the top example, shares constants with query atoms of other examples too, but constant is only relevant to the top example. Constant does not appear in any query atom and thus can be ignored. Similarly, ground evidence atoms having constants that appear in many query atoms will appear in all corresponding examples. This is an expected and desirable behaviour, because such predicates indeed capture knowledge that may be important to many query atoms. For instance, consider a ground predicate stating that is a person. If such a predicate was included in the evidence of Figure 2, it would appear in every example. Moreover, during data partitioning, SPLICE can ignore specific predicates according to a set of given mode declarations (Muggleton, 1995), using the recall number. If the recall number is zero the predicate is ignored.
We henceforth refer to examples as vertices, since each example is represented by a vertex in the graph which is subsequently used by the graph-cut minimisation process. Algorithm 1 presents the pseudo-code for partitioning input data into examples representing the graph vertices. The algorithm requires as an input a training micro-batch and a set of mode declarations, and produces a set of vertices. At line the micro-batch is partitioned into a set of ground query atoms and a set of ground evidence atoms . Then at line the algorithm iterates over all ground query atoms and for each one it finds all true ground evidence atoms sharing constants of the same type. The set includes a constant of an evidence atom if and only if the position of in has type , and is present in the query atom . Then, is added to the vertex of if all constants of appear in , and their positions on both and have the same type. Function , appearing in line , gives the type of the position of constant in atom , while gives all the types of . Finally, for each pair of ground query atom and its corresponding set of relevant ground evidence atoms, the algorithm creates a vertex and appends it to the vertex set.
3.2 Graph Construction
Once the example vertices have been constructed, supervision completion assigns truth values to the unlabelled vertices, by exploiting information present in the labelled ones. A weighted edge between a particular pair of vertices represents the structural similarity of the underlying ground evidence atom sets in the two vertices. Note that the number of vertices is equal to the number of ground query atoms in , that is . Let be the edge weight, i.e., the structural similarity of and . If is large enough, then the truth values of the ground query atoms are expected to be identical. Therefore, the similarity measure essentially controls the quality of the supervision completion solution.
Our approach regarding the computation of the evidence atom similarities is based on a measure of structural dissimilarity over a set of first-order expressions . The distance does not make any syntactical assumptions about the expressions, such as function-free predicates, and thus is applicable to any domain of interest. As described in Section 2.3, we define a measure over sets of ground atoms using the Kuhn-Munkres algorithm, which provides an optimal one-to-one mapping given a cost matrix. In our case the cost matrix essentially holds the distances between each pair of ground atoms, computed by eq. (9), present in the sets being compared. In particular, for each pair of vertices our approach begins by computing the distance between each pair of expressions resulting in a matrix that represents the costs of the assignment problem:
This matrix is square , assuming that the sets and are of equal size. In the general case, of a matrix, where ,
is padded using zero values to complete the smaller dimension in order to be made square. Intuitively, the zero values in the smaller set capture the notion of unmatched atoms. The matrixcan then be used as the input cost matrix for the Kuhn-Munkres algorithm, in order to find the optimal mapping of evidence atoms.
The optimal mapping is denoted here by the function and is the one that minimises the total cost, i.e., the sum of the distances of the mappings, normalised by the dimension of the matrix:
The unmatched evidence atoms constitute an important component of the total cost, due to the term , which penalises every unmatched ground atom by the greatest possible distance, that is . Thus, can be seen as a regularisation term. The need to penalise unmatched atoms stem from the fact that they may represent important features that discriminate a positive from a negative example. The normalised total cost is translated into a similarity and assigned as the weight of the edge connecting the vertices . The measure denoted by the function is symmetric and is used to calculate the similarity of all distinct vertex pairs. The process generates a symmetrical adjacency matrix comprising the weights of all graph edges. Hence, matrix is computed using eq. (10) through function . To avoid self-loops, i.e., edges that connect a vertex to itself, we set the diagonal of the matrix to zero:
In order to turn the similarity matrix into a graph, we use a connection heuristic, which introduces edges only between vertices that are very similar, i.e., they have a high weight. In the simplest case, we connect the vertices if , given some threshold value (NN). Another alternative is to use nearest neighbour (NN) to choose the edges that will be kept. According to this approach, for each vertex we identify the closest (most similar) vertices. Note that if is among ’s nearest neighbours, the reverse is not necessarily true. In order to avoid tie-breaking, we modified NN to select the top distinct weights in a vertex neighbourhood, and then connect all neighbours having such a weight.
3.3 Supervision Completion
Given the weight matrix , we apply one of the two connection heuristics mentioned above to obtain a sparse matrix , having zeros for unconnected vertices and a similarity value for the rest. Matrix is used to perform graph-cut minimisation to assign truth values to the unlabelled ground query atoms.
Let be the number of labelled and unlabelled vertices. The closed-form solution of the optimisation problem for the harmonic function (see Section 2.2) in matrix notation is as follows. Let be the weighted degree of vertex , i.e., the sum of the edge weights connected to . Let be a diagonal matrix, containing on the diagonal, computed over the matrix . Then the unnormalised graph Laplacian matrix is defined as follows:
In this case, the Laplacian matrix essentially encodes the extent to which the harmonic function (see eq. (8)) differs at a vertex from the values of nearby vertices. Assuming that vertices are ordered so that the labelled ones are listed first, the Laplacian matrix can be partitioned into four sub-matrices as follows:
The partitioning is useful in order to visualise the parts of . Sub-matrices and comprise, respectively, the harmonic function differences between labelled vertices, labelled to unlabelled, unlabelled to labelled and unlabelled to unlabelled. Note that and are symmetric.
Let be the vector of values of all vertices and the partitioning of into hold the values of the labelled and unlabelled vertices respectively. Then by solving the constrained optimisation problem, expressed in eq. (7), using the Lagrange multipliers and matrix algebra, one can formulate the harmonic solution as follows:
Since and are symmetric, any of the two can be used to solve the optimisation defined but eq. (11). However, if we use instead of , then its transpose should be used in order for the matrix dimensions to agree during the multiplications. Equation (11) requires the computation of the inverse of matrix that may be singular, due to many zero values (sparsity). In order to avoid this situation, we replace zeros by a very small number. A different solution would be to use the pseudo-inverse, but the computation proved significantly slower in our datasets, without significant differences in accuracy. Since the optimal solution is required to comprise the labels assigned to unlabelled vertices in , the resulting solution is thresholded at zero to produce binary labels111 We also experimented with adaptive threshold approaches designed to handle the possible class imbalance by exploiting the class prior probabilities. We tried a threshold based on log-odds and an approach proposed by
We also experimented with adaptive threshold approaches designed to handle the possible class imbalance by exploiting the class prior probabilities. We tried a threshold based on log-odds and an approach proposed byZhu et al (2003), called class mass normalisation. Both of them yielded much worse results than the harmonic threshold in our experiments..
Algorithm 2 presents the pseudo-code for constructing the graph and performing supervision completion. The algorithm requires as input a connection heuristic, a structural similarity and a set of vertices, and produces as output a set of labels for the unlabelled vertices. First, we compute the similarity between all pairs of vertices (see lines –). Then we apply the connection heuristic to the matrix holding the similarity values, compute the Laplacian matrix and solve the optimisation problem (see lines –). Finally, for the resulting vector holding the values of the unlabelled vertices, we perform thresholding on each value yielding binary labels (see lines -). Since unlabelled examples are typically much more than the labelled ones (in a micro-batch), the inversion of the Laplacian matrix, yielding time , is the main overhead of the algorithm, where denotes the number of unlabelled ground query atoms in a micro-batch222The complexity analysis of all steps of SPLICE may be found at:
3.4 Label Caching and Filtering
In order to handle real-world applications where labelled examples are infrequent, our method — SPLICE — uses a caching mechanism, storing previously seen labelled examples for future usage. At each step of the online supervision completion procedure, SPLICE stores all unique labelled examples that are not present in the cache and then uses the cached examples to complete the missing labels. For each labelled vertex it creates a clause, using the label as the head, the true evidence atoms as the body, and replacing all constants with variables according to a set of given mode declarations (Muggleton, 1995). For instance, the second vertex of Figure 2 can be converted to the following clause:
For each such clause, SPLICE checks the cache for stored vertices that represent identical clauses and selects only the unique ones. The unique cached vertices are then used as labelled examples in the graph construction process of supervision completion in the current and subsequent micro-batches.
In any (online) learning task, noise, such as contradicting examples, is a potential risk that may compromise the accuracy of the learning procedure. In order to make SPLICE tolerant to noise, we use the Hoeffding bound (Hoeffding, 1963)
, a probabilistic estimator of the error of a model (true expected error), given its empirical error (observed error on a training subset)(Dhurandhar and Dobra, 2012)
. Given a random variablewith a value range in and an observed mean of its values after independent observations, the Hoeffding bound states that with probability the true mean of the variable lies in an interval , where . In other words, the true average can be approximated by the observed one with probability given an error margin .
In order to remove noisy examples, we detect contradictions in the cached labelled vertices, using the subset of training data that has been observed so far in the online process. To do so, we use an idea proposed by Domingos and Hulten (2000). Let be the clause of a cached vertex and the number of times the clause has appeared in the data so far. Recall that the clause of a cached vertex is lifted, i.e. all constants are replaced by variables. Thus lifted clauses may appear many times in the data. Similarly, let be the opposite clause of , that is, a clause having exactly the same body but a negated head, and its counts. For instance the opposite clause of (12) is:
The goal is to eventually select only one of the two contradicting clauses. We define a function with range in that represents the probability of clause to appear in the data instead of its opposite clause . Then according to the Hoeffding bound, for the true mean of the probability difference it holds that , with probability . Hence, if , we accept the hypothesis that is indeed the best clause with probability and thus is kept at this point. Similarly, is the best one if . Therefore, in order to select between contradicting labelled examples, it suffices to accumulate observations until their probability difference exceeds . Until that point both example vertices are used in the optimisation.
Although we use the Hoeffding inequality to make the best filtering decision for contradicting examples, given the data that we have seen so far, the examples are not independent as the Hoeffding bound requires. Consequently, we allow this filtering decision to change in the future, given the new examples that stream-in, by keeping frequency counts of the lifted examples333We assume that the examples stem from a stationary stochastic process and thus the difference between contradicting example frequencies eventually converges when a sufficient amount of observations is accumulated.. This is not the case in other applications (Domingos and Hulten, 2000; Abdulsalam et al, 2011) in which the decision is permanent.
Algorithm 3 presents the pseudo-code for cache update and filtering. The algorithm requires as input the labelled vertices of the current micro-batch and the cached vertices along with their counts, and produces as output the set of filtered labelled vertices and the updated cache. If the clause view of a vertex exists in the cache then the counter of that vertex is incremented, otherwise the vertex is appended in the cache and its counter is set to (see lines –). For each vertex in the cache we produce its clause and check if the cache contains a vertex representing the opposite clause. In case the opposite clause exists, the Hoeffding bound is calculated in order to check if one of them can be filtered out (see lines –). In the case that many labelled examples have been accumulated in the cache, update and filtering can have an impact on performance, yielding a total time of , where is the number of micro-batches seen so far, and is the number of unlabelled query atoms in a micro-batch. Algorithm 4 presents the complete SPLICE procedure.
4 Empirical Evaluation
We evaluate SPLICE on the task of composite event recognition (CER), using OSL and OLED as the underlying structure learners (see Section 2.1). We use the publicly available benchmark video surveillance dataset of the CAVIAR project444http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1, as well as a real maritime surveillance dataset provided by the French Naval Academy Research Institute (NARI), in the context of the datAcron project555http://datacron-project.eu/.
4.1 Experimental Setup
The video surveillance dataset comprises surveillance videos, where each video frame is annotated by human experts on two levels. The first level contains SDEs that concern activities of individual persons, such as when a person is walking or staying inactive. Additionally, the coordinates of tracked persons are also used to express qualitative spatial relations, e.g. two persons being relatively close to each other. The second level contains CEs, describing the activities between multiple persons and/or objects, i.e., people meeting and moving together, leaving an object and fighting. Similar to earlier work (Skarlatidis et al, 2015; Katzouris et al, 2016), we focus on the and CEs, and from the videos, we extract sequences that contain annotation for these CEs. The rest of the sequences in the dataset are ignored, as they do not contain positive examples of these two target CEs. Out of the sequences, are annotated with both and activities, are annotated only with and only with . The total length of the extracted sequences is video frames. Each frame is annotated with the (non-)occurrence of a CE and is considered an example instance. The whole dataset contains a total of SDEs and annotated CE instances. There are example instances in which occurs and in which occurs. Consequently, for both CEs the number of negative examples is significantly larger than the number of positive ones.
The maritime dataset (GiB) consists of position signals from vessels sailing in the Atlantic Ocean, around Brest, France. The SDEs express compressed trajectories in the form of ‘critical points’, such as communication gap (a vessel stops transmitting position signals), vessel speed change, and turn. It has been shown that compressing vessel trajectories in this way allows for accurate trajectory reconstruction, while at the same time improving stream reasoning times significantly (Patroumpas et al, 2017). The dataset contains a total of SDEs. We focus on the CE, where two vessels are moving slowly in the open sea and are close to each other. Since the dataset is unlabelled, we produced synthetic annotation by performing CER using the RTEC engine (Artikis et al, 2015) and a hand-crafted definition of . This way, occurs at out of the time-points of the dataset.
Throughout the experimental analysis, the accuracy results for both supervision completion and structure learning were obtained using the -score. All reported statistics are micro-averaged over the instances of CEs. For the CAVIAR dataset, the reported statistics for structure learning were collected using -fold cross validation over the video sequences, while complete videos were left out for testing. In the maritime dataset, the statistics were collected using -fold cross validation over one month of data, while pairs of vessels were left out for testing. The experiments were performed in a computer with an Intel i7 email@example.comGHz CPU ( cores, threads) and GiB of RAM. SPLICE and OSL are included in LoMRF666https://github.com/anskarl/LoMRF, an open-source framework for MLNs, and OLED is available as an open-source ILP solution777https://github.com/nkatzz/OLED. All presented experiments are reproducible888Instructions for reproducing all presented experiments can be found in:
4.2 Hyperparameter Selection
We ran supervision completion on the CAVIAR dataset, for five values of and , controlling the NN and NN connection heuristics (see Section 3.2), in order to select the best configuration. Each micro-batch retained a percentage of the given labels, selected uniformly. We used and supervision levels for the micro-batches, retaining the corresponding proportion of the labels. We repeated the uniform selection times, leading to datasets per supervision level, in order to obtain a good estimate of the performance of the method.
Figure 3 presents the results for all distinct values of and as the supervision level increases per micro-batch. The -score is measured on the same test set for all supervision levels, namely the that remains unlabelled in the setting. The results indicate that is the best choice for both and , achieving the highest accuracy on all supervision levels, including the low ones.
However, in a typical semi-supervised learning task, the assumption that every micro-batch contains some labels is too optimistic. A more realistic scenario is that a number of batches are completely labelled and the rest are completely unlabelled. We repeated the experiments, selecting uniformly a set of completely labelled batches, and present in Figure 4 the results for different values of and as the supervision level increases.
Note that again is the best choice. As expected the -score is lower in this more realistic setting, particularly for low supervision levels (e.g. ). However, the caching mechanism enables SPLICE to maintain a good accuracy despite the presence of completely unlabelled micro-batches. Another notable difference between Figure 3 and Figure 4
is that the standard error is now larger in most settings. This is because SPLICE is affected by the order in which labels arrive. It is also the reason why the standard error reduces as the supervision level increases. Based on these results, we chosefor the rest of the evaluation.
4.3 Experimental Results
4.3.1 Activity Recognition
First, we tested the performance of SPLICE on the CAVIAR activity recognition dataset for both and
CEs, using the less optimistic scenario that a number of batches are completely labelled and the rest remain completely unlabelled. As in the hyperparameter selection process, the labelled micro-batches were selected using uniform sampling, whilesamples were taken at each supervision level. The results for are presented in Figure 5. The top figures present the -score and runtime for the supervision completion, without structure learning, i.e., how well and how fast the true labels are recovered. The runtime of supervision completion is the total time required for completing all missing labels in each supervision level. To compute the -score, however, only the that remains unlabelled in the supervision level is used. The bottom figures present the -score and runtime (average training time per fold) of structure learning using OSL and OLED, i.e., how well and how fast the patterns for and are learned. The setting in the bottom figures corresponds to full supervision, i.e., no unlabelled instances to be completed by SPLICE. In the bottom figures we also compare the performance of structure learning on the completed datasets against the datasets that contain unlabelled instances (incomplete).
Similar to the results shown in Section 4.2, we observe that supervision completion effectively completes missing labels, even at low supervision levels. Also, the statistical error is reduced as the supervision level increases. The supervision completion runtime reduces as the supervision level increases, due to the smaller number of unlabelled instances that SPLICE needs to process. The results also suggest that SPLICE enhances substantially the accuracy of structure learning. The accuracy of both OSL and OLED without supervision completion is poor, due to the fact that the learners need to assume a label for the unlabelled instances, which is the negative label under the closed-world assumption. OSL achieves somewhat higher accuracy than OLED, especially for little given supervision, due to its ability to better handle noisy data. On the other hand, OLED is much faster than OSL.
Figure 6 presents the results for the CE, which mostly lead to the same conclusions as for the CE, that is, that we can effective complete missing labels and consequently enhance significantly the accuracy of structure learning. One difference in this setting is that OSL achieves higher accuracy than OLED only for low levels of supervision. Based on the results for both CEs, therefore, the version of SPLICE using OSL seems to be preferable for low supervision levels, but OLED has the advantage of being computationally more efficient.
4.3.2 Maritime Monitoring
For the maritime monitoring dataset, we ran SPLICE using OLED because it provides better runtime performance than OSL on larger datasets. Recall that the maritime dataset comprises of SDEs, that is, approximately GiB. Similar to the activity recognition evaluation, we used the less optimistic scenario, that assumes some micro-batches are completely labelled and the remaining ones are completely unlabelled. Due to the higher execution times, we performed experiments using only random splits of the data into labelled and unlabelled batches.
The results for the CE are presented in Figure 7 in a similar form to the previous figures. The first observation is that the -score of supervision completion (top left) is high even for of the given labels. On the other hand, the accuracy does not seem to change as the supervision level increases, i.e., there seems to be a ceiling to the number of labelled instances that can be correctly labelled. That stems from the fact that several positive examples share many common features (SDEs) with many negative examples, and are thus considered very similar. These examples are always misclassified, regardless of the given supervision level. The top-right diagram of Figure 7 shows that the supervision completion runtime increases along the supervision level. This is because the unique labelled examples cached by SPLICE greatly increase as the supervision increases, which was not the case with the activity recognition task. As a result, the quadratic term over cached labelled examples of the label caching component (see Section 3.4) starts downgrading the computational cost.
Figure 7 also compares SPLICE along OLED, with OLED alone i.e., without supervision completion. Unlike the activity recognition experiments, OLED without SPLICE has been instructed to use only the starting points of each interval for structure learning. is very rarely re-initiated (approximately at of the time-points of a interval), and thus focusing on the starting points of the intervals when learning the initiating conditions of this concept can be helpful. Similarly, in the activity recognition experiments, OLED was instructed to use all data points, because and are very frequently re-initiated (at and of the time-points of the intervals of these concepts), and thus it was highly desirable to use all available data for structure learning.
SPLICE with OLED operates as in the activity recognition experiments, that is, SPLICE labels all unlabelled examples, and then OLED uses all examples for structure learning. Instructing OLED to use only the starting points of the intervals does not improve performance in this case, since SPLICE makes some mistakes when labelling these points, compromising the performance of OLED.
The bottom-left diagram of Figure 7 shows that SPLICE enhances considerably the accuracy of OLED in the common case of little supervision (below ). This is a notable result. In the case of
supervision, OLED without SPLICE has a large deviation in performance, indicating the sensitivity of OLED in the presence of unlabelled data. On the other hand, SPLICE-OLED is very robust, as the standard deviation suggests. Provided withor more supervision, OLED without SPLICE can achieve better results. In these supervision levels and in the presence of very few re-initiations, it is better to proceed directly to structure learning, considering all unlabelled examples as negative.
With respect to efficiency, SPLICE-OLED seems to be much slower for lower supervision levels (see the bottom-right diagram of Figure 7). This is expected because SPLICE-OLED uses many more examples than OLED alone in the maritime experiments.
5 Related Work
Structure learning is a task that has received much attention in the literature. The main approaches to this task stem either from probabilistic graphical models (Pietra et al, 1997; Heckerman, 1999; McCallum, 2003), or Inductive Logic Programming (ILP) (Quinlan, 1990; Muggleton, 1995; De Raedt and Dehaspe, 1997; Blockeel and Raedt, 1998). Online versions of structure learning methods have also been proposed, such as e.g., (Huynh and Mooney, 2011; Michelioudakis et al, 2016b; Katzouris et al, 2016), and some of them have been applied to real-world tasks, (Michelioudakis et al, 2016a; Artikis et al, 2017). All the aforementioned approaches, however, assume fully labelled training input in order to achieve generalisation.
On the other hand, existing semi-supervised learning techniques (Zhu et al, 2009) attempt to exploit additional information provided by unlabelled data to guide the learning process, and enhance both performance and accuracy. These algorithms assume that training data are represented as propositional feature vectors. As a result, they cannot be directly applied to logic-based formalisms, that assume a relational data representation. Beyond expressiveness, typical approaches to semi-supervised learning also suffer from computational issues. For instance, self-training techniques (Yarowsky, 1995; Ghahramani and Jordan, 1993; Culp and Michailidis, 2008; Albinati et al, 2015), usually require a significant number of iterations over the training data to converge and thus are not appropriate for online learning. Co-training algorithms (Blum and Mitchell, 1998; Goldman and Zhou, 2000; Chawla and Karakoulas, 2005; Zhou and Li, 2005) on the other hand, require that the training data are separated into distinct views, namely disjoint feature sets that provide complementary, ideally conditionally independent information about each instance, while each view alone is sufficient to accurately predict each class. Such limitations render many of these semi-supervised approaches incapable of handling the complexity of the relational structure learning task and inappropriate for online processing, which assumes a single pass over the training sequence.
Our proposed method is based on graph-based semi-supervised learning (Blum and Chawla, 2001; Zhu et al, 2003; Blum et al, 2004), using a distance function that is suitable for first-order logic. A substantial amount of work exists in the literature on distance-based methods for learning from relational data. These approaches originate from instance-based learning (IBL) (Aha et al, 1991), which assumes that similar instances belong to similar classes (e.g. NN). RIBL (Emde and Wettschereck, 1996) extended IBL to the relational case by using a modified version of a similarity measure for logical atoms proposed by Bisson (1992b), together with a
NN classifier.Bisson (1992a, b) uses a similarity measure, based on the structural comparison of logical atoms, to perform conceptual clustering. Although these distance measures have been used with success in several applications (Bisson, 1992b; Kirsten and Wrobel, 1998, 2000), they are limited to function-free Horn logic operating only over constants. Therefore, they require flattening of representations having non-constant terms, and thus cannot be easily applied to nested representations, such as the Event Calculus. Bohnebeck et al (1998) improved RIBL to allow lists and other terms in the input representation, but their approach is not sensitive to the depth of the structure, i.e., functions.
Closest to SPLICE are techniques proposed for semi-supervised Inductive Logic Programming (ILP) and applied to web page classification. ICT (Iterative Cross-Training) (Soonthornphisaj and Kijsirikul, 2003) is a semi-supervised learning method, based on the idea of co-training. ICT uses a pair of learners, a strong and a weak one, to iteratively train each other from semi-supervised training data. Each learner receives an amount of labelled and unlabelled data. The strong learner starts the learning process from the labelled data, given some prior knowledge about the domain, and classifies the unlabelled data of the weak learner. The weak learner, which has no domain knowledge, then uses these recently labelled data produced by the strong learner, to learn and classify the unlabelled data of the strong learner. This training process is repeated iteratively. ICT-ILP (Soonthornphisaj and Kijsirikul, 2004)
is an extension of ICT that uses an ILP system as one of the classifiers, that is, the strong learner, and a Naive Bayes classifier for the weak learner. The ILP system makes use of a background knowledge that encodes the prior domain knowledge and induces a set of rules from the labelled data. These rules are used to classify the unlabelled examples of the weak learner.Li and Guo (2011, 2012) proposed a similar approach based on relational tri-training. Three different relational learning systems, namely Aleph (Srinivasan, 2003), kFOIL (Landwehr et al, 2006) and nFOIL(Landwehr et al, 2007), are initialised using the labelled data and background knowledge. Then the three classifiers are refined by iterating over the unlabelled data. At each iteration, each unlabelled example is labelled by the three classifiers. In case two of them agree on the labelling of the example, then this example is labelled accordingly. The final classification hypothesis is produced via majority voting of the three base classifiers.
The aforementioned approaches to semi-supervised structure learning iterate multiple times over the training data in order to generalise. Therefore, they are not suitable for online structure learning. Consequently, the method presented in this paper is the first to tackle the problem of online semi-supervised structure learning.
6 Conclusions and Future Work
We presented SPLICE, a novel approach to online structure learning that operates on partially supervised training sequences. SPLICE completes the missing supervision continuously as the data arrive in micro-batches, and can be combined with any online supervised structure learning system. As it processes the input stream, SPLICE can cache previously seen labelled examples for future usage and filter noisy, contradicting labelled examples that may compromise the overall accuracy of the structure learning task. Experimental results in the domain of composite event recognition, using a benchmark dataset for activity recognition and a real dataset for maritime monitoring, showed that SPLICE can enable the underlying structure learner to learn good models even in the presence of little given annotation.
We are currently investigating various extensions to SPLICE, including the improvement of the distance function, especially in the case of many unrelated features. Moreover, we are examining the possibility of extending SPLICE with abductive inference, in order to perform structure learning on hidden concepts with partially supervised data. This last extension is desirable for learning CE definitions, because, usually, the provided labels are different from the target concept.
Acknowledgements.The work has been funded by the EU H2020 project datAcron (687591). We would also like to thank Nikos Katzouris for providing assistance on the distance functions for first-order logic and helping us running OLED.
Abdulsalam et al (2011)
Abdulsalam H, Skillicorn DB, Martin P (2011) Classification using streaming random forests. IEEE Trans Knowl Data Eng 23(1):22–36
- Aha et al (1991) Aha DW, Kibler DF, Albert MK (1991) Instance-based learning algorithms. Machine Learning 6:37–66, DOI 10.1023/A:1022689900470
- Albinati et al (2015) Albinati J, Oliveira SEL, Otero FEB, Pappa GL (2015) An ant colony-based semi-supervised approach for learning classification rules. Swarm Intelligence 9(4):315–341
- Alevizos et al (2017) Alevizos E, Skarlatidis A, Artikis A, Paliouras G (2017) Probabilistic complex event recognition: A survey. ACM Comput Surv 50(5):71:1–71:31
Artikis et al (2012)
Artikis A, Skarlatidis A, Portet F, Paliouras G (2012) Logic-based event recognition. Knowledge Engineering Review 27(4):469–506
- Artikis et al (2015) Artikis A, Sergot MJ, Paliouras G (2015) An event calculus for event recognition. IEEE Transactions on Knowledge and Data Engineering 27(4):895–908
- Artikis et al (2017) Artikis A, Katzouris N, Correia I, Baber C, Morar N, Skarbovsky I, Fournier F, Paliouras G (2017) A prototype for credit card fraud management: Industry paper. In: Proceedings of the 11th ACM International Conference on Distributed and Event-based Systems, ACM, pp 249–260
Bisson G (1992a) Conceptual clustering in a first order logic representation. In: Proceedinds of the 10th European Conference on Artificial Intelligence, Wiley, pp 458–462
- Bisson (1992b) Bisson G (1992b) Learning in FOL with a similarity measure. In: Proceedings of the 10th National Conference on Artificial Intelligence, AAAI Press / The MIT Press, pp 82–87
- Blockeel and Raedt (1998) Blockeel H, Raedt LD (1998) Top-down induction of first-order logical decision trees. Artificial Intelligence 101(1-2):285–297
- Blum and Chawla (2001) Blum A, Chawla S (2001) Learning from labeled and unlabeled data using graph mincuts. In: Proceedings of the Eighteenth International Conference on Machine Learning, Morgan Kaufmann, pp 19–26
Blum and Mitchell (1998)
Blum A, Mitchell TM (1998) Combining labeled and unlabeled data with co-training. In: Proceedings of the 11th Annual Conference on Computational Learning Theory, ACM, pp 92–100
- Blum et al (2004) Blum A, Lafferty JD, Rwebangira MR, Reddy R (2004) Semi-supervised learning using randomized mincuts. In: Proceedings of the 21st International Conference on Machine Learning, ACM
- Bohnebeck et al (1998) Bohnebeck U, Horváth T, Wrobel S (1998) Term comparisons in first-order similarity measures. In: Proceedings of the 8th International Workshop on Inductive Logic Programming, Springer, pp 65–79
- Chawla and Karakoulas (2005) Chawla NV, Karakoulas G (2005) Learning from labeled and unlabeled data: An empirical study across techniques and domains. Journal of Artificial Intelligence Research 23(1):331–366
- Cugola and Margara (2012) Cugola G, Margara A (2012) Processing flows of information: From data stream to complex event processing. ACM Computing Survey 44(3):15:1–15:62
- Culp and Michailidis (2008) Culp M, Michailidis G (2008) An iterative algorithm for extending learners to a semi-supervised setting. Journal of Computational and Graphical Statistics 17(3):545–571
- De Raedt and Dehaspe (1997) De Raedt L, Dehaspe L (1997) Clausal discovery. Machine Learning 26(2-3):99–146
- Dhurandhar and Dobra (2012) Dhurandhar A, Dobra A (2012) Distribution-free bounds for relational classification. Knowledge and Information Systems 31(1):55–78
- Domingos and Hulten (2000) Domingos PM, Hulten G (2000) Mining high-speed data streams. In: Proceedings of the 6th International Conference on Knowledge Discovery and Data Mining, pp 71–80
Duchi et al (2011)
Duchi J, Hazan E, Singer Y (2011) Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research 12:2121–2159
- Emde and Wettschereck (1996) Emde W, Wettschereck D (1996) Relational instance-based learning. In: Proceedings of the 13th International Conference on Machine Learning, Morgan Kaufmann, pp 122–130
- Ghahramani and Jordan (1993) Ghahramani Z, Jordan MI (1993) Supervised learning from incomplete data via an EM approach. In: Proceedings of the 7th Conference on Advances in Neural Information Processing Systems 6, Morgan Kaufmann, pp 120–127
- Goldman and Zhou (2000) Goldman SA, Zhou Y (2000) Enhancing supervised learning with unlabeled data. In: Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 - July 2, 2000, Morgan Kaufmann, pp 327–334
- Hausdorff (1962) Hausdorff F (1962) Set Theory. AMS Chelsea Publishing Series, Chelsea Publishing Company
Heckerman D (1999) Learning in Graphical Models. MIT Press, chap A Tutorial on Learning with Bayesian Networks, pp 301–354
- Hoeffding (1963) Hoeffding W (1963) Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58(301):13–30
- Huynh and Mooney (2011) Huynh TN, Mooney RJ (2011) Online Structure Learning for Markov Logic Networks. In: Proceedings of ECML PKDD, vol 2, pp 81–96
- Katzouris et al (2016) Katzouris N, Artikis A, Paliouras G (2016) Online learning of event definitions. Theory and Practice of Logic Programming 16(5-6):817–833
- Katzouris et al (2018) Katzouris N, Michelioudakis E, Artikis A, Paliouras G (2018) Online learning of weighted relational rules for complex event recognition. In: Proceedings of ECML-PKDD
- Kirsten and Wrobel (1998) Kirsten M, Wrobel S (1998) Relational distance-based clustering. In: Proceedings of the 8th International Workshop on Inductive Logic Programming, Springer, pp 261–270
Kirsten and Wrobel (2000)
Kirsten M, Wrobel S (2000) Extending k-means clustering to first-order representations. In: Proceedings of the 10th International Conference on Inductive Logic Programming, Springer, pp 112–129
- Kowalski and Sergot (1986) Kowalski RA, Sergot MJ (1986) A logic-based calculus of events. New Generation Computing 4(1):67–95
- Kuhn (1955) Kuhn HW (1955) The hungarian method for the assignment problem. Naval Research Logistics Quarterly 2:83–97
- Landwehr et al (2006) Landwehr N, Passerini A, Raedt LD, Frasconi P (2006) kfoil: Learning simple relational kernels. In: Proceedings of the 21st National Conference on Artificial Intelligence, AAAI Press, pp 389–394
- Landwehr et al (2007) Landwehr N, Kersting K, Raedt LD (2007) Integrating naïve bayes and FOIL. Journal of Machine Learning Research 8:481–507
- Li and Guo (2011) Li Y, Guo M (2011) Web page classification using relational learning algorithm and unlabeled data. Journal of Computers 6(3):474–479
- Li and Guo (2012) Li Y, Guo M (2012) A new relational tri-training system with adaptive data editing for inductive logic programming. Knowl-Based Syst 35:173–185
- McCallum (2003) McCallum A (2003) Efficiently inducing features of conditional random fields. In: Proceedings of the 19th conference on Uncertainty in Artificial Intelligence, pp 403–410
- Michelioudakis et al (2016a) Michelioudakis E, Artikis A, Paliouras G (2016a) Online structure learning for traffic management. In: Proceedings of the 26th International Conference on Inductive Logic Programming, pp 27–39
- Michelioudakis et al (2016b) Michelioudakis E, Skarlatidis A, Paliouras G, Artikis A (2016b) Online structure learning using background knowledge axiomatization. In: Proceedings of ECML-PKDD, vol 1, pp 242–237
- Mueller (2008) Mueller ET (2008) Event Calculus. In: Handbook of Knowledge Representation, Foundations of Artificial Intelligence, vol 3, Elsevier, pp 671–708
- Muggleton (1995) Muggleton S (1995) Inverse Entailment and Progol. New Generation Computing 13:245–286
- Nienhuys-Cheng (1997) Nienhuys-Cheng SH (1997) Distance Between Herbrand Interpretations: A Measure for Approximations to a Target Concept. In: Proceedings of the 7th International Workshop on Inductive Logic Programming, Springer-Verlag, pp 213–226
- Patroumpas et al (2017) Patroumpas K, Alevizos E, Artikis A, Vodas M, Pelekis N, Theodoridis Y (2017) Online event recognition from moving vessel trajectories. GeoInformatica 21(2):389–427
- Pietra et al (1997) Pietra SD, Pietra VD, Lafferty J (1997) Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(4):380–393
- Quinlan (1990) Quinlan JR (1990) Learning logical definitions from relations. Machine Learning 5:239–266
- Raedt (2008) Raedt LD (2008) Logical and Relational Learning: From ILP to MRDM (Cognitive Technologies). Springer-Verlag New York, Inc., Secaucus, NJ, USA
- Ramon and Bruynooghe (1998) Ramon J, Bruynooghe M (1998) A framework for defining distances between first-order logic objects. In: Proceedings of the 8th International Workshop on Inductive Logic Programming, Springer, pp 271–280
- Richards and Mooney (1992) Richards BL, Mooney RJ (1992) Learning relations by pathfinding. In: Proceedings of AAAI, AAAI Press, pp 50–55
- Richardson and Domingos (2006) Richardson M, Domingos PM (2006) Markov logic networks. Machine Learning 62(1-2):107–136
- Skarlatidis et al (2015) Skarlatidis A, Paliouras G, Artikis A, Vouros GA (2015) Probabilistic Event Calculus for Event Recognition. ACM Transactions on Computational Logic 16(2):11:1–11:37
- Soonthornphisaj and Kijsirikul (2003) Soonthornphisaj N, Kijsirikul B (2003) Iterative cross-training: An algorithm for web page categorization. Intelligent Data Analysis 7(3):233–253
- Soonthornphisaj and Kijsirikul (2004) Soonthornphisaj N, Kijsirikul B (2004) Combining ILP with semi-supervised learning for web page categorization. In: Proceedings of the International Conference on Computational Intelligence, pp 322–325
- Srinivasan (2003) Srinivasan A (2003) The aleph manual. Tech. Rep. 4, Computing Laboratory, Oxford University, URL http://web.comlab.ox.ac.uk/oucl/research/areas/machlearn/Aleph/aleph
- Yarowsky (1995) Yarowsky D (1995) Unsupervised word sense disambiguation rivaling supervised methods. In: Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pp 189–196
- Zhou and Li (2005) Zhou Z, Li M (2005) Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering 17(11):1529–1541
- Zhu et al (2003) Zhu X, Ghahramani Z, Lafferty JD (2003) Semi-supervised learning using gaussian fields and harmonic functions. In: Proceedings of the 20th International Conference on Machine Learning, AAAI Press, pp 912–919
- Zhu et al (2009) Zhu X, Goldberg AB, Brachman R, Dietterich T (2009) Introduction to Semi-Supervised Learning. Morgan and Claypool Publishers