Sequence Covering Similarity for Symbolic Sequence Comparison

01/22/2018 ∙ by Pierre-François Marteau, et al. ∙ 0

This paper introduces the sequence covering similarity, that we have formally define for evaluating the similarity between a symbolic sequence and a set of symbolic sequences. From this covering similarity we derive a pair-wise distance to compare two symbolic sequences. We show that this covering distance is a metric.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Estimating efficiently the similarity between symbolic sequences is a recurrent task in various application domains, in particular in bio-informatics, text processing or computer or network security. Numerous similarity measures have been defined to cope with symbolic sequences such the edit distance and its implementation proposed by Wagner and Fisher [Wagner1974], BLAST [Korf2003], the Smith and Waterman or Levenshtein [SMITH1981, Levenshtein66] and the Needleman Wunch [NEEDLEMAN1970] distances or the local sequence kernels [Vert2004].

We present in this paper a new approach to characterize similarity between sequences by introducing the notion of sequence covering. Basically, this similarity is based on a set of reference sequences which defines a dictionary of subsequences that are used to ’optimally’ cover any sequence. Originally this sequence covering principle has been introduced in the context of Host Intrusion Detection [Marteau2017]. We derive hereinafter a pairwise similarity measure and show that this measure is a semimetric on the set of strings. We finally highlights through some examples the utility of this measure.

2 The Sequence Covering Similarity

[scale=.7]principle.pdf

Figure 1: Example of the covering of a sequence () using subsequences of sequences in a set ().

The notion of sequence covering is simple and depicted in Fig. 1. The sequence is covered by subsequences of the sequences that belong to set . On this example, the covering is optimal in the sense that it is composed with a minimal number of subsequences. It is total in the sense that all the elements of are covered.

The sequence covering similarity between and set relates the size (in number of subsequences) of the optimal covering of using sequences of , to the size of (in number of elements) itself, , such that it is maximum equal to one if the covering is of size 1, and minimal equal to if the covering is composed with subsequences of size 1.

We define precisely these notions in the following subsection.

2.1 Definitions and notation

Let be a finite alphabet and let be the set of all sequences (or strings) define over . We note the empty sequence.

Let be any set of sequences, and let be the set of all subsequences that can be extracted from any element of . We denote by the set of all the multisets111A multiset is a collection of elements in which elements are allowed to repeat; it may contain a finite number of indistinguishable copies of a particular element. that we can compose from the elements of .

is called a partial covering of sequence iif

  1. all the subsequences of are also subsequences of ,

  2. indistinguishable copies of a particular element in correspond to distinct occurrences of the same subsequence in .

If entirely covers , meaning that we can find an arrangement of the elements of that covers entirely , then we will call it a full covering for .

Finally, we call a -optimal covering of any full covering of which is composed with a minimal number of subsequences in .

Let be a -optimal covering of .

We define the covering similarity measure between any non empty sequence and any set as

(1)

where is the number of subsequences composing a -optimal covering of , and is the length of sequence .

Note that in general is not unique, but since all such coverings have the same cardinality, , is well defined.

Properties of :

  1. If is a non empty subsequence in , then is maximal.

  2. In the worse case, the -optimal covering of has a cardinality equal to , meaning that it is composed only with subsequences of length . In that case, is minimal.

  3. If is non empty, (notice that if , ).

Furthermore, as is a subsequence of any sequence in , we define, for any set ,

As an example, let us consider the following case:

The -optimal covering of 222([0,0,1,1][0,0,1,1],[0,0,1,1][0,0,1,1]) is a -optimal covering of is of size , hence , and the -optimal covering of 333([0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1]) is a -optimal covering for is of size , leading to .

2.2 Finding a -optimal covering for any tuple

The brute-force approach to find a -optimal covering for a sequence is presented in algorithm 1. It is an incremental algorithm that, first, finds the longest subsequence of that is contained in and that starts at the beginning of . This first subsequence is the first element of the -optimal covering. Then, it searches for the following longest subsequence that is in and that starts at the end of the first element of the covering, adds it to the covering in construction, and iterate until reaching the end of sequence .

input : , a set of sequences
input : , a test sequence
output : , a (-optimal) covering for
1 ;
2 ;
3 ;
4 while continue do
5       ;
6       while  and  do
7             ;
8            
9      ;
10       if  then ;
11       ;
12      
13return ;
Algorithm 1 Find a -optimal covering for
Proposition 2.1.

Algorithm 1 outputs a -optimal covering for sequence .

Proof.

i) First we notice that since all the subsequences of length constructed on are included into , algorithm 1, by construction, necessarily outputs a full covering of (meaning that is entirely covered by the subsequences of the covering provided the algorithm).

ii) Second we notice that, for all and in such that is a subsequence of , and any , .

We finalize the proof by induction on , the cardinality (the size) of the coverings.

The proposition is obviously true for : for all sequence for which a covering of size exists (meaning that is a subsequence of one of the sequences in ), algorithm 1 finds the S-optimal covering that consists of itself.

Then, assuming that the proposition holds for , such that (IH), we consider a sequence that admits a -optimal covering of size .

Let , be the decomposition of according to the full covering provided by algorithm 1, where is the prefix of the covering (first element) and the remaining suffix subsequence (concatenation of the remaining covering elements). is the sequence concatenation operator. Similarly, let , be the decomposition of according to a -optimal covering of . Necessarily, , which is also a prefix of , is a subsequence of (otherwise, since is in , algorithm 1 would have increased the length of at least to the length of ). Hence, is a subsequence of and, according to ii), . This shows that is a sequence that admits a -optimal covering, , of size at most equal to . According to (HI), algorithm 1 returns such an optimal covering for . This shows that the covering that is returned by algorithm 1 for the full sequence , is at most of size , meaning that it is actually a -optimal covering for of size . Hence, by induction, the proposition is true for all , which proves the proposition. ∎

2.2.1 Other property

Proposition 2.2.

By definition of the -optimal covering of a sequence, it is easy to show that

For all , all and all , , leading to .

2.3 Pairwise similarity and pairwise distance for comparing pairs of symbolic sequences (strings)

The covering similarity between a sequence and a set of sequences as defined in Eq. 1 allows for the definition of a covering similarity measure on the sequence set, , itself. For any pair of non empty sequences , we define it as follows

(2)

where is defined in Eq. 1.

Then, we define , and for any non empty , we get that

Finally we define straightforwardly a pairwise distance on as

(3)

Leading to

(4)
Proposition 2.3.

is a semimetric on

Proof.

It is easy to verify that is non negative: for all , and all , . Hence, for all , , , and, according to Eq. and Eq. 3 4, for all , , .

identity of indiscernibles: First, for all , , if , then hence .

Conversely, for all , s.t. ,

  • if , then necessarily , otherwise and

  • If if , then necessarily and, since

    , necessarily , which means that is a subsequence of and conversely, is a subsequence of , showing that .

symmetry: As is symmetric by construction, so is .

3 Algorithmic complexity

A suffix tree implementation of algorithm 1 leads to a time complexity that is upper bounded by , where is the size of a -optimal covering for .

The previous time complexity does not depend on , which means that we can increase the size of without loosing on the processing time. This property is particularly important for applications for which is potentially large such as in plagiarism detection for instance.

For the pairwise distance , the time complexity is where is the size of a -optimal covering for and is the size of a -optimal covering for . In comparison, the Levenshtein’s distance is in .

4 Examples

We give below some examples that present the use of the covering similarity or distance for string matching and processing. A python 3 implementation available at https://github.com/pfmarteau/STree4CS allows to play these examples.

4.1 Pairwise distances on strings

Table 1 presents the covering distance values obtained for some pairs of strings. As a comparative baseline, the Levenshtein’s distance [Levenshtein66] is also given for the same pairs of strings.

! Levenshtein 444Levenshtein, Vladimir I. (February 1966). ”Binary codes capable of correcting deletions, insertions, and reversals”. Soviet Physics Doklady. 10 (8): 707–710 ’amrican’ ’american’ .196 .067 ’european’ ’american’ .75 .375 ’european’ ’indoeuropean’ .167 .25 ’indian’ ’indoeuropean’ .5 .583 ’indian’ ’american’ 0.708 .417 ’narcotics’ ’narcoleptics’ .222 .167 ’little big man’ ’big little man’ .143 .286

Table 1: Covering and Levenshtein’s distances on some pairs of strings. Min and max values for each distance are in bold fonts.

4.2 Detection of plagiarism

We show in this example how the sequence covering similarity is able to detect lifted passage of an original source text spread in a plagiarized text.

This example (Example 2) is borrowed from
https://www.princeton.edu/pr/pub/integrity/pages/plagiarism/

Original source text
”From time to time this submerged or latent theater in Hamlet becomes almost overt. It is close to the surface in Hamlet’s pretense of madness, the “antic disposition” he puts on to protect himself and prevent his antagonists from plucking out the heart of his mystery. It is even closer to the surface when Hamlet enters his mother’s room and holds up, side by side, the pictures of the two kings, Old Hamlet and Claudius, and proceeds to describe for her the true nature of the choice she has made, presenting truth by means of a show. Similarly, when he leaps into the open grave at Ophelia’s funeral, ranting in high heroic terms, he is acting out for Laertes, and perhaps for himself as well, the folly of excessive, melodramatic expressions of grief.”

Plagiarism: Lifting selected passages and phrases without proper acknowledgment (lifted passages are underlined)
”Almost all of Shakespeare’s Hamlet can be understood as a play about acting and the theater. For example, in Act 1, Hamlet adopts a pretense of madness that he uses to protect himself and prevent his antagonists from discovering his mission to revenge his father’s murder. He also presents truth by means of a show when he compares the portraits of Gertrude’s two husbands in order to describe for her the true nature of the choice she has made. And when he leaps in Ophelia’s open grave ranting in high heroic terms, Hamlet is acting out the folly of excessive, melodramatic expressions of grief.

Covering Distance = 0.219
Covering Similarity = 0.801
Covering = [’A’, ’lmost ’, ’al’, ’l’, ’ of ’, ’S’, ’ha’, ’k’, ’es’, ’pe’, ’ar’, ’e’, ’’s ’, ’Hamlet ’, ’c’, ’an’, ’ be’, ’ u’, ’nd’, ’ers’, ’to’, ’od’, ’ as ’, ’a ’, ’pl’, ’a’, ’y ’, ’a’, ’b’, ’out ’, ’acting ’, ’and ’, ’the t’, ’heater’, ’. ’, ’F’, ’or ’, ’ex’, ’am’, ’pl’, ’e, ’, ’in ’, ’A’, ’ct ’, ’1’, ’, ’, ’Hamlet a’, ’d’, ’op’, ’ts ’, ’a ’, ’pretense of madness’, ’ th’, ’at ’, ’he ’, ’us’, ’es ’, ’to protect himself and prevent his antagonists from ’, ’dis’, ’co’, ’ver’, ’ing ’, ’his m’, ’is’, ’sion’, ’ to ’, ’reven’, ’ge’, ’ his ’, ’fa’, ’ther’s ’, ’m’, ’ur’, ’de’, ’r’, ’. ’, ’H’, ’e a’, ’l’, ’s’, ’o pr’, ’esent’, ’s t’, ’ruth by means of a show’, ’ when he ’, ’com’, ’p’, ’ar’, ’es ’, ’the p’, ’or’, ’tr’, ’a’, ’it’, ’s of ’, ’G’, ’ert’, ’ru’, ’de’, ’’s ’, ’two ’, ’h’, ’us’, ’b’, ’and’, ’s in’, ’ or’, ’de’, ’r to ’, ’describe for her the true nature of the choice she has made’, ’. ’, ’A’, ’nd ’, ’when he leaps in’, ’ Ophelia’s ’, ’open grave ’, ’ranting in high heroic terms, ’, ’Hamlet ’, ’is acting out ’, ’the folly of excessive, melodramatic expressions of grief.’]

The small differences between lifted passages that are underlined in the original text and in the optimal covering is due to the non-uniqueness of the optimal covering. A simple post-processing can easily correct these small discrepancies. We notice also that few covering substrings such as ’when he leaps in’ have not been underlined in the original text.

Indeed, if the plagiarized text is re-written with the same text structure but different wording, then the similarity would drop, and the covering won’t be so informative.

5 Conclusion

We have introduced the notion of sequence covering given a set of reference sequences which define a dictionary of subsequences that are used to ’optimally’ cover any sequence. Originally this notion has been introduced in the context of host intrusion detection. From this notion we have defined a pairwise distance measure that can be used to compare two sequences and shown that this measure is a semimetric. As the nature of the sequence covering similarity is somehow complementary to other existing similarity defined for sequential data, one may conjecture it could help by bringing some complementary discriminant information. In particular, as efficient implementations exist using suffix trees or arrays, this similarity could bring some benefits in bioinformatics or in text processing applications.

References