This paper presents Multi-Event-Class Synchronization (MECS), a new algorithm to measure the amount of synchronization between events detected in two or more time series. MECS belongs to a family of Event Synchronization (ES) techniques and it is inspired by the work of R. Q. Quiroga and colleagues (Quiroga et al., 2002). The techniques to perform ES analysis are also known under the name of Measures of spike train synchrony (Kreuz, 2011). ES analysis is performed for measuring the degree of synchronization between events occurring in a set of time series. The term “events” denotes a significant behavior for a system.
With respect to other existing techniques, MECS brings substantial extensions which allow to model a large set of real-life phenomena. First of all, it deals with multiple classes of events. After grouping events into classes, synchronization is computed within a class, i.e., between events belonging to the same class (intra-class synchronization) and between classes, i.e., between events belonging to different classes (inter-class synchronization). Additionally, events can be combined in macro-events on which synchronization is measured. Each macro-event is an aggregation of the events that satisfy some constraints. A relevant example of macro-event is a sequence of events. Events and macro-events can be grouped in macro-classes and synchronization can be computed within and between them.
While many of the existing ES algorithms (e.g., (Quiroga et al., 2002; Kreuz et al., 2009) were developed in the context of brain signal analysis, MECS was created with the purpose of studying multimodal human-human and human-machine interaction. MECS can be applied to a large variety of problems and, in particular, it can be used by human centered systems, multi-modal interfaces for human-machine interaction, or to study multi-modal expressive behaviors of individuals as well as social signals in groups. Indeed, intra-personal synchronization between expressive behaviors of one or more body modalities is an important cue of several emotion displays (Keltner, 1995; Niewiadomski et al., 2011), synchronization between physiological signals and movement kinematics allows one to distinguish between different qualities of human full-body movement (Lussu et al., 2016). At the same time, interpersonal synchronization of expressive behaviors in a group of people is an important cue of group cohesion (Lakens and Stel, 2011; Hung and Gatica-Perez, 2010) and soft-entrainment (Alborno et al., 2016).
This paper is organized as follows: in Section 2 we describe background and present related work. Section 3 describes the MECS algorithm. In Section 4 we discuss macro-events and macro-classes. Section 5 presents a simple implementation of the MECS algorithm with pseudo-code, then Section 6 provides examples of the algorithm execution over a few synthetic sample signals. Finally in Section 7, we discuss a concrete case study in the framework of human behavior understanding: intra-personal synchronization between two sensory modalities for analysis of expressive behavior.
|(Quiroga et al., 2002)||2||1||not handled||not handled|
|(Iqbal and Riek, 2016)||M||N||not handled||not handled|
|(Kreuz et al., 2009)||M||1||not handled||not handled|
2. Related work
Event Synchronization techniques measure synchronization and time-delay patterns between two time series. An example of ES technique was proposed by Quiroga et al. (Quiroga et al., 2002). Given two time series of samples and , , events are detected and the time instants and , at which events occur in time series and respectively, are computed. Synchronization between the detected events is measured as:
where and are, the number of times an event in time series appears within a time interval defined by parameter after an event appears in time series , and vice-versa. is the number of events detected in time series and is the number of events detected in time series . is computed as:
where is defined as follows:
In particular, the second line of Equation 3 is needed, because when two events occur simultaneously they will be counted 2 times by Quiroga’s algorithm, a total contribution of 1 to the computation of . Also, we highlight that the quantity can be greater than 1.
Iqpal & Riek (Iqbal and Riek, 2016) proposed an extension of Quiroga’s algorithm to deal with multiple types of events. In their approach, given two time series and and the event type they first compute the synchronization index for all events of type using the Equations 1 - 3. Next, they compute synchronization of multiple types of events between and as the average of , weighted by the number of events of type . In the last step, they consider different time series and they compute their pairwise and overall synchronization. In particular, the individual synchronization of a given time series is the average of all the pairwise synchronizations between and each time series that are beyond a certain fixed threshold . The overall synchronization is the average of the products of the individual synchronization indexes multiplied by their connectivity values, where the connectivity value is the number of time series pairs having a pairwise synchronization above the threshold , divided by the total number of pairs.
Kreuz et al. (Kreuz et al., 2009) presented a multivariate extension to measure the synchrony from the relative number of spike occurrences.
The Multi-Event Class Synchronization (MECS) algorithm we present in this paper introduces new characteristics that are missing in works byQuiroga et al., Iqpal & Riek, and Kreuz et al., as highlighted in Table 1. The table provides a comparison between these algorithms in terms of: i) the maximum number of input time series; ii) the maximum number of classes; iii) the possibility of handling macro-events, and iv) macro-classes.
The MECS algorithm can compute the synchronization between time series and a single event class (). Differently from Quiroga et al., it provides a normalized output in (see the above comment on the outputs of Quiroga’s algorithm). It can also compute synchronization of multivariate time series, but, unlike Kreuz et al. which considers only single event class (), it manages multiple classes of events () within the time series. Additionally to Iqpal & Riek, it manages the computation of synchronization between events belonging to different classes (i.e., inter-class synchronization).
Finally, differently from all the algorithms mentioned above, MECS introduces the computation of synchronization between event classes over time series, handling macro-events and macro classes.
3. Multi-Event Class Synchronization
Multi-Event Class Synchronization (MECS) computes the amount of synchronization between events occurring in a set of time series. Events may belong to different classes: MECS computes (i) a separate synchronization index for each class (intra-class synchronization), (ii) a synchronization index for a specific aggregation of classes (inter-class synchronization), and a global synchronization index for all classes.
To present MECS, we introduce an example scenario, which is helpful to explain the major features of the algorithm. For example, suppose that we are interested in measuring the level of motor coordination between the members of users’ group performing a motor task (e.g., a fitness exercise). A measure of coordination is obtained by evaluating the amount of synchronization between the movements of the users. Let us consider:
A set of time series:
A set of event classes:
Time series contain information about significant events in the data. In other words, it describes occurrences of certain phenomena. Introducing the events means to fit a continuous information (e.g., a velocity of a joint) to the discrete domain. An event can be, for instance, a local maximum/minimum of the continuous signal (e.g., joints velocity). Such events can be identified automatically (see, e.g., (Alborno et al., 2016; Lussu et al., 2016)) or annotated manually. Events can belong to several classes. Each class characterizes a different type of event, e.g., class might correspond to local maxima of , while to local minima of the same .
Coming back to our example, the time series can be used describe the motor activity of users (i.e., time series models the motor activity of user ) and contain events of any class while each class identifies a specific movement (for example “step performed”, “object grabbed”, “object released”, and so on).
3.1. Intra-class synchronization
MECS relies its computation on the temporal distances between events. As in (Quiroga et al., 2002; Kreuz, 2011), the computation consists of two steps: i) the algorithm first detects events coincidences of two different time series in a specific time interval (coincidence detection) and counts them, ii) then the number of detected coincidences is normalized with respect to the total number of possible coincidences (normalization). We associate to each event of class , occurring in time series , its occurrence time:
where represents the total number of detected events of class occurring in the -th time series . For example, represents the time at which the fourth event belonging to the third class, i.e., , occurred in . In the coincidences detection phase, for each pair of time series , the MECS algorithm computes the amount of coincidence , between an event detected on time series and another event (of the same class of ) detected on time series (with ) by measuring the extent to which they are close in time within a certain interval (coincidence window) that depends on the class of the events. The temporal distance between events and is computed as:
The amount of coincidence between and is defined as follows:
Different than (Quiroga et al., 2002; Kreuz, 2011), where coincidence is only detected, in our algorithm it is also quantified, such that . The dimension of the coincidence window is either empirically chosen for a specific problem or it is automatically calculated, e.g., for each pair of events and as proposed in (Kreuz, 2011), i.e.:
For each class , the overall coincidence of all the events of class in time series with respect to the events of the same class in time series is computed as follows:
First, the average of the of each event in time series with respect to all events in time series is calculated;
Then the sum of the average of all the events in time series is taken.
Equation 7 shows that event in can contribute to the overall coincidence by being coincident (at different extents) with more than one event in . Multiple coincidences are usually unwanted and the computation of the coincidence window, as in Equation 3.1, is often performed with the exact purpose of minimizing the likelihood of counting multiple coincidences in one coincidence window . Since MECS enables to weight coincidences so that a perfect coincidence has a weight of 1.0, and the amount of coincidence decreases along the coincidence window , it supports managing multiple coincidences that may indeed happen in some application contexts. Analogously, the overall coincidence of all the events of class in time series with respect to the events of class in time series is computed by taking the average coincidence of each event in time series with respect to all events in time series and then taking the sum of averages:
Pairwise synchronization of the events of class for the pair of time series is computed as:
Having defined the set of all the 2-combinations of the set (i.e., each element is a distinct pair , with , , ), the overall synchronization for the events of class is finally obtained as:
where the cardinality of set is given by the number of 2-combinations of , that is:
3.2. Global intra-class synchronization
To compute a global synchronization index
for the events of all classes, we define the multi-class synchronization vectoras:
is obtained as a function of , i.e., . A straightforward choice for is the average over the components of . If event classes have e.g., different priorities, a set of weights can be associated to each class of events and a weighted synchronization index is computed as:
3.3. Inter-class synchronization
Given the set of event classes , we may compute inter-class synchronization, i.e., the synchronization between events that belong to the different classes and .
For each couple of time series , , , , the coincidence between an event found in the first time series and another event found in the second time series measured by releasing the constraint that they belong to the same event class , i.e., belongs to class and to class ( and ). The measure of how much events and are close in time is computed within a certain interval that may depend on the considered pair of classes of events. The temporal distance between events and (see Equation 5) is reformulated as:
Accordingly, the relative amount of coincidence (see Equation 6) becomes:
For a pair of classes and , the overall coincidence of all the events of class in time series with respect to the events of class in time series and analogously, the overall coincidence of all the events of class in time series with respect to the events of class in time series are computed by:
Finally, inter-class pairwise synchronization of the events of class and events of class , for the pair of time series is computed as:
and the overall synchronization for the pair of class and (refer to Equation 9):
where is the set of all the 2-combinations of the set (refer to Equation 11), and and both belong to .
4. Macro classes and Macro events
Compared with previous algorithms ((Quiroga et al., 2002; Kreuz et al., 2009; Iqbal and Riek, 2016), see Section 2) MECS is characterized by two important extensions: Macro classes and Macro events. The first one introduces the possibility to regroup the classes and compute the synchronization on different levels of abstraction corresponding to a hierarchical organization of the classes. The second extension permits to compute the synchronization between aggregations of events belonging to different classes.
4.1. Macro Classes
Let’s define as the power set of the event classes set minus the empty set, i.e., .
has cardinality . For example, if , will contain the following elements:
We then define e.g., ) and we consider as the new set of event classes, i.e., in the presented example, with and .
It is possible to take subsets of the original elements of , combining classes and merging them in macro classes. Each generated macro class is actually a single class, or the combination of two or more classes of . MECS consider each item of each set in as belonging to the same class. Synchronization is computed using the set of equations explained in Section 3.1 and Section 3.3 by using as input event class set.
It is important to note that events that belong to one of the original classes can belong to more than one macro class e.g., events of class may belong to both and in .
4.2. Macro events
Events can be grouped into macro-events i.e., aggregations defined by a set of constraints. An example of a macro-event is a sequence of events, where the constraint to be satisfied is the order of occurrence of each event in the sequence. By considering again , a sequence is defined as an ordered -tuple of elements (with repetitions) where each element is referred by a sequence index: Starting from the elements of , examples of sequences are , , and so forth.
Let’s define the following quantities:
: i-th element of sequence that is an event class i.e., ,
: the occurrence time of a generic event found in the time series and that belongs to the class of sequence i.e., any with .
: the Inter-Event Interval i.e. the maximum time allowed between two events that belongs to two consecutive classes and of sequence not to interrupt the sequence.
Then, within a time series , a particular sequence is detected if the following three conditions are true:
no other occurs in [, ] where
When a sequence is detected within a time series , it will be treated as an event belonging to a new class named . Since sequences allows repetitions of the same elements, it is possible to define an infinite number of sequences. Let us call the set of all the sequences defined starting by .
The synchronization degree between sequences is computed using the set of equations explained in Section 3.1 with used as input event class set.
5. MECS Implementation
In this section we present an implementation of MECS algorithm. The schema represented in Figure 1 provides a graphical representation of a sample MECS application. Individual elements are shown with their interrelations.
For simplicity, we assume that all the input signals are being generated at the same time and with a fixed frequency. Input signals are sampled and streamed to the Discrete Events Generator module, that:
identifies the presence of events in the input streams and dispatches them into classes (through event detection techniques).
generates discrete output streams called Channels and forward them to the Time Series Generators modules. Each channel correspond to a single event class of the set (see Section 3).
Event detection and differentiation techniques are deliberately undefined because they strictly depends on each specific application context. From the The Discrete Events Generator events are sent to the Time Series Generators. Each Time Series Generators module performs the following actions:
for each channel fills a buffer with samples taken from the channel streams.
fills an internal matrix ( rows and columns) with the produced buffers, where .
forwards the matrix to the MECS algorithm module.
Finally, before explaining the MECS algorithm module, let us first introduce a set of auxiliary data structures:
: data streams used by the Discrete Events Generator that identifies events, and dispatches them correctly, i.e., contains events of class .
: represents a single portion of data. During the execution of the algorithm, each channel stream is divided into buffers of size .
: matrix of channels and samples. Namely each has rows and columns. The value of each element of the matrix determines the presence (value ) or absence (value ) of an event.
or Event Class Matrix: stores all the absolute positions of all the detected events. Each element of the matrix is a list of positions.
and are internal data structures used to store the values of and .
The , and data structure are re-initialized every time a new buffer arrives.
In Algorithm 1, the body of the main routine of MECS is reported. The main routine runs at every received buffer. To correctly compute the synchronization vector, at each execution cycle, the algorithm stores a pre-fixed number of samples taken from the last processed buffer. Such samples are stored in a support data structure called Accumulator buffer. The dimension of Accumulator buffer is computed according to and on the number of overlapped samples . Possible cases are reported in Figures 3 and 4.
Next, the content of the Accumulator buffer is concatenated to the next input buffer as shown in Figure 2 resulting in a Merged Buffer of size .
The initialization function Init() fills the (Event Class Matrix) with all the positions of all the events found in all the available channels. Absolute positions represents the occurrence timings of the events in the whole period of execution. To compute the absolute position of the detected events, the MECS algorithm uses the following quantities:
the number of received buffers
the original dimension of the buffers
the dimension of the accumulator buffer
the number of overlapping samples
the relative position of an event (the position of the sample in the buffer ).
The absolute position of each event is computed by using the number of received buffers () and an equals to the difference between and , as explained in Algorithm 2.
The Compute() routine calculates the distances between all the events found in all the possible pairs of time series, using the absolute positions stored in , and saves the results in the matrix. When this routine completes its execution, the matrix stores the total contribution to synchronization for each pair of time series and for each event class .
The Finalize() routine performs the following steps:
computes pairwise synchronization of the events of class for each pair of time series .
computes the overall synchronization for each class dividing pairwise synchronizations by the set of all the 2-combinations of the considered time series.
5.3. Computational complexity
The core procedure of MECS is defined in Algorithm 3, which is executed for each input time series buffer. The size of the input data of the procedure is: time series; channels per time series; samples per channel. Consequently, the complexity of the algorithm is:
In most applications . Provided that, we conclude that the complexity of the algorithm is .
We tested the algorithm in the case study illustrated in Section 7 on a Intel i7-6700k CPU @4 GhZ, 16 Gb, Windows 10 machine. With an input consisting of time series with channels and samples the algorithm took on average 6.8 seconds over 10 executions.
6. Application on synthetic data
s We present two application examples of the MECS algorithm. We manually construct some signals and events sequences, and we provide them as input to the algorithm, reporting and commenting the corresponding output.
6.1. Inter-class synchronization between two time series
We assume that the input signal under investigation (shown in the top plot of Figure 5) is the result of the composition of three signals:
a large sinusoidal signal with constant frequency (the second plot from top),
a smaller sinusoidal signal with decreasing frequency (the third plot from top),
and a noise component (the fourth plot from top).
The aim of this example is to demonstrate how the MECS algorithm can be exploited to find out the frequency of the main harmonic of a signal. The main harmonic is the second signal shown from the top of Figure 5. To do that, we propose to compute the synchronization of the input signal with a reference signal corresponding to the input signal main harmonic (see Figure 6). We first extract the peaks values of the three input signal components and the reference signal: the peaks are highlighted as dots in Figure 5 and 6. Afterwards, we create 2 time-series and :
contains the events of classes , , , corresponding to the peaks of, respectively, the input signal and three components;
contains the events of class corresponding to the reference signal peaks.
and are provided as input to the MECS algorithm, which computes inter-class synchronization between all the pairs of classes in and (that is, for each it computes between events of and ) and provides the values reported in Figure 7 as output. In this example, the MECS algorithm is set up with a value of .
As illustrated in Figure 7, the synchronization between the input signal and the reference signal (the first plot from top) exhibits a rhythmical (quasi-periodic) pattern with a high constant amplitude. The same exists for the first component, which is identical to the reference signal. Conversely, the second and third components of the input signal, that is, the signal with a variable frequency and the noise component, do not exhibit the same kind of pattern.
6.2. Macro events synchronization between two time series
In this second example we apply MECS to two input time series and consisting of events belonging to three classes (i.e., , , ) (see Figure 8). Such events are represented within the time series by the corresponding positive integer numbers , while, when no event is triggered, a value of zero is set. For example, in Figure 8 starts with an event of class , then no events are triggered for the second sample of the time series, then an event of class is triggered, and so on.
We set up the MECS algorithm to detect synchronization of macro events (sequences) consisting of an event of class , followed by an event of class , followed by an event of class (the concept of sequence has been introduced in Section 4.2). We chose a value of is . The output of the algorithm, illustrated in Figure 9, is the amount of synchronization between = events in the two input time series. The Figure shows that the outputs are non-discrete values, depending on the distance.
7. A Case Study
We describe an example of application of MECS to measure multi-modal, intra-personal synchronization between respiration phases and kinetic energy captured while performing body movements characterized by different expressive qualities.
Taking a breath is a physical action that can influence the body movements performed at the same time. Similarly, body movements expressing abrupt changes of velocity and acceleration can influence the respiration pattern. Rhythm of respiration synchronizes with repetitive motor activities such as running (Bernasconi and Kohl, 1993; Hoffmann et al., 2012), or rowing (Bateman et al., 2006). Moreover, respiration plays an important role in learning physical activities, such as yoga or tai-chi.
7.1. Definitions of expressive qualities
We analyze synchronization on multi-modal data of the dancers performing different expressive qualities. During a performance, dancers are used to display a large variety of expressive qualities. As in (Lussu et al., 2016), we focus on movements displaying two particular expressive qualities: Fluidity and Impulsivity.
A fluid movement is characterized by the following properties: (i) the movements of every single body joint are smooth; (ii) energy is free to propagate along the kinematic chains of the body (e.g., from head to trunk, from shoulders to arms) according to a coordinated wave-like propag