Modelling sparsity, heterogeneity, reciprocity and community structure in temporal interaction data

1 Introduction

There is a growing interest in modelling and understanding temporal dyadic interaction data. Temporal interaction data take the form of time-stamped triples $(t, i, j)$ indicating that an interaction occurred between individuals $i$ and $j$ at time $t$ . Interactions may be directed or undirected. Examples of such interaction data include commenting a post on an online social network, exchanging an email, or meeting in a coffee shop. An important challenge is to understand the underlying structure that underpins these interactions. To do so, it is important to develop statistical network models with interpretable parameters, that capture the properties which are observed in real social interaction data.

One important aspect to capture is the community structure of the interactions. Individuals are often affiliated to some latent communities (e.g. work, sport, etc.), and their affiliations determine their interactions: they are more likely to interact with individuals sharing the same interests than to individuals affiliated with different communities. An other important aspect is reciprocity. Many events are responses to recent events of the opposite direction. For example, if Helen sends an email to Mary, then Mary is more likely to send an email to Helen shortly afterwards. A number of papers have proposed statistical models to capture both community structure and reciprocity in temporal interaction data (Blundell et al., 2012; Dubois et al., 2013; Linderman and Adams, 2014). They use models based on Hawkes processes for capturing reciprocity and stochastic block-models or latent feature models for capturing community structure.

In addition to the above two properties, it is important to capture the global properties of the interaction data. Interaction data are often sparse: only a small fraction of the pairs of nodes actually interact. Additionally, they typically exhibit high degree (number of interactions per node) heterogeneity: some individuals have a large number of interactions, whereas most individuals have very few, therefore resulting in empirical degree distributions being heavy-tailed. As shown by Karrer and Newman (2011), Gopalan et al. (2013) and Todeschini et al. (2016)

, failing to account explicitly for degree heterogeneity in the model can have devastating consequences on the estimation of the latent structure.

Recently, two classes of statistical models, based on random measures, have been proposed to capture sparsity and power-law degree distribution in network data. The first one is the class of models based on exchangeable random measures (Caron and Fox, 2017; Veitch and Roy, 2015; Herlau et al., 2016; Borgs et al., 2018; Todeschini et al., 2016; Palla et al., 2016; Janson, 2017a). The second one is the class of edge-exchangeable models (Crane and Dempsey, 2015; 2018; Cai et al., 2016; Williamson, 2016; Janson, 2017b; Ng and Silva, 2017). Both classes of models can handle both sparse and dense networks and, although the two constructions are different, connections have been highlighted between the two approaches (Cai et al., 2016; Janson, 2017b).

The objective of this paper is to propose a class of statistical models for temporal dyadic interaction data that can capture all the desired properties mentioned above, which are often found in real world interactions. These are sparsity, degree heterogeneity, community structure and reciprocity. Combining all the properties in a single model is non trivial and there is no such construction to our knowledge. The proposed model generalises existing reciprocating relationships models (Blundell et al., 2012) to the sparse and power-law regime. Our model can also be seen as a natural extension of the classes of models based on exchangeable random measures and edge-exchangeable models and it shares properties of both families. The approach is shown to outperform alternative models for link prediction on a variety of temporal network datasets.

The construction is based on Hawkes processes and the (static) model of Todeschini et al. (2016) for sparse and modular graphs with overlapping community structure. In Section 2, we present Hawkes processes and compound completely random measures which form the basis of our model’s construction. The statistical model for temporal dyadic data is presented in Section 3 and its properties derived in Section 4. The inference algorithm is described in Section 5. Section 6 presents experiments on four real-world temporal interaction datasets.

2 Background material

2.1 Hawkes processes

Let $(t_{k})_{k \geq 1}$ be a sequence of event times with $t_{k} \geq 0$ , and let $H_{t} = (t_{k} | t_{k} \leq t)$ the subset of event times between time $0$ and time $t$ . Let $N (t) = \sum_{k \geq 1} 1_{t_{k} \leq t}$ denote the number of events between time $0$ and time $t$ , where $1_{A} = 1$ if $A$ is true, and 0 otherwise. Assume that $N (t)$ is a counting process with conditional intensity function $λ (t)$ , that is for any $t \geq 0$ and any infinitesimal interval $d t$

Pr (N (t + d t) - N (t) = 1 | H_{t}) = λ (t) d t .

(1)

Consider another counting process $~ N (t)$ with the corresponding $({~ t}_{k})_{k \geq 1}, {~ H}_{t}, ~ λ (t)$ . Then, $N (t), ~ N (t)$ are mutually-exciting Hawkes processes (Hawkes, 1971) if the conditional intensity functions $λ (t)$ and $~ λ (t)$ take the form

λ (t) = μ + \int_{0}^{t} g_{ϕ} (t - u) d ~ N (u) ~ λ (t) = ~ μ + \int_{0}^{t} g_{~ ϕ} (t - u) d N (u)

where $μ = λ (0) > 0, ~ μ = ~ λ (0) > 0$ are the base intensities and $g_{ϕ}, g_{~ ϕ}$ non-negative kernels parameterised by $ϕ$ and $~ ϕ$ . This defines a pair of processes in which the current rate of events of each process depends on the occurrence of past events of the opposite process.

Assume that $μ = ~ μ, ϕ = ~ ϕ$ and $g_{ϕ} (t) \geq 0$ for $t > 0$ , $g_{ϕ} (t) = 0$ for $t < 0$ . If $g_{ϕ}$ admits a form of fast decay then this results in strong local effects. However, if it prescribes a peak away from the origin then longer term effects are likely to occur. We consider here an exponential kernel

g_{ϕ} (t - u) = η e^{- δ (t - u)}, t > u

(2)

where $ϕ = (η, δ)$ . $η \geq 0$ determines the sizes of the self-excited jumps and $δ > 0$ is the constant rate of exponential decay. The stationarity condition for the processes is $η < δ$ . Figure 1 gives an illustration of two mutually-exciting Hawkes processes with exponential kernel and their conditional intensities.

Figure 1: Illustration of mutually-exciting Hawkes processes with exponential kernel with parameters $μ = 0.25$ , $η = 1$ and $δ = 2$ . (a) Counting process $N (t)$ and (b) its conditional intensity $λ (t)$ . (c) Counting process $~ N (t)$ and its conditional intensity $~ λ (t)$ .

2.2 Compound completely random measures

A homogeneous completely random measure (CRM) (Kingman, 1967; 1993) on $R_{+}$ without fixed atoms nor deterministic component takes the form

W = \sum i \geq 1 w_{i} δ_{θ_{i}}

(3)

where $(w_{i}, θ_{i})_{i \geq 1}$ are the points of a Poisson process on $(0, \infty) \times R_{+}$ with mean measure $ρ (d w) H (d θ)$ where $ρ$ is a Lévy measure, $H$ is a locally bounded measure and $δ_{x}$ is the dirac delta mass at $x$ . The homogeneous CRM is completely characterized by $ρ$ and $H$ , and we write $W \sim CRM (ρ, H)$ , or simply $W \sim CRM (ρ)$ when $H$ is taken to be the Lebesgue measure. Griffin and Leisen (2017) proposed a multivariate generalisation of CRMs, called compound CRM (CCRM). A compound CRM $(W_{1}, \dots, W_{p})$ with independent scores is defined as

W_{k} = \sum i \geq 1 w_{i k} δ_{θ_{i}}, k = 1, \dots, p

(4)

where $w_{i k} = β_{i k} w_{i 0}$ and the scores $β_{i k} \geq 0$

are independently distributed from some probability distribution

F_{k}

and

W_{0} = \sum_{i \geq 1} w_{i 0} δ_{θ_{i}}

is a CRM with mean measure

ρ_{0} (d w_{0}) H_{0} (d θ)

. In the rest of this paper, we assume that

F_{k}

is a gamma distribution with parameters

(a_{k}, b_{k})

H_{0} (d θ) = d θ

is the Lebesgue measure and

ρ_{0}

is the Lévy measure of a generalized gamma process

ρ_{0} (d w) = \frac{1}{Γ (1 - σ)} w^{- 1 - σ} e^{- τ w} d w

(5)

where $σ \in (- \infty, 1)$ and $τ > 0$ .

3 Statistical model for temporal interaction data

Consider temporal interaction data of the form $D = (t_{k}, i_{k}, j_{k})_{k \geq 1}$ where $(t_{k}, i_{k}, j_{k}) \in R_{+} \times N_{*}^{2}$ represents a directed interaction at time $t_{k}$ from node/individual $i_{k}$ to node/individual $j_{k}$ . For example, the data may correspond to the exchange of messages between students on an online social network.

We use a point process $(t_{k}, U_{k}, V_{k})_{k \geq 1}$ on $R_{+}^{3}$ , and consider that each node $i$ is assigned some continuous label $θ_{i} \geq 0$ . the labels are only used for the model construction, similarly to (Caron and Fox, 2017; Todeschini et al., 2016), and are not observed nor inferred from the data. A point at location $(t_{k}, U_{k}, V_{k})$ indicates that there is a directed interaction between the nodes $U_{k}$ and $V_{k}$ at time $t_{k}$ . See Figure 2 for an illustration.

Figure 2: (Left) Representation of the temporal dyadic interaction data as a point process on $R_{+}^{3}$ . A point at location $(τ, θ_{i}, θ_{j})$ indicates a directed interaction from node $i$ to node $j$ at time $τ$ , where $θ_{i} > 0$ and $θ_{j} > 0$ are labels of nodes $i$ and $j$ . Interactions between nodes $i$ and $j$ , arise from a Hawkes process $N_{i j}$ with conditional intensity $λ_{i j}$ given by Equation (7). Processes $N_{i k}$ and $N_{j k}$ are not shown for readability. (Right) Conditional intensities of processes $N_{i k} (t)$ , $N_{i j} (t)$ and $N_{j k} (t)$ .

For a pair of nodes $i$ and $j$ , with labels $θ_{i}$ and $θ_{j}$ , let $N_{i j} (t)$ be the counting process

N_{i j} (t) = \sum k | (U_{k}, V_{k}) = (θ_{i}, θ_{j}) 1_{t_{k} \leq t}

(6)

for the number of interactions between $i$ and $j$ in the time interval $[0, t]$ . For each pair $i, j$ , the counting processes $N_{i j}, N_{j i}$ are mutually-exciting Hawkes processes with conditional intensities

λ_{i j} (t) = μ_{i j} + \int_{0}^{t} g_{ϕ} (t - u) d N_{j i} (u), λ_{j i} (t) = μ_{j i} + \int_{0}^{t} g_{ϕ} (t - u) d N_{i j} (u)

(7)

where $g_{ϕ}$ is the exponential kernel defined in Equation (2). Interactions from individual $i$ to individual $j$ may arise as a response to past interactions from $j$ to $i$ through the kernel $g_{ϕ}$ , or via the base intensity $μ_{i j}$ . We also model assortativity so that individuals with similar interests are more likely to interact than individuals with different interests. For this, assume that each node $i$ has a set of positive latent parameters $(w_{i 1}, \dots, w_{i p}) \in R_{+}^{p}$ , where $w_{i k}$ is the level of its affiliation to each latent community $k = 1, \dots, p$ . The number of communities $p$ is assumed known. We model the base rate

μ_{i j} = μ_{j i} = p \sum k = 1 w_{i k} w_{j k} .

(8)

Two nodes with high levels of affiliation to the same communities will be more likely to interact than nodes with affiliation to different communities, favouring assortativity.

In order to capture sparsity and power-law properties and as in Todeschini et al. (2016), the set of affiliation parameters $(w_{i 1}, \dots, w_{i p})$ and node labels $θ_{i}$ is modelled via a compound CRM with gamma scores, that is $W_{0} = \sum_{i = 1}^{\infty} w_{i 0} δ_{θ_{i}} \sim CRM (ρ_{0})$ where the Lévy measure $ρ_{0}$ is defined by Equation (5), and for each node $i \geq 1$ and community $k = 1, \dots, p$

w_{i k} = w_{i 0} β_{i k}, where β_{i k} % ind \sim Gamma (a_{k}, b_{k}) .

(9)

The parameter $w_{i 0} \geq 0$ is a degree correction for node $i$ and can be interpreted as measuring the overall popularity/sociability of a given node $i$ irrespective of its level of affiliation to the different communities. An individual $i$ with a high sociability parameter $w_{i 0}$ will be more likely to have interactions overall than individuals with low sociability parameters. The scores $β_{i k}$ tune the level of affiliation of individual $i$ to the community $k$ . The model is defined on $R_{+}^{3}$ . We assume that we observe interactions over a subset $[0, T] \times [0, α]^{2} \subseteq R_{+}^{3}$ where $α$ and $T$ tune both the number of nodes and number of interactions. The whole model is illustrated in Figure 2.

Figure 3: Degree distribution for a Hawkes graph with different values of $σ$ (left) and $τ$ (right). The degree of a node $i$ is defined as the number of nodes with whom it has at least one interaction. The value $σ$ tunes the slope of the degree distribution, larger values corresponding to a higher slope. The parameter $τ$ tunes the exponential cut-off in the degree distribution.

The model admits the following set of hyperparameters, with the following interpretation:

∙

The hyperparameters

ϕ = (η, δ)

where

η \geq 0

and

δ \geq 0

of the kernel

g_{ϕ}

tune the reciprocity.

∙

The hyperparameters

(a_{k}, b_{k})

tune the community structure of the interactions.

a_{k} / b_{k} = E [β_{i k}]

tunes the size of community

k

while

a_{k} / b_{k}^{2} = var (β_{i k})

tunes the variability of the level of affiliation to this community; larger values imply more separated communities.

∙

The hyperparameter

σ

tunes the sparsity and the degree heterogeneity: larger values imply higher sparsity and heterogeneity. It also tunes the slope of the degree distribution. Parameter

τ

tunes the exponential cut-off in the degree distribution. This is illustrated in Figure 3.

∙

Finally, the hyperparameters

α

and

T

tune the overall number of interactions and nodes.

We follow (Rasmussen, 2013) and use vague Exponential $(0.01)$ priors on $η$ and $δ$ . Following Todeschini et al. (2016) we set vague Gamma $(0.01, 0.01)$ priors on $α$ , $1 - σ$ , $τ$ , $a_{k}$ and $b_{k}$ . The right limit for time window, $T$ is considered known.

4 Properties

4.1 Connection to sparse vertex-exchangeable and edge-exchangeable models

The model is a natural extension of sparse vertex-exchangeable and edge-exchangeable graph models. Let $z_{i j} (t) = 1_{N_{i j} (t) + N_{j i} (t) > 0}$

be a binary variable indicating if there is at least one interaction in

[0, t]

between nodes

i

and

j

in either direction. We assume

Pr (z_{i j} (t) = 1 | (w_{i k}, w_{j k})_{k = 1, \dots, p}) = 1 - e^{- 2 t \sum_{k = 1}^{p} w_{i k} w_{j k}}

which corresponds to the probability of a connection in the static simple graph model proposed by Todeschini et al. (2016). Additionally, for fixed $α > 0$ and $η = 0$ (no reciprocal relationships), the model corresponds to a rank- $p$ extension of the rank-1 Poissonized version of edge-exchangeable models considered by Cai et al. (2016) and Janson (2017a). The sparsity properties of our model follow from the sparsity properties of these two classes of models.

4.2 Sparsity

The size of the dataset is tuned by both $α$ and $T$

. Given these quantities, both the number of interactions and the number of nodes with at least one interaction are random variables. We now study the behaviour of these quantities, showing that the model exhibits sparsity. Let

I_{α, T}, E_{α, T}, V_{α, T}

be the overall number of interactions between nodes with label

θ_{i} \leq α

until time

T

, the total number of pairs of nodes with label

θ_{i} \leq α

who had at least one interaction before time

T

, and the number of nodes with label

θ_{i} \leq α

who had at least one interaction before time

T

respectively.

\begin{matrix} V_{α, T} & = \sum i 1_{\sum_{j \neq i} N_{i j} (T) 1_{θ_{j} \leq α} > 0} 1_{θ_{i} \leq α} I_{α, T} = \sum i \neq j N_{i j} (T) 1_{θ_{i} \leq α} 1_{θ_{j} \leq α} E_{α, T} & = \sum i < j 1_{N_{i j} (T) + N_{j i} (T) > 0} 1_{θ_{i} \leq α} 1_{θ_{j} \leq α} \end{matrix}

We provide in the supplementary material a theorem for the exact expectations of $I_{α, T}$ , $E_{α, T}$ $V_{α, T}$ . Now consider the asymptotic behaviour of the expectations of $V_{α, T}$ , $E_{α, T}$ and $I_{α, T}$ , as $α$ and $T$ go to infinity.¹¹1 We use the following asymptotic notations. $X_{α} = O (Y_{α})$ if $lim X_{α} / Y_{α} < \infty$ , $X_{α} = ω (Y_{α})$ if $lim Y_{α} / X_{α} = 0$ and $X_{α} = Θ (Y_{α})$ if both $X_{α} = O (Y_{α})$ and $Y_{α} = O (X_{α})$ . Consider fixed $T > 0$ and $α$ that tends to infinity. Then,

E [V_{α, T}] = {\begin{matrix} Θ (α) & if σ < 0 ω (α) & if σ \geq 0 \end{matrix}, E [E_{α, T}] = Θ (α^{2}), E [I_{α, T}] = Θ (α^{2})

as $α$ tends to infinity. For $σ < 0$ , the number of edges and interactions grows quadratically with the number of nodes, and we are in the dense regime. When $σ \geq 0$ , the number of edges and interaction grows subquadratically, and we are in the sparse regime. Higher values of $σ$ lead to higher sparsity. For fixed $α$ ,

E [V_{α, T}] = ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} Θ (1) & if σ < 0 Θ (log T) & if σ = 0 Θ (T^{σ}) & if σ > 0 \end{matrix}, E [E_{α, T}] = ⎧ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ \begin{matrix} Θ (1) & if σ < 0 O (log T) & if σ = 0 O (T^{3 σ / 2}) & if σ \in (0, 1 / 2] O (T^{(1 + σ) / 2}) & if σ \in (1 / 2, 1) \end{matrix}

and $E [I_{α, T}] = Θ (T)$ as $T$ tends to infinity. Sparsity in $T$ arises when $σ \geq 0$ for the number of edges and when $σ > 1 / 2$ for the number of interactions. The derivation of the asymptotic behaviour of expectations of $V_{α, T}$ , $E_{α, T}$ and $I_{α, T}$ follows the lines of the proofs of Theorems 3 and 5.3 in (Todeschini et al., 2016) $(α \to \infty)$ and Lemma D.6 in the supplementary material of (Cai et al., 2016) $(T \to \infty)$ , and is omitted here.

5 Approximate Posterior Inference

Assume a set of observed interactions $D = (t_{k}, i_{k}, j_{k})_{k \geq 1}$ between $V$ individuals over a period of time $T$ . The objective is to approximate the posterior distribution $π (ϕ, ξ | D)$ where $ϕ$ are the kernel parameters and $ξ = ((w_{i k})_{i = 1, \dots,}$

Modelling sparsity, heterogeneity, reciprocity and community structure in temporal interaction data

Online Community Detection for Event Streams on Networks

A Bayesian model for sparse graphs with flexible degree distribution and overlapping community structure

A Nonparametric Bayesian Model for Sparse Temporal Multigraphs

Deep Structural Point Process for Learning Temporal Interaction Networks

An Empirical Study: Extensive Deep Temporal Point Process

Network Hawkes Process Models for Exploring Latent Hierarchy in Social Animal Interactions

Nonparametric Embeddings of Sparse High-Order Interaction Events

1 Introduction