Analysis of sojourn time distributions for semi-Markov models

1 Introduction

This report describes and analyses sojourn time distributions which can then be applied to parametrise the semi-Markov model (SMM). Soujourn time distributions are the distributions that govern how long a semi-Markov model remains in each state. Unlike usual Markov-models where the sojourn time distribution is geometric, semi-Markov models allow any distribution to be used.

Such sojourn time distributions have been modelled before by using known distributions such as the gamma, Poisson, multinomial, more general exponential family members and Coxian phase-type distributions particularly in the context of hidden semi-Markov models (HSMMs) [14, 19, 17, 2, 11, 7, 1]. An R-package is available for modelling some of these distributions [3]. Non-parametric versions have also been studied in the context of expectation maximisation for HSMMs [2, 19], however the underlying properties of these distributions have not been studied in the literature and we seek to do this Section 2. Further, the two subfamilies of the distributions we study have not previously appeared in the literature.

In general, the sojourn time distributions we study can be considered discrete generalisations of the geometric distribution in finite or infinite time. There are many approaches for generalisations of the geometric distribution [15, 18, 9]

, although they usually only consider infinite time. For example, considering the exponential as the continuous analog of the geometric distribution and the gamma distribution as generalising the exponential distribution, then discretisations of the gamma distribution can be considered generalisations of the geometric distribution as in

[4].

Unlike other geometric generalisations, ours will involve the parameters $ρ$

. These parameters can be considered as the marginal probabilities that a semi-Markov chain stays in the same state for the next time step, as in

Section 2.4. For a Markov chain, the geometric sojourn time distribution corresponds to constant

ρ

for each state, and here we extend this to consider linear and simple polynomial factor models for

ρ

. We are not aware of any other research on generalisations of the geometric distribution that has such an approach.

Markov models are often applied to model behaviours where only small amounts of data may be available, so we require parsimonious models which nevertheless have sufficient generality to capture different kinds of qualitative sojourn time behaviours. We also require models for which statistically efficient estimation methods can be defined. At this stage, we make the obvious point that whilst maximum likelihood estimators (MLEs) are statistically unbiased and efficient for large sample sizes [6]; this may not be the case for small sample sizes such in our experimental data. So the study of the performance of MLEs forms an important part of this study which we undertake in Section 3.

Statistical inference methods for hidden SMMs (HSMMs) are typically more computationally demanding than the equivalents for HMMs, so it is also of interest to determine whether an observed process has Markovian state transitions or the more general semi-Markov ones. We show in Section 4 how we approached this for our experimental data, which we can use as a methodology for testing this kind of hypothesis in general. We also see a strong fit between this experimental data and the simple polynomial factor model, suggesting the theory performs well in experimental settings.

Overall, the contents of this report gives a novel approach to modelling sojourn time distributions that will be of use for (hidden) semi-Markov models, and already holds appeal for modelling experimental data. The results concerning the sojourn time distributions may be of independent interest for the discrete distribution literature.

1.1 Layout of the report

The contents of this report is as follows. In Section 2.1 we define the family of distributions we aim to study, which generalises the geometric distribution. We give general properties including the mean, variance, moment and probability generating functions as well as results that determine when the moments exist in general. These results are new. In Section 2.2 and Section 2.3 we define the subfamilies of interest, the linear factor model and simple polynomial factor models, and describe their distributions, which is also new research. In Section 2.4 we show the relationship to semi-Markov models and discuss further results in this area.

In Section 3 we study how to determine the MLEs of the subfamilies of interest, studying the linear factor model in Section 3.1 and the polynomial factor models in Section 3.2. In each case, we determine numerically the MLEs from various samples and show how the variance of the MLEs closely follows the inverse Fisher information which is related to the Cramér-Rao lower bound [6] suggesting our numerical methods are behaving efficiently. We study how this changes with different sample sizes. We also discuss methods to determine the size of the support of the distribution, i.e. the maximum sojourn time.

In Section 4 we analyse our experimental data. We show how the data has sojourn times which do not appear to follow a geometric distribution, and why a simple polynomial factor model is a better fit. We then find the MLEs for such a model for each of the three tasks, including determining the maximum sojourn time, and discuss how well these models fit. We are satisfied with the model for tasks 1 and 3, and suggest some improvements for task 2. In Section 5 we conclude and discuss future work.

In Appendix A the second author expands on the work for the linear factor model. He confirms the results in Section 3.1 with separate simulations, as well as shows how the expected log-likelihood explains some of the behaviour of the MLEs and how to further understand estimation when the maximum sojourn time is unknown. In Appendix B we include the experimental data analysed in Section 4.

2 Generalising the geometric distribution

Here we define a family of discrete distributions that generalise the geometric distribution in both finite and infinite state cases. We give properties of this family, including formulae for the moments, and prove results relating to their existence in the infinite support case. We note that this family is particularly general as it can be used to characterise all discrete distributions that are supported on $1, 2, 3, \dots, T$ with $T$ a positive integer and possibly infinite.

We then focus on two specific subfamilies, the linear factor model in Section 2.2 and some simple polynomial factor models in Section 2.3. Finally we detail the relationship to semi-Markov models in Section 2.4 and the relationship to matrix analytic methods.

As far as we are aware, all results in sections Section 2.1, Section 2.2 and Section 2.3 are new contributions to the literature on discrete distributions, which we expect will also lead to new contributions in Markov processes research.

2.1 Definition and properties

We define the family of discrete distributions that we intend to study then study properties of the distributions.

Definition 2.1.

Take any $ρ (1), ρ (2), \dots, ρ (T - 1) \in (0, 1]$ with $ρ (1) > 0$ , and set $ρ (T) = 0$ , where $T$ is a positive integer (which we may allow to go to $\infty$ ). Then we can define a discrete distribution with support contained in ${1, 2, \dots T}$ having probability mass function (PMF)

	$f (k) = (1 - ρ (k)) ρ (k - 1) ρ (k - 2) \dots ρ (1)$		(2.1)
	$= (1 - ρ (k)) k - 1 \prod t = 1 ρ (t) for k = 1, 2, \dots, T,$

where we use the convention that product $\prod_{t = 1}^{0} ρ (t)$ is equal to 1.

Note that we could allow some $ρ (k) = 0$ for $1 \leq k < T$ , but then $f (k) = 1$ and then $f (k + i) = 0$ for all $i = 1, 2, \dots, T$ , reducing the support of $f$ . Also, whenever $ρ (k) = 1$ we have $f (k) = 0$ , so as we see below, this is sufficiently general to capture all discrete distributions with support contained in ${1, 2, \dots \infty}$ . We can also shift the distribution of $f$ to distribution $f_{t}$ with support contained within ${1 + t, 2 + t, \dots, T + t}$ for any integer $t$ by setting $f_{t} (t + k) = f (k)$ . We will consider this in Section 4.

It is clear that each $f (k) \in [0, 1]$ as products of numbers in $[0, 1]$ remains in $[0, 1]$ . We have

	$T \sum k = 1 f (k)$	$= T \sum k = 1 (1 - ρ (k)) ρ (k - 1) ρ (k - 2) \dots ρ (1)$
		$= 1 + T \sum k = 2 k - 1 \prod t = 1 ρ (t) - T \sum k = 1 k \prod t = 1 ρ (k)$
		$= 1 + T - 1 \sum k = 1 k \prod t = 1 ρ (t) - T \sum k = 1 k \prod t = 1 ρ (k)$
		$= 1 - T \prod t = 1 ρ (k) = 1,$

as $ρ (T) = 0$ . So $f$ gives a valid PMF.

If we let $ρ (k) = p$ for some fixed $p \in (0, 1)$ and take $T \to \infty$ then this is gives the geometric distribution with PMF for $k = 1, 2, \dots$

f (k) = p^{k - 1} (1 - p) .

(2.2)

In general, we can interpret the $ρ (k)$ as the probability of an event occurring at time $k$ given it also occurred at time $k - 1, k - 2, \dots, 1$ . Then $f (k)$ is the probability of the event not occurring at time $k$ given it occurred at time $1, 2, \dots, k - 1$ . In the geometric case, the time steps are independent and $ρ (k)$ does not change, however the general $f$ described above allows for dependence between each time. We expand on this in Section 2.4 where we show how this relates to a semi-Markov model.

Lemma 2.2.

We have the relationship

T \sum t = k f (t) = 1 - k - 1 \sum t = 1 f (t) = \frac{f (k)}{1 - ρ (k)} = k - 1 \prod t = 1 ρ (t) .

(2.3)

Proof.

By induction, we can show

ρ (k) = \frac{1 - f (1) - f (2) - \dots - f (k)}{1 - f (1) - f (2) - \dots - f (k - 1)} = 1 - \frac{f (k)}{\sum_{t = k}^{T} f (t)} = 1 - \frac{f (k)}{1 - \sum_{t = 1}^{k - 1} f (t)},

(2.4)

and the result follows. ∎

Note that Equation 2.4 shows that any discrete PMF supported $k = 1, 2, \dots, T$ (with $T$ possibly infinite) can be defined in terms of $ρ (k)$ , so in fact creftypecap 2.1 is an alternative way to describe all such distributions. If $F$

is the corresponding cumulative distribution function to

f

, then

ρ (k) = \frac{1 - F (k)}{1 - F (k - 1)}

and also

$F (k)$	$= k \sum t = 1 f (k) = k \sum t = 1 (1 - ρ (t)) t - 1 \prod s = 1 ρ (s)$
	$= 1 + k - 1 \sum t = 1 t - 1 \prod s = 1 ρ (s) - k \sum t = 1 t - 1 \prod s = 1 ρ (s)$
	$= 1 - k \prod t = 1 ρ (t) .$	(2.5)

Note that if $F (k) = 1$ then $ρ (k) = 0$ and this means the distribution has reached the end of its support. We use this in Section 4. The next proposition characterises all moments of the distribution in terms of the $ρ (k)$ .

Proposition 2.3.

For a random variable

X

with PMF

f

as in Equation 2.1 we have moment generating function (MGF)

M_{X} (t) = T - 1 \sum k = 1 (e^{t (k + 1)} - e^{t k}) k \prod t = 1 ρ (k) + e^{t} .

The $n$ -th derivatives are

M_{X}^{(n)} (t) = T - 1 \sum k = 1 ((k + 1)^{n} e^{t (k + 1)} - k^{n} e^{t k}) k \prod t = 1 ρ (t) + e^{t} .

So we have

E (X^{n}) = M_{X}^{(n)} (0) = T - 1 \sum k = 1 ((k + 1)^{n} - k^{n}) k \prod t = 1 ρ (t) + 1,

and then

	$E (X)$	$= T - 1 \sum k = 1 k \prod t = 1 ρ (t) + 1,$
	$Var (X)$	$= E (X) + 2 T - 1 \sum k = 1 k k \prod t = 1 ρ (t) - E (X)^{2} .$

The probability generating function is

P_{X} (z) = T - 1 \sum k = 1 (z^{k + 1} - z^{k}) k \prod t = 1 ρ (t) + z

with $n$ -th derivatives

P_{X}^{(n)} (z) = T - 1 \sum k = n (\frac{(k + 1)! z^{k + 1 - n}}{(k + 1 - n)!} - \frac{k! z^{k - n}}{(k - n)!}) k \prod t = 1 ρ (t) + \frac{d z}{d z^{n}} .

The factorial moments are

E (X (X - 1) \dots (X - n + 1))

= P_{X}^{(n)} (1) = T - 1 \sum k = n \frac{k! n}{(k + 1 - n)!} k \prod t = 1 ρ (t) + \frac{d z}{d z^{n}} .

Proof.

The moment generating function is

	$M_{X} (t)$	$= T \sum k = 1 e^{k t} f (k)$
		$= T \sum k = 1 e^{k t} (1 - ρ (k)) k - 1 \prod t = 1 ρ (t)$
		$= T \sum k = 1 e^{k t} k - 1 \prod t = 1 ρ (t) - T \sum k = 1 e^{k t} k \prod t = 1 ρ (t)$
		$= e^{t} + T \sum k = 2 e^{k t} k - 1 \prod t = 1 ρ (t) - T \sum k = 1 e^{k t} k \prod t = 1 ρ (t)$
		$= e^{t} + T - 1 \sum k = 1 e^{(k + 1) t} k \prod t = 1 ρ (t) - T - 1 \sum k = 1 e^{k t} k \prod t = 1 ρ (t)$
		$= T - 1 \sum k = 1 (e^{t (k + 1)} - e^{t k}) k \prod t = 1 ρ (k) + e^{t}$

The formulas for the derivatives $M_{X}^{(n)} (t)$ and $E (X)$ follow directly. We then have the variance

	$Var (X)$	$= E (X^{2}) - E (X)^{2} = T - 1 \sum k = 1 (2 k + 1) k \prod t = 1 ρ (k) + 1 - E (X)^{2}$
		$= E (X) + 2 T - 1 \sum k = 1 k k \prod t = 1 ρ (k) - E (X)^{2} .$

The probability generating function is

P_{X} (t) = T \sum k = 1 z^{k} f (k) = M_{X} (log (z)) = T - 1 \sum k = 1 (z^{k + 1} - z^{k}) k \prod t = 1 ρ (k) + z .

The $n$ -th derivatives $P_{X}^{(n)} (z)$ are straightforward to calculate, so the factorial moments are

	$E (X (X - 1) \dots (X - n + 1))$	$= P_{X}^{(n)} (1) = T - 1 \sum k = n (\frac{(k + 1)!}{(k + 1 - n)!} - \frac{k!}{(k - n)!}) k \prod t = 1 ρ (t) + \frac{d z}{d z^{n}}$
		$= T - 1 \sum k = n \frac{k! n}{(k + 1 - n)!} k \prod t = 1 ρ (t) + \frac{}{d z} d z^{n} .$

∎

If $T$ is finite, it is clear that all moments are finite. Below we consider the case when $T$ is infinite.

Proposition 2.4.

Let $X$ be a random variable with PMF $f$ as in Equation 2.1 with $T = \infty$ , and say

lim k \to \infty ρ (k) = r

where we must have $r \in [0, 1]$ as $ρ (k) \in [0, 1]$ . Then all moments exist whenever $r < 1$ . If $r = 1$ then some moments may exist.

Proof.

The ratio test says that a series $\sum_{i = 1}^{\infty} a_{i}$ for $a_{i} \in R$ converges whenever

lim k \to \infty ∣ ∣ ∣ \frac{a_{k + 1}}{a_{k}} ∣ ∣ ∣ < 1.

It diverges if the limit is $> 1$ and may or may not converge if this limit is $1$ . Say we have ${lim}_{k \to \infty} ρ (k) = r$ , then we can consider the series

\infty \sum k = 1 k^{n} k \prod t = 1 ρ (t)

for some $n = 1, 2, \dots, \infty$ , which appears in the formula for the moments $E (X^{n})$ in creftypecap 2.3. Then

	$lim k \to \infty ∣ ∣ ∣ \frac{a_{k + 1}}{a_{k}} ∣ ∣ ∣$	$= lim k \to \infty ∣ ∣ ∣ ∣ \frac{(k + 1)^{n} \prod_{t = 1}^{k + 1} ρ (t)}{k^{n} \prod_{t = 1}^{k} ρ (t)} ∣ ∣ ∣ ∣$
		$= lim k \to \infty (1 + \frac{1}{k^{n}}) ρ (k + 1)$
		$\to 1 \cdot r$

therefore all moments centred at $0$ converge when $r < 1$ by the ratio test. As all moments of order $n$ can be written in terms of moments centred at $0$ of up to order $n$ , then all moments converge for $X$ when $r < 1$ . Similarly, they diverge if $r > 1$ (which is not possible as we have $ρ (k) \in [0, 1]$ for all $k$ ) and may or may not converge if $r = 1$ . ∎

Note that we do not necessarily have that ${lim}_{k \to \infty} ρ (k)$ converges, so a more general statement would be than any subsequence of the $ρ (k)$ that converges must converge to something less than $1$ to guarantee existence of the moments. This is equivalent to the following stronger corollary of creftypecap 2.4.

Corollary 2.5.

Let $X$ be a random variable with PMF $f$ as in Equation 2.1 with $T = \infty$ , and say

limsup k \to \infty ρ (k) = r

where we must have $r \in [0, 1]$ as $ρ (k) \in [0, 1]$ . Then all moments exist whenever $r < 1$ . If $r = 1$ then some moments may exist.

If ${limsup}_{k \to \infty} ρ (k) = 1$ , that is, there are (sub)sequences of the $ρ (k)$ which converge to $1$ , more complicated arguments must be considered as in the following example.

Example 2.6.

If $ρ (k) = \frac{k}{k + 1}$ for $k = 1, 2, \dots, \infty$ then ${lim}_{k \to \infty} ρ (k) = 1$ and we have

\infty \sum k = 1 k \prod t = 1 ρ (t) = \infty \sum k = 1 \frac{1}{k + 1},

which diverges and no moments exist. These $ρ$ correspond to PMF

f (k) = \frac{1}{k} - \frac{1}{1 - k} = \frac{1}{k (k + 1)} .

More generally, if $ρ (k) = \frac{k^{m}}{(k + 1)^{m}}$ for some positive integer $m$ then still ${lim}_{k \to \infty} ρ (k) = 1$ , however we have

\infty \sum k = 1 k^{n} k \prod t = 1 ρ (t) = \infty \sum k = 1 \frac{k^{n}}{} (k + 1)^{m} .

When $n + 1 < m$ , we have

	$\infty \sum k = 1 \frac{k^{n}}{(k + 1)^{m}}$	$= \infty \sum k = 1 \frac{1}{(1 + \frac{1}{k})^{n}} \frac{1}{(1 + k)^{2}} \frac{1}{(1 + k)^{m - 2}}$
		$\leq \infty \sum k = 1 \frac{1}{(1 + k)^{2}} = \frac{π^{2}}{6} - 1$

so this sum converges and this means moments up to order $m - 2$ exist. When $n = m - 1$ we have

	$\infty \sum k = 1 \frac{k^{n}}{(k + 1)^{m}}$	$= \infty \sum k = 1 \frac{1}{(1 + \frac{1}{k})^{m - 1}} \frac{1}{1 + k}$
		$\geq \frac{1}{2^{m - 1}} \infty \sum k = 1 \frac{1}{1 + k}$
		$\to \infty$

so this diverges, and all moments of order $m - 1$ or higher do not exist. These $ρ$ correspond to PMF

f (k) = \frac{1}{k^{m}} - \frac{1}{(1 + k)^{m}} .

Example 2.7.

For a random variable $X$ with the geometric distribution $ρ (k) = p$ , $T \to \infty$ as in Equation 2.2 then using creftypecap 2.3 we have mean

E (X) = 1 + \infty \sum j = 1 p^{j} = 1 + \frac{1}{1 - p} - 1 = \frac{1}{1 - p},

and variance

	$Var (X)$	$= E (X) + 2 lim T \to \infty T - 1 \sum k = 1 k k \prod t = 1 ρ (t) - E (X)^{2}$
		$= \frac{1}{1 - p} + 2 lim T \to \infty T - 1 \sum k = 1 k p^{k} - \frac{1}{(1 - p)^{2}}$
		$= \frac{1}{1 - p} + 2 lim T \to \infty p \frac{d}{d p} T - 1 \sum k = 1 p^{k} - \frac{1}{(1 - p)^{2}}$
		$= \frac{1}{1 - p} + 2 lim T \to \infty p \frac{d}{d p} (\frac{1 - p^{T}}{1 - p} - 1) - \frac{1}{(1 - p)^{2}}$
		$= \frac{1}{1 - p} + 2 p \frac{d}{d p} (\frac{1}{1 - p} - 1) - \frac{1}{(1 - p)^{2}}$
		$= \frac{1}{1 - p} + \frac{2 p}{(1 - p)^{2}} - \frac{1}{(1 - p)^{2}}$
		$= \frac{1 - p + 2 p - 1}{(1 - p)^{2}}$
		$= \frac{p}{(1 - p)^{2}} .$

Both of these results are as expected from the geometric distribution.

2.2 Linear factor model

In this section we consider the case where we have a linear $ρ$ , that is

ρ (k) = a k + b for all k = 1, 2, \dots, T .

For this to be valid, we require that $a k + b \in [0, 1]$ for all $k$ . This then means if we send $T \to \infty$ we must make $a = 0$ , resulting in the geometric distribution. If we consider $T$ finite then we require the endpoints $k = 1, T - 1$ are in $[0, 1]$ , which means we need $a + b \in [0, 1]$ , so $a \in [- b, 1 - b]$ and $a (T - 1) + b \in [0, 1]$ , so $a \in [- \frac{b}{T - 1}, \frac{1 - b}{T - 1}]$ . Combining these we have

max {- b, \frac{- b}{T - 1}} \leq a \leq min {1 - b, \frac{1 - b}{T - 1}} .

To make sure the end points are non-overlapping, we then also require that $b \in [- \frac{1}{T}, \frac{T - 1}{T - 2}]$ for $T \geq 3$ .

A natural choice in this case would be to set $a, b$ such that $a (T) + b = 0$ , which gives $a = - b / T$ and $b \in (0, \frac{T}{T - 1}]$ . Then

ρ (k) = b (1 - \frac{k}{T})

and the corresponding $f$ is as follows

	$f (k)$	$= (1 - ρ (k)) ρ (k - 1) ρ (k - 2) \dots ρ (1)$
		$= (1 + b (\frac{k}{T} - 1)) (b (1 - \frac{k - 1}{T})) (b (1 - \frac{k - 2}{T})) \dots (b (1 - \frac{1}{T}))$
		$= (1 + b (\frac{k}{T} - 1)) {(\frac{b}{T})}^{k - 1} (T - k + 1) (T - k + 2) \dots (T - 1)$
		$= (1 + b (\frac{k}{T} - 1)) {(\frac{b}{T})}^{k - 1} \frac{(T - 1)!}{(T - k)!} .$

This is a natural extension of the geometric distribution over a finite domain.

Note that we can similarly use $a$ instead of $b$ , where we have

ρ (k) = a (k - T),

with $b = - a T$ , $a \in [\frac{1}{1 - T}, 0)$ .

We can graph $f$ for certain values of $b$ (and corresponding $a$ ) and $T = 10$ , as in Figure 2.1.

Figure 2.1: Plot of $f$ with $ρ (k) = b (1 - \frac{k}{T}) = a (k - T)$ with $T = 10$ and various $b$ .

We see quite different behaviours depending on the value of $b$ . The larger $b$ exhibit a non-monotonic, uni-modal distribution with maximum near the centre of the support, while the smaller $b$ exhibit more geometric-like qualities, with maximum at $k = 1$ and decreasing for larger $k$ . Varying $T$ gives similar results.

2.3 Simple polynomial factor models

Here we concentrate on models of the form

ρ (k) = a (k - c)^{n} + b

with $ρ (T) = 0$ . We require $n$ to be a fixed positive integer.

We again set $ρ (T) = 0$ which gives $a (T - c)^{n} + b = 0$ , so we must have $a = \frac{- b}{(T - c)^{n}}$ provided we assume $b \neq 0$ . Note that if $b = 0$ and $ρ (T) = 0$ then either $c = T$ and we are considering linear cases $ρ (T) = a (k - T)$ from the previous section, or $a = 0$ and we have $ρ = 0$ .

Given this we assume $c \neq T$ then writing in terms of $b$ we have

	$ρ (k)$	$= b (1 - \frac{(k - c)^{n}}{(T - c)^{n}}),$
	$f (k)$	$= (1 + b (\frac{(k - c)^{n}}{(T - c)^{n}} - 1)) {(\frac{b}{(T - c)^{n}})}^{k - 1} k - 1 \prod t = 1 ((T - c)^{n} - (t - c)^{n}) .$

Similarly, writing in terms of $a$ we have

	$ρ (k)$	$= a ((k - c)^{n} - (T - c)^{n}),$
	$f (k)$	$= (1 + a ((T - c)^{n} - (k - c)^{n})) a^{k - 1} k - 1 \prod t = 1 ((t - c)^{n} - (T - c)^{n}) .$

We require $ρ (k) \in (0, 1]$ for all $k = 1, \dots, T - 1$ . Whether $n$

is even or odd changes the requirements on

b

and

c

to ensure this. We get the conditions that

c \in R ∖ {T}

and that

$n even, c \leq \frac{T + 1}{2},$	$0 \leq b \leq \frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}},$	$\frac{1}{(k - c)^{n} - (T - c)^{n}} \leq a \leq 0$
$n even, c > T,$	$\frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}} \leq b \leq 0,$	$0 \leq a \leq \frac{1}{(k - c)^{n} - (T - c)^{n}}$
$n odd, c < T,$	$0 \leq b \leq \frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}},$	$\frac{1}{(k - c)^{n} - (T - c)^{n}} \leq a \leq 0$
$n odd, c > T,$	$\frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}} \leq b \leq 0,$	$0 \leq a \leq \frac{1}{(k - c)^{n} - (T - c)^{n}} .$

Note that $n$ even and $c \in [\frac{T + 1}{T}, T)$ implies $b = 0$ , which we will again exclude here. With certain applications in mind as in Section 4, we are most interested in the odd $n$ cases particularly $n = 3$ with $c < T$ , however we will consider all cases in generality when possible.

In Figure 2.2 we graph $f$ for certain values of $b$ (and corresponding $a$ ) with $n = 3$ , $c = 4$ and $T = 15$ .

Figure 2.2: Plot of $f$ with $ρ (k) = b (1 - \frac{(k - c)^{n}}{(T - c)^{n}}) = a ((k - c)^{n} - (T - c)^{n})$ with $T = 15$ , $n = 3$ , $c = 4$ and various $b$ .

Note that here we see quite different behaviours depending on the value of $b$ . The larger $b$ exhibit a non-monotonic, uni-modal or even bi-modal distributions with maxima towards $T$ , while the smaller $b$ exhibit more geometric-like qualities, with maximum at $k = 1$ and decreasing for larger $k$ . Varying $T$ gives similar results.

We also graph $f$ while varying $c$ and $b$ with fixed $n = 3$ , and $T = 15$ , as in Figure 2.3

Figure 2.3: Plot of $f$ with $ρ (k) = b (1 - \frac{(k - c)^{n}}{(T - c)^{n}}) = a ((k - c)^{n} - (T - c)^{n})$ with $T = 15$ , $n = 3$ , $c = - 5, - 4, \dots, 14$ and various $b$ . Note that $max (b) = {min}_{k} \frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}}$ .

In Figure 2.4 we further separate out the different behaviours for fixed $b$ and changes in $c$ .

Figure 2.4: Plot of $f$ with $ρ (k) = b (1 - \frac{(k - c)^{n}}{(T - c)^{n}}) = a ((k - c)^{n} - (T - c)^{n})$ with $T = 15$ , $n = 3$ , $c = - 5, - 4, \dots, 14$ and various $b$ that are fixed in each plot. Note that $max (b) = {min}_{k} \frac{(T - c)^{n}}{(T - c)^{n} - (k - c)^{n}}$ .

2.4 Relationship to SMM

In this section we describe the relationship between the family of distributions in creftypecap 2.1 and semi-Markov models. This is one of the motivations for this paper. This section also contains details on how semi-Markov models can be considered using Matrix analytic methods. This will be important in future applications of the sojourn time distributions studied so far.

2.4.1 Sojourn time distributions arising from SMMs

In a semi-Markov model, we have a random process $(X_{n}, τ_{n})$ $n = 1, 2, \dots$ . Here $X_{n}$ is a random variable for each $n$ with state space $S = {1, 2, 3, \dots, s}$ and $τ_{n}$ is a random variable for each $n$ . Either $τ_{n}$ has a continuous state space in $(0, T_{max})$ for some $T_{max} > 0$ possibly infinite or a discrete state space in $1, 2, \dots, T_{max}$ for some integer $T_{max} > 0$ possibly infinite. Here we will only consider discrete state spaces.

Note that we differ from some traditional presentations of semi-Markov models as in [13, §1.9] where often $X_{n}$ is only the transitions of the process and $τ_{n}$ is the times these transitions occur, assumed to be continuous times, giving a Markov renewal process $(X_{n}, τ_{n})$ . Instead, here our $X_{n}$ will be a discrete random process that may remain in the same state as $n$ increments, and the $τ_{n}$ will record the number of times the process has remained in the same state continuously.

We require the independence condition that

	$P ((X_{n + 1}, τ_{n + 1}) = (j, t) \| (X_{n}, τ_{n}), (X_{n - 1}, τ_{n - 1}), \dots, (X_{1}, τ_{1}))$
	$= P ((X_{n + 1}, τ_{n + 1}) = (j, t) \| (X_{n}, τ_{n})) .$

and the time-homogenous condition that

P ((X_{n + m + 1}, τ_{n + m + 1}) | (X_{n + m}, τ_{n + m})) = P ((X_{n + 1}, τ_{n + 1}) | (X_{n}, τ_{n}))

for all $m = 1, 2, \dots$ .

This means that $Z_{n} = (X_{n}, τ_{n})$ is a Markov chain. We require that $τ_{n}$ count the number of consecutive time steps $X_{n} = j$ has spent in state $j$ , so that if we observe that both $X_{n}$ and $X_{n + 1}$ are equal to $j$ , then

P ((X_{n + 1}, τ_{n + 1}) = (j, t) | (X_{n} = j, τ_{n})) = 0 if τ_{n} \neq t - 1, t \geq 2,

and otherwise if $X_{n} = i \neq j$ then

P ((X_{n + 1}, τ_{n + 1}) = (j, t) | (X_{n} = i, τ_{n})) = 0 unless t = 1.

Then $(X_{n}, τ_{n})$ is specified by the following probabilities

Here the $ρ_{i} (k)$ can be considered as the margin probability that the chain stays in the same state for the next time step. With this set up, we call $X_{n}$ a semi-Markov process. Figure 2.5 shows the allowable state transitions for the process $Z_{n}$ .

Figure 2.5: Shows allowable transitions out of a state for the augmented chain

Z_{n}

Example 2.8.

Note semi-Markov processes are an extension of Markov processes. Given a time homogenous Markov process $X_{n}$ specified by transition probabilities $P (X_{n + 1} = j | X_{n} = i) = A_{i, j}$ , then we can define $τ_{n}$ as the consecutive time $X_{n}$ has spent in the current observed state since it entered this state. We then get that

P (X_{n + 1} = i, τ_{n} = t + 1 | X_{n} = i, τ_{n} = t) = P (X_{n + 1} = i | X_{n} = i) = A_{i, i} = ρ_{i} (t)

and for $j \neq i$ we get that

P (X_{n + 1} = j, τ_{n} = 1 | X_{n} = i, τ_{n} = t) = P (X_{n + 1} = j | X_{n} = i) = A_{i, j} .

Note that neither of these probabilities depend on the time $t$ .

We have suggestively used $ρ_{i}$ in the notation above. This is due to the $ρ_{i}$ satisfying the conditions in creftypecap 2.1 for each $i$ . We have that

	$P ($	$X_{n} enters state i then stays at state i for % exactly k consecutive steps)$
		$= \sum j_{1} \neq i, j_{2} \neq i P (X_{n + k} = j_{1} \neq i, X_{n} = X_{n + 1} = \dots = X_{n + k - 1} = i, X_{n - 1} = j_{2})$
		$= \sum j_{1} \neq i, j_{2} \neq i P (X_{n + k} = j \| X_{n} = \dots = X_{n + k - 1} = i, X_{n - 1} = j_{2}) \cdot$
		$P (X_{n} = \dots = X_{n + k - 1} = i, X_{n - 1} = j_{2})$
		$= (1 - ρ_{i} (k)) \sum j_{2} \neq i P (X_{n + k - 1} = i \| X_{n} = \dots = X_{n + k - 2} = i, X_{n - 1} = j_{2}) \cdot$
		$P (X_{n} = \dots = X_{n + k - 2} = i, X_{n - 1} = j_{2})$
		$= (1 - ρ_{i} (k)) ρ_{i} (k - 1) \sum j_{2} \neq i P (X_{n} = \dots = X_{n + k - 2} = i, X_{n - 1} = j_{2})$
		$⋮$
		$= (1 - ρ_{i} (k)) ρ_{i} (k - 1) ρ_{i} (k - 2) \dots ρ_{i} (1) \sum j_{2} \neq i P (X_{n} = i, X_{n - 1} = j_{2}),$

Then we have that

P (

X_{n + k + 1} \neq i, X_{n + k} = X_{n + k - 1} = \dots = X_{n + 1} = i | X_{n} = i, X_{n - 1} \neq i)

Analysis of sojourn time distributions for semi-Markov models

Generalised empirical likelihood-based kernel density estimation

On a General Class of Discrete Bivariate Distributions

Learning Factor Graphs in Polynomial Time & Sample Complexity

Maximum likelihood estimation of parameters of spherical particle size distributions from profile size measurements and its application for small samples

Simulation study of estimating between-study variance and overall effect in meta-analyses of mean difference

Strong structure recovery for partially observed discrete Markov random fields on graphs

Discriminating between and within (semi)continuous classes of both Tweedie and geometric Tweedie models