Recency-weighted Markovian inference

1 Introduction

Markovian inference methods allow for on-line processing of data with time and memory requirements that are constant in the total number of observations received at each time step. Each observation $y_{t}$ corresponds to a latent variable $z_{t}$

, which over time forms a Markov chain, giving rise to the latent state space model

[1]. At every time step

t

a new model

z_{t}

is inferred using the Bayes theorem based on the previous state of the model

z_{t - 1}

and current observation

y_{t}

p (z_{t} | y_{t}) \propto p (y_{t} | z_{t}) \cdot p (z_{t} | z_{t - 1}),

(1)

where the probability distribution (or density)

p (z_{t} | z_{t - 1})

describes a state transition function responsible for the evolution of the latent state

z

. However, the fixed computational cost of such inference depends on assuming conditional independence between current and past

m

states of the latent variable so that:

p (z_{t} | z_{t - 1}, \dots, z_{t - m}) = p (z_{t} | z_{t - 1}) .

(2)

Contrastingly, in many human learning phenomena the model of the environment seems to be inferred across multiple past states and weighed with respect to their recency (see [2] for a review). If we want to weigh the effect of each past $m$ states separately on the current state $z_{t}$ then the Markov property (Eq 2) does not hold and we need to find another way of defining the evolution of the latent state. To solve this problem we use the mixture state transition function approach for high-order Markov chains [3, 4]

. This approach represents the conditional probability distribution

p (z_{t} | z_{t - 1}, \dots, z_{t - m})

as a mixture of past

m

states:

p (z_{t} | z_{t - 1}, \dots, z_{t - m}) = θ_{1} p (z_{t} | z_{t - 1}) + \dots + θ_{m} p (z_{t} | z_{t - m}) = M \sum m = 1 θ_{m} p (z_{t} | z_{t - m}),

(3)

where $θ$ is a mixing coefficient so that

0 < θ_{m} \leq 1, and M \sum m = 1 θ_{m} = 1.

Mixing coefficients make the dependence of the future on the past explicit by quantifying the decay in dependence as the future moves farther from the past [5]. If we use an equal mixing coefficient for all previous states, $θ = 1 / m$ , then all past states, independent of their lag, contribute equally to the prediction of the current state. Instead, we might want the more recent states represented proportional to their recency. Hence, we can assume that the contribution of past states, reflected by the mixing coefficient $θ$ , declines over $m$ time steps as given by some decay function $ϕ$ and the rate of decay parameter $β$ . Here we choose a decay function $ϕ$ , so that:

θ_{m} = ϕ (m, β) = α θ_{0} (1 - β)^{m},

(4)

where $β$ is the rate of decrease ( $0 \leq β < 1$ ) and $α$ normalising constant. Substituting $θ_{m}$ into equation (Eq 3) gives:

p (z_{t} | z_{t - 1}, \dots, z_{t - m}) = M \sum m = 1 α θ_{0} (1 - β)^{m} p (z_{t} | z_{t - m}) .

(5)

As a result we have a decaying time window into the past $m$ states defined by the rate parameter $β$ of the decreasing mixing coefficient $θ$ . Fig 1 illustrates the relationship between the $β$ and $θ$ parameters: the bigger the $β$ the faster the utility of past states decreases and greater the contributions of most recent states to the mixture distribution (Eq 5). As $β$ approaches 1, the mixture begins to resemble $p (z_{t} | z_{t - 1})$ and approximate a first-order Markov chain:

lim β \to 1, p (z_{t} | z_{t - 1}, \dots, z_{t - m}) = p (z_{t} | z_{t - 1}) .

Fig 1: Values of the mixing coefficient $θ$ over past 5 states ( $z_{t - 1}, \dots, z_{t - 5}$ ) based on different $β$ values. $θ_{t}$ represents the proportion of a past state $z_{t}$ in the mixture distribution (Eq 5).

Conversely, as $β$ approaches 0, the mixing coefficient $θ$ does not decay across time steps and all past states contribute equally to the mixture. Intuitively, $β$ could be interpreted as the bias towards more recent states.

However, such an approach poses two fundamental problems. First, to evaluate Eq 5 we still need to explicitly store all past $m$ states and the processing time of the algorithm would increase with $m$

. Second, such sequentially estimated mixtures are themselves mixtures of previous mixture distributions, making the estimation analytically intractable. Therefore, such a model would not allow for processing of data with time and memory requirements that are constant at each time step.

Next, we show how a practical solution to both of these problems can be obtained by using sampling methods.

2 State estimation with mixture sampling

In order to limit the computational cost of performing inference at every time step we represent the distribution of latent variable $z$ with a fixed number of samples $L$ . As a result the proportion of samples assigned to a particular component of the mixture distribution (representing a past state) is determined by the mixing coefficient $θ$ :

p (z_{t} | z_{t - 1}, \dots, z_{t - m}) = M \sum m = 1 θ_{m} p (z_{t} | z_{t - m}) ≃ M \sum m = 1 θ_{m} L \sum l = 1 {p (z_{t} | z_{t - m})}^{(l)}

(6)

where $L$ is the total number of samples, $θ_{m} L$ a subset of samples allocated to $p (z_{t} | z_{t - m})$ , ${p (z_{t} | z_{t - m})}$ a set of samples from that distribution, and $θ L_{m}$ is rounded to the nearest integer.

For example, if we use $L = 100$ samples to represent a mixture of $m = 5$ components with a fixed $θ = 1 / 5$ , then every component would be assigned 20 samples. With a decaying mixing coefficient (Eq 4) the number of samples assigned to any $z_{t - m}$ decreases with $m$ , but the number of samples assigned to any particular $z_{t - m}$ will remain constant (because of the constant rate parameter $β$ in Eq 5; also see Fig 1). For example, if $t > m$ , $L = 100$ , $β = 0.5$ , and $θ_{0} = 1$ , then $p (z_{t} | z_{t - 1})$ will be represented with 50 samples, $p (z_{t} | z_{t - 2})$ with 25 samples, and so forth.

The property of constant number of samples for every $m$ -th component of the mixture at any time is important since it greatly simplifies the approximation of the mixture distribution (Eq 5). If at every time-step $t$ we choose a fixed proportion $β$ of samples from the existing mixture distribution and reassign those samples to represent $p (z_{t} | z_{t - 1})$ , then after $m$ steps we end up with samples allocated across mixture components as given by (Eq 5). To make this explicit, consider the following evolution of mixture (Eq 5) symbolically in a table:

$t$	Mixture representation (Eq 3)	Sample representation
1	$θ_{1} p (z_{1} \| z_{0}) + θ_{2} p (z_{0})$	${z_{1}}^{(L)} = {p (z_{1} \| z_{0})}^{(L β)} + {z_{0}}^{(L (1 - β))}$
2	$θ_{1} p (z_{2} \| z_{1}) + θ_{2} p (z_{1} \| z_{0}) + θ_{3} p (z_{0})$	${z_{2}}^{(L)} = {p (z_{2} \| z_{1})}^{(L β)} + {z_{1}}^{(L (1 - β))}$
3	$θ_{1} p (z_{3} \| z_{2}) + θ_{2} p (z_{2} \| z_{1}) + θ_{3} p (z_{1} \| z_{0}) + θ_{4} p (z_{0})$	${z_{3}}^{(L)} = {p (z_{3} \| z_{2})}^{(L β)} + {z_{2}}^{(L (1 - β))}$
	…	…
t	$θ_{1} p (z_{t} \| z_{t - 1}) + θ_{2} p (z_{t - 1} \| z_{t - 2}) + \dots + θ_{m} p (z_{0})$	${z_{t}}^{(L)} = {p (z_{t} \| z_{t - 1})}^{(L β)} + {z_{t - 1}}^{(L (1 - β))}$

Table 1: The evolution of the latent state

z

via mixture state transition function. The left column shows the evolution of the mixture representation, the right column shows the same as a sample representation. Sample representation: at every time-step

t

we choose a fixed proportion

β

of samples and reassign those samples to represent the most recent mixture component. Here

z_{0}

is the initial distribution, {

z_{t}}^{(L)}

is a set of

L

samples representing the mixture distribution at time

t

, and

{p (z)}^{(l)}

denotes a set of

l

samples from

p (z)

It follows that at every time step $t$ a mixture distribution of past $m$ states can be approximated by sampling from just $p (z_{t} | z_{t - 1})$ and the previous state of the mixture ${z_{t - 1}}$ .

1.25

Algorithm 1 State estimation with mixture sampling

t = 0

{z_{0}}^{(L)} \sim p (z)

▹

Initialise

L

by sampling from some distribution

p (z)

repeat

t = t+1

{z_{t}}^{(L)} = {p (z_{t} | z_{t - 1})}^{(L β)} + {z_{t - 1}}^{(L (1 - β))}

▹

Update

{z_{t}}

by taking

L β

samples from

p (z_{t} | z_{t - 1})

and

L (1 - β)

samples from

{z_{t - 1}}

until t=T

After $m$ steps these $L$ samples represent a mixture of past $m$ states. The number of samples assigned to a particular mixture component decreases with its distance from the present state at the rate of $β$ . This algorithm has constant time and memory requirements and uses only two parameters: re-sampling coefficient $β$ and the number of total samples $L$ .

3 Posterior estimation

So far we have just dealt with high-order chains of latent variables $z$ . However, our goal is to incrementally infer the model of the environment $z_{t}$ every time a new observation $y_{t}$

arrives. In other words, we are interested in inferring the posterior probability of the latent variable:

p (z_{t} | y_{t}) \propto p (y_{t} | z_{t}) \cdot p (z_{t} | z_{t - 1}, \dots, z_{t - m}) .

We assume that the evolution of the latent variable $z$ is predicted by the mixture state transition function $h ()$ and system noise $q$ :

z_{t} = h (z_{t - 1}, \dots, z_{t - m}) + q_{t}

(7)

Having previously derived an approximation for $p (z_{t} | z_{t - 1}, \dots, z_{t - m})$ sampling from the posterior distribution becomes straightforward. Here we use a sequential Monte Carlo approach called particle filtering [6, 7], where the estimation of the the posterior distribution is based on generating proposals of the latent state $p (z_{t} | z_{t - 1}, \dots, z_{t - m})$ with Algorithm 1 and weighing them with the likelihood function $p (y_{t} | z_{t})$ . The process can be summarised as follows.

At every time-step $t$ a set of $L$ particles ${z_{t}}^{(L)}$ represents the mixture of past states (Eq 6). In other words, each individual particle $z_{t}^{l}$ is a prediction of the latent state at the current time step. Next we calculate the likelihood of the current observation $y_{t}$ under each particle. This likelihood serves as the importance weight $w_{t}^{l}$ . We then normalise the importance weights across particles. The set of particles ${z_{t}}^{(L)}$ together with the corresponding weights ${w_{t}}^{(L)}$ represents the posterior distribution $p (z_{t} | y_{t})$ . Next, we create a new predictive distribution for the next state $p (z_{t + 1} | z_{t})$ represented by ${z_{t + 1}}^{(L)}$ by replacing randomly $L β$ particles in ${z_{t}}^{(L)}$ with samples from the posterior $p (z_{t} | y_{t})$ . Finally we add system noise $q_{t}$ to the particles. This process is described fully in Algorithm 2 below.

This algorithm is based on a widely used method of importance sampling: if we change the re-sampling step so that the predictive prior ${z_{t + 1}}^{(L)}$ is re-sampled solely from the posterior distribution $(z_{t} | y_{t})$ by setting $β = 1$ , then the algorithm becomes equivalent to the standard importance resampling implementation of a 1st order Markov chain of latent states [8], where the system transition function is identity and system noise is zero (in other words, the posterior distribution of the current state $p (z_{t} | y_{t})$ serves as the predictive prior for the next state $p (z_{t + 1} | z_{t})$ , Fig 2, column 1).

1.7

Algorithm 2 Importance sampling

t = 0

{z_{0}}^{(L)} \sim f (z)

▹

Initialise

L

particles by sampling from some distribution

f (z)

{z_{1}}^{(L)} = {z_{0}}^{(L)}

▹

Use

z_{0}

as a prediction for

z_{1}

repeat

t = t+1

{w_{t}}^{(L)} = p (y_{t} | {z_{t}}^{(L)})

▹

Assign weights to samples

{w_{t}}^{(L)} = {w_{t}}^{(L)} / \sum {w_{t}}^{(L)}

▹

Normalise weights

{z_{t}^{(i)}}_{1 \leq i \leq N}^{(L β)}

▹

Randomly select

β L

particle indices

j_{i} \in {1, \dots, N}

{z_{t}^{(i)}}_{1 \leq i \leq N}^{(L β)} = {p (z_{t} | y_{t})}^{(L β)}

▹

Assign selected particles to samples from the posterior

{z_{t + 1}}^{(L)} = {p (z_{t} | y_{t})}^{(L β)} + {z_{t}}^{(L (1 - β))}

▹

New predictive latent distribution is a mixture of current posterior and previous mixture

{z_{t + 1}}^{(L)} = {z_{t + 1}}^{(L)} + q

▹

Add system noise to the particles

until t=T

Fig 2: Components of the state mixture distribution for different values of the $β$ parameter (horisontal axis). When $β = 1$ the predictive prior for the next state consists entirely of samples from the last state $z_{t - 1}$ . As $β$ approaches zero the mixture distribution will contain roughly equal amount of samples from all of the past states.

4 Conclusions

In many human incremental learning situations the model of the environment is inferred across multiple past states and weighed with respect to their recency [2]. Here we presented a probabilistic model of incremental learning where the current latent state given some observation $p (z_{t} | y_{t})$ depends on recency-weighted past states. Despite the fact that our model approximates high-order dependencies we have described a posterior distribution estimation algorithm which is itself 1st order Markovian and can hence be processed with fixed time and memory costs.

References

Bishop [2006] Christopher M. Bishop. 13 Sequential Data. In Pattern Recognition and Machine Learning, pages 605–652. Cambridge University Press, 2006.
Kiyonaga et al. [2017] Anastasia Kiyonaga, Jason M. Scimeca, Daniel P. Bliss, and David Whitney. Serial Dependence across Perception, Attention, and Memory. Trends in Cognitive Sciences, 21(7):493–497, 2017. ISSN 1879307X. doi: 10.1016/j.tics.2017.04.011. URL http://dx.doi.org/10.1016/j.tics.2017.04.011.
Berchtold and Raftery [2002] A Berchtold and A Raftery. The Mixture Transition Distribution Model for High-Order Markov Chains and Non-Gaussian Time Series. Statistical Science, 17(3):328–356, 2002. ISSN 0883-4237. doi: 10.1214/ss/1042727943.
Saul and Jordan [1999] Lawrence K. Saul and Michael I. Jordan.
Mixed memory Markov models: Decomposing complex stochastic processes as mixtures of simpler ones.
Machine Learning, 37(1):75–87, 1999. ISSN 08856125. doi: 10.1023/A:1007649326333.
McDonald et al. [2011] Daniel J. McDonald, Cosma Rohilla Shalizi, and Mark Schervish. Estimating beta-mixing coefficients. In International Conference on Artificial Intelligence and Statistics, pages 516–524, mar 2011. URL http://arxiv.org/abs/1103.0941.
Cappe et al. [2007] O Cappe, S Godsill, and E Moulines. An Overview of Existing Methods and Recent Advances in Sequential Monte Carlo. Proceedings of the IEEE, 95(5):899–924, may 2007. ISSN 0018-9219. doi: 10.1109/JPROC.2007.893250.
Doucet et al. [2001] A Doucet, N de Freitas, and N Gordon. An Introduction to Sequential Monte Carlo methods. In A Doucet, N de Freitas, and N Gordon, editors, Sequential Monte Carlo Methods in Practice. Springer, NY, 2001.
Rubin [1987] Donald B. Rubin.
The Calculation of Posterior Distributions by Data Augmentation: Comment: A Noniterative Sampling/Importance Resampling Alternative to the Data Augmentation Algorithm for Creating a Few Imputations When Fractions of Missing Information Are Modest: The SIR.
Journal of the American Statistical Association, 82(398):543, jun 1987. ISSN 01621459. doi: 10.2307/2289460. URL http://www.jstor.org/stable/2289460?origin=crossref.

$t$	Mixture representation (Eq 3)	Sample representation
1	$θ_{1} p (z_{1} \| z_{0}) + θ_{2} p (z_{0})$	${z_{1}}^{(L)} = {p (z_{1} \| z_{0})}^{(L β)} + {z_{0}}^{(L (1 - β))}$
2	$θ_{1} p (z_{2} \| z_{1}) + θ_{2} p (z_{1} \| z_{0}) + θ_{3} p (z_{0})$	${z_{2}}^{(L)} = {p (z_{2} \| z_{1})}^{(L β)} + {z_{1}}^{(L (1 - β))}$
3	$θ_{1} p (z_{3} \| z_{2}) + θ_{2} p (z_{2} \| z_{1}) + θ_{3} p (z_{1} \| z_{0}) + θ_{4} p (z_{0})$	${z_{3}}^{(L)} = {p (z_{3} \| z_{2})}^{(L β)} + {z_{2}}^{(L (1 - β))}$
	…	…
t	$θ_{1} p (z_{t} \| z_{t - 1}) + θ_{2} p (z_{t - 1} \| z_{t - 2}) + \dots + θ_{m} p (z_{0})$	${z_{t}}^{(L)} = {p (z_{t} \| z_{t - 1})}^{(L β)} + {z_{t - 1}}^{(L (1 - β))}$

Recency-weighted Markovian inference

Authors

Variational Tracking and Prediction with Generative Disentangled State-Space Models

Combined parameter and state inference with automatically calibrated ABC

Estimation and selection for high-order Markov chains with Bayesian mixture transition distribution models

Analysis of Markov Jump Processes under Terminal Constraints

Explainable Genetic Inheritance Pattern Prediction

Bayesian recurrent state space model for rs-fMRI

Abstraction-Guided Truncations for Stationary Distributions of Markov Population Models

1 Introduction

2 State estimation with mixture sampling

3 Posterior estimation

4 Conclusions

References

Recency-weighted Markovian inference

Authors

Variational Tracking and Prediction with Generative Disentangled State-Space Models

Combined parameter and state inference with automatically calibrated ABC

Estimation and selection for high-order Markov chains with Bayesian mixture transition distribution models

Analysis of Markov Jump Processes under Terminal Constraints

Explainable Genetic Inheritance Pattern Prediction

Bayesian recurrent state space model for rs-fMRI

Abstraction-Guided Truncations for Stationary Distributions of Markov Population Models

This week in AI

1 Introduction

2 State estimation with mixture sampling

3 Posterior estimation

4 Conclusions

References