I Introduction
In the problem of quickest change detection, the objective is to detect a change in the distribution of a sequence of random variables with the minimum possible delay, subject to a constraint on the rate of false alarms
[1, 2, 3]. Optimal or asymptotically optimal algorithms for quickest change detection are available in the literature. The results can be divided broadly into two categories: results for independent and identically distributed (i.i.d.) processes with algorithms that can be computed recursively and using finite memory and enjoy strong optimality properties [4, 5], and results for noni.i.d. data with algorithms that cannot be necessarily computed recursively or using finite memory but are asymptotically optimal [6, 7, 8, 9].In this paper, we develop theory and algorithms for detecting changes in stochastic processes that have periodically varying statistical characteristics. In this noni.i.d. setting, we will show that the optimal algorithms can be computed recursively and using finite memory. The motivation for this problem comes from the following anomaly detection problems in cyberphysical systems and biology where such periodic behavior is observed.

[leftmargin=*]

Traffic monitoring: In [11] and [12]
, we reported results on multimodal traffic data we collected from NYC around a 5K run; during, before and after the run. We collected CCTV images, Twitter and Instagram data. We extracted counts of persons and vehicles appearing in CCTV images over time using a deep neural networkbased object detector. We observed that in the absence of the event (in the normal regime), the counts have a periodic statistical behavior (over a day or a week) with increased intensity every day during morning and evening rush hours.

Power grid monitoring: The power usage by end users have a periodic pattern with low usage at nighttime and high usage at daytime [10].

Neural spike patterns: In braincomputer interface studies where single neural spike data is collected, the spike firing pattern can exhibit statistically periodic behavior in the absence of any external stimuli; see, for example, [13].

ECG: Several biological signals including the ECG have a periodic behavior [14].
The problem of anomaly detection in the abovementioned applications can be seen as a problem of detecting changes in deviation from periodic statistical behavior.
In this paper, we develop a Bayesian theory for anomaly detection in problems where the statistical characteristics are periodic. We introduce a class of stochastic processes called independent and periodically identically distributed (i.p.i.d) processes that can be used to model such periodic statistical behavior. We then develop algorithms for quickest detection of changes in i.p.i.d. processes and prove their optimality with respect to the Bayesian criterion of Shiryaev [15]. In the Shiryaev formulation, the objective is to detect a change in the distribution of a stochastic process to minimize the average detection delay, subject to a constraint on the probability of false alarm. In the Shiryaev problem, each time we take an observation, we pay a penalty for delay if the change has already occurred. If an alarm is raised and the change has not yet occurred, we pay a penalty for a false alarm. In this paper, we also study a more general or modified Shiryaev formulation where the penalties on the delay and the false alarm are dependent on time. The latter problem is relevant for detecting changes in noni.i.d. processes. The definition of i.p.i.d. processes and precise problem formulations are given in Section II.
When the observations are i.i.d. before and after the change, optimal algorithms are obtained in the Bayesian setting of Shiryaev using optimal stopping theory or dynamic programming [5], [17], [22]. However, since the processes under investigation here are not i.i.d., but i.p.i.d., the traditional optimal control theory cannot be applied. We show in this paper that the change detection problems for i.p.i.d. processes we study in this paper can be mapped to a problem of optimal control of a Markov decision process (MDP) with a nonstationary but periodic transition and cost structure. As a result, in Section III, we first develop an optimal control theory for periodic MDPs. With a view towards optimal stopping, we develop the optimal control theory for the total cost problem with finite control spaces [17]. For stationary problems, the optimal policy can be obtained using the framework of dynamic programming and can be shown to be Markovian and stationary [17]. A general recipe for solving nonstationary (including periodic) problems can also be found in [17]; e.g., see pg. 256. In fact, it is suggested in [17] that the optimal policy for periodic problems is periodic in nature; see also [18]. In this paper, we explicitly derive optimal policies for periodic problems using a more direct approach and prove that the optimal policies are indeed periodic. We obtain a fixed point equation satisfied by the optimal cost function. We also obtain a value iteration algorithm for computing the optimal cost using which the periodic optimal policy can be computed. The optimal control theory developed here for periodic MDPs should be of independent interest for other control applications as well.
In Section IV, we apply the optimal control theory developed in Section III to the change detection problems. We show that the optimal change detection algorithm is periodic. The change detection statistic, as in the classical i.i.d. setting, is the a posteriori probability that the change has already occurred. But, unlike the i.i.d. setting where a singlethreshold policy is strictly optimal, the stopping threshold for the i.p.i.d. problem varies with time. In fact, we show that the sequence of thresholds is periodic. We provide examples where a periodic and nonstationary policy is strictly better than a single threshold stationary policy.
In Section V, however, we show that if the constraint on the probability of false alarm is small, then we can, in fact, use a fixed timeinvariant threshold. Specifically, we show that a singlethreshold algorithm is asymptotically optimal for the classical Shiryaev formulation by showing that the proposed algorithm achieves a universal lower bound on the delay of any change detection procedure [7]. We will show that while the exact optimality result and periodic MDP theory is valid only for geometric priors, the asymptotic optimality result is valid for a large class of distributions on the change point.
Ii Mathematical Model
We begin by first introducing a class of stochastic processes that can be used to model data showing periodic statistical behavior.
Definition 1
A stochastic process is called independent and periodically identically distributed (i.p.i.d) if

[leftmargin=*]

The random variables are independent.

If has density , for , then there is a positive integer such that the sequence of densities is periodic with period :
We say that the process is i.p.i.d. with the law . Note that the law of an i.p.i.d. process is completely characterized by the finitedimensional product distribution involving . We assume that in the normal regime, the data can be modeled as an i.p.i.d. process. At some point in time, due to an event, the distribution of the i.p.i.d. process deviates from . Our objective in this paper is to develop algorithms to observe in real time and detect changes in the distribution as quickly as possible, subject to a constraint on the rate of false alarms. If the period , an i.p.i.d. process reduces to an i.i.d. process. Optimal algorithms for change detection in i.i.d. processes has been extensively developed in the literature [1, 2, 3, 4, 5, 6, 7, 8, 9].
We now define a change point model. Consider another periodic sequence of densities such that
We assume that at some point in time , called the change point in the following, the law of the i.p.i.d. process is governed not by the densities , but by the new set of densities :
(1) 
The densities need not be all different from the set of densities , but we assume that there exists at least an such that they are different:
(2) 
Iia Classical Shiryaev Formulation
Let be a stopping time for the process , i.e., a positive integervalued random variable such that the event belongs to the algebra generated by . In other words, whether or not is completely determined by the first observations. We declare that a change has occurred at the stopping time . To find the best stopping rule to detect the change in distribution, we need a performance criterion. Towards this end, we model the change point as a random variable with a prior :
For each , we use to denote the law of the observation process when the change occurs at , and use to denote the corresponding expectation. Using this notation, we define the average probability measure
To capture a penalty on the false alarms, we use the probability of false alarm defined as
To penalize the detection delay, we use the average detection delay given by
or its conditional version
where .
The optimization problem we are interested in solving is
(3) 
where
and is a given constraint on the probability of false alarm. In the above problem, we can also use the conditional version of the delay .
When the change point is a geometric random variable, the classical approach to solving problem (3) is to solve a relaxed version using dynamic programming. Specifically, let
Then
The relaxed Bayesian optimization problem is
(4) 
where is a penalty on the cost of false alarms. The above optimization problem can be stated as a problem in partially observable MDPs (POMDPs); see Section IIB, and also [1, 22], and [11]. Specifically, define and
(5) 
Then, it can be shown that problem (4) is equivalent to solving
(6) 
If the period and the processes are i.i.d., then the problem in (6) can be solved using classical belief state MDPs [1, 22, 11, 17]. However, in our case, the belief updates are not stationary.
Lemma II.1
Proof:
The proof is provided in the appendix.
Note that the likelihood ratios are a function of the time index . Thus, the belief updates are nonstationary. However, because the processes are i.p.i.d. in nature and there are only finitely many densities and , the belief updates have a periodic structure that repeats after time slots.
The optimal stopping problem (6) cannot be solved using classical optimal stopping theory or dynamic programming [5], [17], [22]. This is because in these theories it is assumed that the Markov process to be controlled is homogeneous. The Markov process to be controlled in (6) is not homogeneous. However, as shown in Lemma V.1, the transition structure is periodic. Motivated by this observation, in Section III, we develop the optimal stopping theory or optimal control theory for periodic MDPs. In Section IV, we will apply the results obtained for periodic MDPs to the periodic optimal stopping problem (6).
IiB Modified Shiryaev Formulation
In this section, we formulate a more general optimal stopping problem than stated in (6). In the problem in (6), the delay penalty at all times is unit and the false alarm penalty is units. Since the processes under study here are not i.i.d., we formulate a POMDP where the delay and false alarm penalties are a function of time. Since we are investigating i.p.i.d. processes, we assume that the delay and false alarm penalties are periodic as well. The precise problem formulation is given below.

States: Let be a sequence of states with values
. The state process is a finitestate Markov chain taking values
(8) The state is a special absorbing state introduced for mathematical convenience in a stopping time POMDP [22].

Control: The control sequence for the POMDP is the process taking values and is binary valued:
(9) The control is used to continue the observation process and is used to stop it. At the time of stopping, an alarm is raised indicating that a change in the distribution of the observations has occurred.

Observations: The distribution of the observations depends on the state and if the control is to continue: for ,
(10) with the understanding that the observation process is i.p.i.d. with law before the change, and i.p.i.d. with law after the change. No observation is collected at time and the distribution of the observations when the state equals is irrelevant to the problem.

Transition Structure: The Markov chain evolves according to a transition structure that depends on the control process . Let be the transition matrix for the Markov chain, given the control is . Then, we have
(11) where
(12) and
(13) The initial distribution for the Markov chain is
(14) Thus, the Markov chain starts at . As long as the control , which means to continue, the states evolve according to the transition probability matrix . The values selected for elements of matrix ensure that the absorption time to the state
is inevitable and is a geometrically distributed random variable with parameter
. Before the absorption, the distributions of the observations are i.p.i.d. with law . After the state is absorbed in the distributions of the observations changes to that of an i.p.i.d. process with law . 
Cost: The cost associated with state and control is defined for as
(15) Thus, is the penalty on the false alarm at time and is the penalty on the delay at time . We assume that the penalty sequences are periodic with period : for any ,
(16) 
Policy: Let
be the information at time . Also define a policy
to be a sequence of mappings such that .
We want to find a control policy to optimize the long term cost
(17) 
Using arguments similar to those used to obtain (6), it can be shown that the probability sequence
is a sufficient statistics also for the problem in (17). Consequently, solving (17) is equivalent to solving the following MDP problem:
(18) 
If and , , then the problem in (18) reduces to the problem in (6). If the period and the processes are i.i.d., then the problem in (18) can also be solved using classical MDP theory [1, 22, 11, 17]. However, for , the observation process is i.p.i.d. and the classical theory cannot be applied. The optimal control theory developed in Section III below for periodic MDP can and will be used to solve (18) and hence its special case (6).
Iii Optimal Control of Periodic MDPs
In this section, we develop an optimal control theory for MDPs with periodic cost and transition structure. We have stochastic processes , , and taking values is spaces as follows:
(19) 
The process is an MDP generated according to the transition structure
(20) 
Here is the control process and is the disturbance process. We assume that given and , the distribution of the disturbance is independent of the past disturbances . We use to denote this conditional distribution. Thus, the state and disturbance spaces are realvalued and the control spaces are finite. The results in the paper are, in fact, valid for more general spaces. To accommodate more general spaces, the proof techniques may have to be slightly modified [17, 20].
The main assumption in our model is that the transition functions and the conditional distributions are periodic: there is a positive integer such that for ,
(21) 
The objective is to choose the control process so as to minimize the cost
(22) 
where is the discount factor which is allowed to be equal to with a view towards problems in optimal stopping. The cost functions are assumed to be nonnegative and periodic with the same period : for ,
(23) 
The assumption of nonnegativity of the cost functions (which is the same as the assumption in [17]) ensures that all infinite summations are welldefined by monotone convergence theorem [21].
In order to minimize this longterm additive cost (22), we search over Markov control policies of type
such that
As done in [17], it can be argued that restricting the search over Markov policies is sufficient. For a policy , we define the costtogo function starting with the state as
(24) 
where the expectation is with respect to the disturbances. We are interested in solving the following problem: for ,
(25) 
In this section, we show that the optimal policy for the problem in (25) is periodic with period , i.e., it is of the type
We also provide an explicit way to compute this optimal periodic policy.
For , and , define the operator
(26) 
where the expectation is defined with respect to the conditional distribution . We also define the operator for a Markov map and :
(27) 
Finally, define the fold operator
(28) 
which is the successive application of the operators defined in (26). Our first result is the following.
Theorem III.1
The optimal cost function in (25) satisfies the following fixedpoint equation: for any ,
(29) 
Proof:
The proof is provided in the appendix.
Next, we show that if the optimal cost function is known, then the optimal policy can be obtained and shown to be periodic.
Theorem III.2
The optimal policy is periodic. Specifically, let be the optimal cost function and let be such that for , and ,
(30) 
Then, the optimal policy is given by
(31) 
Proof:
The proof is provided in the appendix.
An optimal policy always exists in our case because we assume the control spaces to be finite.
The previous result is useful only when we have an algorithm to compute the optimal cost function . This is facilitated by the theorem below. Define
(32) 
where we have used to denote the operator in (28) applied times to the allzero function .
Theorem III.3
Proof:
The proof is provided in the appendix.
Iv Detecting Changes in I.P.I.D. Processes
In this section, we solve the problem in (18), and hence the problem in (6), using Theorems III.1, III.2, and III.3. Specifically, we can state the following result. Let
(34) 
where .
Theorem IV.1
The optimal cost function in (34) satisfies
(35) 
where for ,
(36) 
In the above equation,
(37) 
where , and
(38) 
In the above theorem, the densities are assumed to be with respect to the Lebesgue measure. The expressions can be modified to allow for counting measures (summations) or more general measures.
If and , then we are reduced to the classical formulation of Shiryaev. It is well known that the optimal policy for the classical case is a singlethreshold policy in which the change is declared the first time the statistic or probability is above a predefined threshold. The threshold depends on the choice of the false alarm penalty .
For , it is interesting to ask if the Shiryaev singlethreshold stopping rule is still the optimal policy. In the subsections below, we will show that the optimal policy, in fact, utilizes multiple thresholds, where the number of distinct thresholds can be up to . For reference below, we define the Shiryaev stopping rule here: for ,
(39) 
In the rest of the paper, we call this stopping rule or policy the periodicShiryaev stopping rule or algorithm to emphasize that the recursion for is periodic.
Iva Example: Change Detection in I.P.I.D. Processes For Different Values of
In this section, we show two examples in the i.p.i.d. setting where the optimal policy is not stationary. In fact, it is periodic with periodic thresholds.
In Fig. 1, we report results for . Specifically, consider a change detection problem where the period of the i.p.i.d. processes is and the pre and postchange i.p.i.d. densities are Gaussian:
(40) 
The parameters for change point, false alarm and delay are as follows:
(41) 
The optimal cost function was obtained using value iteration (33) with the operators as defined in (36). The cost functions at each iteration
with being the allzero function on , are plotted in Fig. 0(a). We used a point resolution or discretization of the interval in the value iteration. In Fig 0(b), we have plotted the norm distance as a function of the iteration index . In Fig. 0(c), we have plotted the stopping cost and the continue cost appearing in (36) for , where we use stages to refer to the distinct time slots. It can be inferred from Fig. 0(c) that the optimal policy has alternating thresholds: and . For stopping, is compared with the threshold
during the odd time slots and compared with the threshold
during the even time slots. The optimal cost achieved by this alternating threshold policy is , as can be seen in Fig. 0(a). In Fig. 0(d), we have plotted the total cost achieved by the periodicShiryaev algorithm (39) for different values of constant threshold . These costs were obtained through MonteCarlo simulations using sample paths. The best achievable cost of the singlethreshold periodicShiryaev algorithm is establishing that the optimal cost of cannot be achieved using a singlethreshold policy.Next, we consider a change detection problem where the period of the i.p.i.d. processes is and the pre and postchange i.p.i.d. densities are given by
(42) 
The parameters for change point, false alarm and delay are as follows:
(43) 
Results for this case are reported in Fig. 2. Similar to results for in Fig. 1, we see here also that the optimal cost is again (Fig. 1(a)). The Fig. 1(c) also shows that there are four thresholds in this case, one for each of the four stages in a cycle or period of length . Again, the best cost achievable by the periodicShiryaev algorithm equals and is strictly larger than the cost of the optimal policy.
IvB Change Detection in I.I.D. Data
In this section, we show examples where the periodicShiryaev algorithm is not strictly optimal even for i.i.d. data, as long as and we use the modified Shiryaev formulation discussed in Section IIB. Specifically, we consider the change point problem with parameters
(44) 
and
(45) 
In Table I, we have reported the comparison between the optimal policy and the periodicShiryaev algorithm for the above parameters for different choices of the postchange parameter . All points are obtained using MonteCarlo simulations with sample paths.
Cost of  Cost of  

PeriodicShiryaev  Optimal Policy  
IvC Performance Comparison For Different Mean Choices
In this section, we report comparison between the performance of the optimal policy and the periodicShiryaev algorithm for different choices of mean parameters for :
(46) 
The parameters for change point, false alarm and delay are as follows.
(47) 
The results are collected in Table II. The values in the table suggests that the superiority of the optimal policy over the periodicShiryaev is maintained for different values of the mean parameters and .
Cost of  Cost of  

SingleThreshold Policy  Optimal Policy  
IvD Performance Comparison For Different Choices of Delay and False Alarm Penalties
In the previous sections, we have shown examples where the periodicShiryaev is strictly suboptimal. In this section, we show that the performance gap depends on the choice of the delay and false alarm penalties . In Table III, we show the performance comparison for
(48) 
The performance gap reduces if the false alarm penalties are kept different but the delay penalties are set to the same value. The performance gap, in fact, vanishes as the false alarm and delay penalties are equal and the problem reduced to the classical Shiryaev case. In the next section, we provide a theoretical basis for this observation by showing that the periodicShiryaev algorithm is, in fact, asymptotically optimal for the classical Shiryaev formulation, as the false alarm rate goes to zero. While we could not find an example where the two algorithms have different performance for the classical Shiryaev formulation, we conjecture that such an example exists and necessarily involve high values of probability of false alarm.
Cost of  Cost of  

SingleThreshold  Optimal  
Policy  Policy  
V Asymptotic Optimality of SingleThreshold Policies
In this section, we show that the periodicShiryaev algorithm is asymptotically optimal for the classical Shiryaev formulation (3) as the probability of false alarm goes to zero.
For easy reference, we recall the definition of the periodicShiryaev algorithm here. Define
(49) 
and stop the first time this probability is above a threshold, i.e., use the stopping rule
(50) 
While the statistic is always welldefined in a Bayesian setting, recall that in general for a noni.i.d. model, the Shiryaev statistic cannot be computed recursively using a finite amount of memory [7, 3]. Another convenient way to compute the statistic is to compute its transformation defined as
(51) 
The statistic can also be computed recursively.
Lemma V.1
In the i.p.i.d. setting, the statistic (51) can be computed recursively as
(52) 
with . Further, if the prior is then the above recursion simplifies to
(53) 
Proof:
The proof is provided in the appendix.
Va Universal Performance Bounds For Change Detection in I.P.I.D. Processes
In this section, we obtain a universal lower bound on the performance of any stopping rule for detecting changes in an i.p.i.d. process.
Let there exist such that
(54) 
If , then
Thus, .
Further, let
(55) 
where
is the KullbackLeibler divergence between the densities
and . We assume thatand
Theorem V.1
Proof:
The proof is provided in the appendix.
VB Optimality of PeriodicShiryaev Algorithm
We now show that the periodicShiryaev algorithm (50) is asymptotically optimal for problem (3) as the false alarm constraint . We will establish the optimality by showing that the periodicShiryaev algorithm achieves the lower bound specified in Theorem V.1.
Let
and define for
(57) 
We assume for simplicity that and are densities with respect to Lebesgue measure on the real line. The results below can be easily extended to densities with respect to more general measures (including the counting measure).
Proof:
The proof is provided in the appendix.
Thus, the periodicShiryaev algorithm achieves the lower bound and is asymptotically optimal. The arguments provided in the proofs of the theorems above can be extended to also establish asymptotic optimality with respect to higher order moments of the detection delay.
VC Numerical Results
In Fig. 3, we have plotted the average detection delay (ADD) as a function of the magnitude of the logarithm of the probability of false alarm (PFA) for the following set of parameters:
(61) 
The values for simulations were obtained using sample paths. The values for the analysis curve in the figure were obtained by setting the probability of false alarm using the threshold and using the delay expression
. As can be observed from the figure, the analytical expression provides an accurate estimate of the delay.
Vi Conclusion
We established the optimality of periodic policies for optimal control of MDPs in which the cost structure and the transition probabilities are periodic, all with the same period. We then applied this result to solve an optimal stopping problem using the framework of partially observable MDPs. The optimal stopping problem we studied is the problem of detecting changes in i.p.i.d. processes. The exact optimality theory suggests that the optimal policy has multiple thresholds (alternating, in case the period ). This structural behavior, or its effect, is absent in the low false alarm regime, where we showed that using a single, fixed, threshold is asymptotically optimal, as the probability of false alarm goes to zero. A Bayesian analysis often provides important insights into a problem. The insight we obtain from this paper is that when analyzing a nonBayesian or minimax version of the problem studied in this paper, one can conjecture that a singlethreshold policy like cumulative sum is not strictly optimal for all values of the false alarm rate [4].
Proof:
For we have
where . This can be formally proved using the rigorous definition of conditional expectations using subsigmaalgebras [21]. This implies
Using this we can obtain an expression for :
Comments
There are no comments yet.