I Introduction
Many emerging applications depend on the timely delivery of status updates from a number of sources to a central monitor over a wireless network. Examples include sensor and actuator data for networked control systems, collecting information for IoT applications, mobility data in vehicular networks, and realtime surveillance and monitoring.
Age of Information (AoI) is a metric that captures timeliness of received information at a destination [14, 18]. Unlike packet delay, AoI measures the lag in obtaining information at a destination node, and is therefore suited for applications involving gathering or dissemination of time sensitive updates. Age of information, at a destination, is defined as the time that has elapsed since the last received information update was generated at the source. AoI, upon reception of a new update packet, drops to the time elapsed since generation of the packet, and grows linearly otherwise. Over the past few years, there has been a rapidly growing body of work on analyzing AoI for queuing systems [14, 18, 2, 7, 8, 13], and using AoI as a metric for scheduling policies in networks [12, 11, 10, 19, 20, 9, 6, 4].
The problem of minimizing age of information in singlehop networks was first considered in [12] and [11]
. In these works, the authors considered a base station collecting timesensitive information from a number of sources over a wireless broadcast network, where only one source can send an update at any given time. They looked at weighted linear combinations of AoI of all sources as the metric to be optimized. This prompted the design of low complexity scheduling policies that provably minimize weighted sum AoI at the base station, up to a constant multiplicative factor. These results crucially depend on the fact that for linear AoI, one can find a stationary randomized policy that is factor2 optimal. As we will see later, this observation does not hold for general functions of AoI. In fact, stationary randomized policies can be arbitrarily worse than simple heuristic policies.
Scheduling problems with weighted linear combinations of age have also been considered with throughput constraints in [10] and with general interference constraints in [19]. AoIbased scheduling with stochastic arrivals was considered in [6], where a Whittle Index policy was shown to have good performance.
On the other hand, nonlinear cost functions of age were introduced as a natural extension to the AoI metric in [18] for characterizing how the level of dissatisfaction depends on data staleness in a more general manner. Nonlinear functions of age of information were also discussed in the context of queuing systems in [16] and [17]. These papers develop the notion of value of information and use nonlinear cost of update delays, which correspond to nonlinear age cost functions.
Nonlinear functions of age have also been discussed in the context of networked control systems in [3],[15] and [1]. In [3], the authors discuss a real time networked control system and show that the cost function is characterized as a nondecreasing, possibly nonlinear, function of AoI. In [15]
, the authors formulated the state estimation problem for an LTI system, where the state of a discretetime LTI system can be observed in any timeslot by paying a fixed transmission cost. The problem of minimizing the timeaverage of the sum of the estimation error and transmission cost reduces to minimizing a nondecreasing agecost function for a single source with a fixed transmission cost.
In this work, we consider a setting similar to the one in [12] and [11]. We look at wireless broadcast network with sources generating realtime updates that need to be sent to a monitoring station. In any timeslot, only one source can attempt a transmission to the base station. Instead of weighted sum AoI, we are interested in minimizing the timeaverage of general nondecreasing cost functions of AoI, summed over all sources. Examples of such functions include , , , etc. See Fig.1 for examples. We develop a restless mutliarmed bandit formulation for the problem and use a Whittle Index based approach to find low complexity scheduling policies that have good performance.
Closer to our work, scheduling to minimize functions of age has also been considered in [20] and [9]. In [20], the authors deals with minimizing symmetric functions of age of sources over multiple orthogonal unreliable channels and show that simple greedy schemes are asymptotically optimal. In [9], the authors formulate the general functions of age problem with reliable channels and develop a high complexity algorithm that achieves minimum age. They also derive a key structural property of the optimal policy in this setting  the optimal policy is always periodic. However, their approach does not extend to the setting with unreliable channels. In this work, we consider unreliable channels and also build upon results from [9] and [6] to derive stronger structural properties for optimal policies. These properties hint at why the performance of the heuristic Whittle index policy may be close to optimal.
The remainder of the paper is organized as follows. In Section II, we describe the general system model. In Section III, we describe the equivalent restless multiarmed bandit formulation and discuss why we use the Whittle Index approach to solve the problem. In Section IV, we discuss the functions of age problem with reliable channels, develop the Whittle Index solution for this setting, and also prove key structural properties that an optimal policy must satisfy. In Section V, we find the Whittle Index policy for the functions of age problem with unreliable channels.
Ii Model
Consider a singlehop wireless network with active sources generating realtime status updates that need to be sent to a base station. We consider a slotted system in which each source takes a single timeslot to transmit an update to the base station. Due to interference, only one of the sources can transmit in any given timeslot.
For every source , the age of information at the base station measures the time elapsed since it received a fresh information update from the source. We assume active sources, i.e. in any timeslot, sources can generate fresh updates at will. Let be the source activated in timeslot and
be a Bernoulli random variable with parameter
that denotes channel reliability between the source and the base station. Then, we have(1) 
In this work, we consider general cost functions of age as our metric of interest. For each source , let denote a positive nondecreasing cost function.
Let be a scheduling scheme that decides which sources to schedule in every timeslot. The age process depends on and the channel processes. Then, the expected average cost of age for source is given by
(2) 
where is age process for the source under policy .
Our goal is find a schedule that minimizes the sum of average costs of age of sources. Let denote the set of causal scheduling policies, then we want to solve the following optimization problem
(3) 
where is minimum average cost and is an optimal scheduling policy.
Iii RMAB Formulation
In this section, we show that scheduling to minimize such a metric can be reformulated as a restless multiarmed bandit (RMAB).
Consider a restless multiarmed bandit problem with arms. The state space for every arm is the set of positive integers . The state evolution of the arm depends on whether it is currently active or not. Let the state of arm at time be denoted by . If arm is active in timeslot then the state evolution is given by
(4) 
If the arm is not active in timeslot , then the state evolution is given by
(5) 
For every arm , there is a cost function which maps the states of the arm to their associated costs. Thus, the cost of a state is given by , where
is a vector of ages and
is the age of the source. Given that only one arm can be activated in any timeslot, the goal of the RMAB framework is to find a scheduling policy that minimizes the total time average cost of running this system.This establishes the equivalence between the functions of age problem discussed earlier and a corresponding restless multiarmed bandit. Observe that the “restless” part of our construction cannot be dropped, since the states of the arms do not freeze when they are not active and there is no way to reformulate our problem as just a multiarmed bandit problem. If that were the case, we could have found an optimal policy by solving for the Gittins index [5]. However, finding optimal policies for restless bandits is much harder. The usual approach is to find the Whittle Index policy which provides good performance under certain conditions, namely indexability of the RMAB problem.
In [12] and [11], the authors develop three methods to solve the minimum age scheduling problem. First, they look at stationary randomized policies, where a source
is scheduled at random with a fixed probability
. They find a stationary randomized policy that is factor2 optimal for weighted sum AoI. However, this result does not hold for general functions: even the best stationary randomized policies in our setting can lead to an unbounded overall cost, despite there being very simple policies that have bounded cost. We demonstrate this with a simple example.Consider two identical sources with cost functions given by and reliable channels, i.e . Any stationary randomized policy schedules at least one of the sources with probability less than or equal to . For this source, the average cost is lower bounded by since with probability at least , it does not get to transmit and its age increases by 1 in every timeslot. Observe that this lower bound goes to and hence the average cost also goes to for all stationary randomized policies. On the other hand, a simple roundrobin scheme that schedules the two sources in alternating timeslots guarantees bounded cost for both sensors. Thus, stationary randomized policies can be infinitely worse than the optimal policy for the functions of age problem.
The second method developed for agebased scheduling in [12, 11] uses a MaxWeight approach. The authors design a quadratic Lyapunov function for the weighted sum of linear functions of AoI and find the maxweight policy  the policy that maximizes the amount of negative drift in the Lyapunov function in every timeslot. Performance guarantees for the maxweight policy crucially rely on the fact that there exists a stationary randomized policy that is factor2 optimal for linear functions of age. Since this is not the case for general functions of age, we cannot develop a similar MaxWeight policy for the general functions of age problem.
This finally leaves us with the third method  using a Whittle Index based approach. In the following two sections, we use the RMAB formulation to establish indexability for the functions of age problem and derive a Whittle Index policy. We also show that for the case with 2 sources and reliable channels, the Whittle index policy is exactly optimal. This is a novel result since the optimality of Whittle Index policies is typically shown either only asymptotically, or in symmetric settings for finite systems. On the other hand, our optimality result holds for two asymmetric sources.
Iv Reliable Channels
We first look at the problem with reliable channels between the sources and the base station. This leads to simpler analysis and a better understanding of the problem. Consider the setup described in Section I with channel reliability , for all and . In other words, the probability of success .
In Section III, we showed that the functions of age minimization problem is equivalent to a restless multiarmed bandit problem. Next, we use a Whittle Index based approach to try and solve the problem.
The first step in the Whittle Index approach is to formulate the decoupled problem, where we consider a single arm in isolation with a fixed charge required to activate the arm.
Our goal is to find a scheduling policy that minimizes the timeaverage cost of running this system. Assuming that the cost function is nonnegative and nondecreasing, we solve the decoupled problem using dynamic programming. The case when the activation charge is set to zero is trivial. The optimal policy is to always activate the arm. So, we consider to be strictly positive. The single source decoupled problem has also been solved in a slightly different setting in [15].
Theorem 1
The optimal policy for the decoupled problem is a stationary threshold policy. Let satisfy
(6) 
Then, the optimal policy is to activate the arm at timeslot if and to let it rest otherwise. If no such exists, the optimal policy is to never activate the arm.
Proof:
See Appendix A.
Theorem 1 establishes that the optimal policy for the decoupled problem has a threshold structure. We now want to show that the indexability property also holds for the decoupled problem. The indexability property states that as the activation charge increases from to , the set of states for which it is optimal to activate the arm decreases monotonically from the entire set to the empty set .
Theorem 2
The indexability property holds for the decoupled problem.
Proof:
See Appendix B.
The Whittle index approach states that if the decoupled problem satisfies the indexability property, we can formulate a heuristic index policy called the Whittle Index Policy that has good performance.
Observe that using (6), is the minimum value of the activation charge that makes both actions equally desirable in state . This gives us the expression for the Whittle index.
Let represent the index function for the decoupled problem. By the monotonicity of , it is easy to see that the functions are also monotonically nondecreasing. This is because since is nondecreasing. Using these functions, we define the Whittle Index Policy.
Consider the cost functions to be weighted linear functions of AoI, i.e let , with positive weights . This is the setting considered in [12] and [11]. The Whittle Index for source is then given by . This is the same as the Whittle index found in [12], where the authors showed that the Whittle policy is optimal for symmetric settings when all the weights are equal. We also establish that for , the Whittle index policy is optimal even for asymmetric settings.
Theorem 3
For the functions of age problem with reliable channels and two sources, the Whittle index policy is exactly optimal.
Proof:
See Appendix F.
Next, we discuss some general properties that an optimal policy satisfies. These properties help us establish the optimality of the Whittle index policy for .
Iva Properties of an Optimal Policy
For the functions of age problem, a policy is stationary if it depends only on the current values of age. A cyclic policy is one that repeats a finite sequence of actions in a fixed order. We define the space of policies that are stationary and periodic.

Stationary Cyclic Policies
A stationary cyclic policy is a stationary policy that cycles through a finite subset of points in the state space, repeating a fixed sequence of actions in a particular order.
In [9], the authors show that for reliable channels there exists an optimal policy that is stationary, cyclic and can be found by solving the minimum average cost cycle problem over a large graph.
We look at this cyclic policy and analyze its properties. If there are multiple such cycles, we consider a cycle with the shortest length. We denote the length of the cycle by and age vectors on the cycle to be . Let the corresponding scheduling decisions be . This implies that for state , taking action leads to the state , where the subscripts cycle back to after .
We establish an important structural property that such an optimal policy must satisfy, which we call the strongswitchtype property. We call the policies that satisfy this property strongswitchtype policies.
In words, the strongswitchtype property implies that if a policy decides to activate arm for a state vector , then for a state vector with a higher age for the source and lower ages for all the other sources, it still decides to activate source . Note that our definition of strongswitchtype policies is a stronger version of the switchtype policies introduced in [6].
Theorem 4
For the functions of age problem with reliable channels, no stateaction pairs that are a part of the shortest length optimal cyclic policy can violate the strongswitchtype property.
Proof:
See Appendix C.
We can prove this result for general values of . However, to extend the strongswitchtype property over the entire statespace, we consider systems with up to three sources.
Theorem 5
There exists an optimal stationary policy for the functions of age problem with reliable channels and up to three sources that has the strongswitchtype property over the entire statespace.
Proof:
We have already established that points on the minimum average cost cycle satisfy the strongswitchtype property. In Appendix D, we extend this policy over the entire state space while maintaining the strongswitch property to obtain a well defined stationary policy.
While we prove this result for up to three source and reliable channels, we believe that the strongswitchtype property is a natural property that an optimal policy must have in general, due to monotonicity of cost functions.
We now define the space of policies that can be found as a result of the Whittle Index based approach.
Observe that if are the same as in the above definition, then we get back the Whittle Index Policy. Also, note that an index policy always satisfies the strongswitchtype property by definition. This is because the index functions are monotonically nondecreasing. We now show that index policies are in fact the same as strongswitchtype policies.
Theorem 6
For the functions of age problem, every policy that is strongswitchtype is also an index policy.
Proof:
The proof is based on induction on the number of sources. We assume that every strongswitchtype policy can be represented as an index policy for systems with sources. Using this fact, we show that strongswitchtype policies can also be represented as index policies for systems with sources. We also show that the two types of policies are equivalent for the single source decoupled problem, thus completing the proof. The details are in Appendix E.
An important point to notice is that while we use the reliability of channels in the proof of Theorem 5, we do not use any such condition for the proof of Theorem 6. Thus, strongswitchtype policies are equivalent to index policies regardless of channel connectivity.
Corollary 1
For the functions of age problem with reliable channels and up to three source, there exists a stationary optimal policy that is an index policy.
In other words, there exists an optimal policy that looks like the Whittle Index policy in that the arm to be activated has the maximum value among monotone index functions that take as arguments only the states of individual arms. This hints at why the performance of Whittle Index policies may be close to optimal.
Observe that the Whittle Index policy would be optimal in general if we could show that it achieves a cost that is the minimum cost among the space of index policies and that the strongswitchtype property holds for some optimal policy. We show that this is indeed the case for . However, we later provide an example that shows that the Whittle policy is not optimal, but only close to optimal, for .
We leave the question of whether the Whittle index policy is at most a constant factor away from optimal in general to future work. We believe that the structural properties introduced here provide a recipe to proving constant factor optimality of the Whittle index policy, even for general bandit problems with similar underlying structure.
V Unreliable Channels
We now consider independent Bernoulli channels between every source and the base station, with probability of success for source . We derive a Whittle index in this setting and establish indexability of the RMAB problem by enforcing a bounded cost condition on the functions .
An important fact to notice is that monotonicity in itself is not sufficient to ensure that the system has finite average cost even for , in the case of unreliable channels. Consider a single source case where and the probability of success . If the source attempts a transmission in every timeslot, the expected average cost satisfies
(9) 
since with probability , the transmission fails and age increases by 1 in every timeslot. However, observe that the summation on the left goes to infinity and thus the expected average cost goes to infinity. This happens despite the source attempting a transmission in every timeslot. To prevent such a situation from happening we enforce the following bounded cost condition on the age cost functions in addition to monotonicity
(10) 
It can be shown that this condition ensures that the single source case has bounded cost. We define the decoupled problem in this case as follows
As before, our goal is to find a scheduling policy that minimizes the timeaverage cost of running this system.
Theorem 7
The optimal policy for the decoupled problem is a stationary threshold policy. Let satisfy
(12) 
Then, the optimal policy is to activate the arm at timeslot if and to let it rest otherwise. If no such exists, the optimal policy is to never activate the arm.
Proof:
See Appendix G.
Observe that taking the limit as in Theorem 7, we get back the threshold policy for reliable channels derived in Theorem 1. We now establish indexability and derive the functional form of the Whittle Index.
Theorem 8
The indexability property holds for the decoupled problem. Denote by the Whittle index in state . Given indexability, is the infimum charge that that makes makes both decisions (activate, not activate) equally desirable in state . The expression for is given by
(13) 
Proof:
See Appendix H.
Again, observe that taking the limit as , we get back the Whittle Index derived in Section IV. Further, if we assume that the cost functions are weighted linear functions of AoI, i.e. where all the weights are positive, then the index functions for the Whittle policy are given by . This corresponds to the index policy developed in [12], where the authors showed that for symmetric settings when all the weights and channels probabilities are equal, the Whittle index policy is optimal.
Vi Simulations
First, we compare the optimal policy, found using dynamic programming, with the Whittle index policy for two sources. We consider six different settings in total  3 sets of functions, each with reliable and unreliable channels.
For settings and , the cost functions are chosen to be and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically and . For settings and , the cost functions are chosen to be and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically and . For settings and , the cost functions are chosen to be and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically and . Simulation results are presented in Table I.
Setting  Optimal Cost  Whittle Index Cost 

(reliable)  21.95  21.95 
(unreliable)  36.12  36.28 
(reliable)  8.48  8.48 
(unreliable)  23.16  23.37 
(reliable)  5.69  5.69 
(unreliable)  21.54  21.54 
We find the optimal cost for each setting using finite horizon dynamic programming over a horizon of 500 timeslots. For reliable channels, we find the cost of the Whittle index policy by simply implementing it once over 500 timeslots. For unreliable channels, we estimate the expected Whittle index cost by averaging the performance of the Whittle policy over 500 independent runs.
Observe that the Whittle index policy is exactly optimal when the channels are reliable, as expected from our theoretical results. The expected cost for the Whittle index policy is very close to the optimal cost for unreliable channels as well. Also, for the same set of functions, having unreliable channels increases the cost compared to reliable channels, as is expected.
Next, we compare the optimal policy with the Whittle index policy for more than two sources. Simulation results are presented in Table II.
For settings and , we consider 3 sources. The cost functions are chosen to be , and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically , and .
For settings and , we consider 4 sources. The cost functions are chosen to be , , and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically , , and .
No. of Sources  Setting  Optimal Cost  Whittle Index Cost 

3  (reliable)  44.23  44.23 
(unreliable)  161.19  161.39  
4  (reliable)  73.36  73.36 
(unreliable)  129.02  130.94  
4  (reliable)  87.66  88.27 
(unreliable)  158.35  159.81 
For settings and , we consider 4 sources. The cost functions are chosen to be , , and . In , we consider reliable channels, i.e. . In , we consider unreliable channels, specifically , , and .
We observe that the cost of the Whittle index policy is the same as that obtained using dynamic programming for settings and . However, for setting , we observe a small gap in performance between the two policies, thus giving us an example that shows that the Whittle index policy need not be optimal, in general. We also verify that the optimal policy found using dynamic programming follows a cyclic pattern that satisfies the strongswitchtype property and is distinct from the Whittle index policy. This is also in line with our discussion on structural properties.
We also note that computing the optimal policy using dynamic programming becomes progressively harder in terms of space and time complexity for larger values of , as the statespace to be considered grows exponentially with . The Whittle index policy, on the other hand, is very easy to compute and implement with only a linear increase in space and time complexity with the number of sources. Also, as is evident from simulations, the performance of the Whittle policy is close to optimal in every setting considered, thus making it a very good low complexity heuristic.
Vii Conclusion
In this work, we presented the problem of minimizing functions of age of information over a wireless broadcast network. We used a restless multiarmed bandit approach to establish indexability of the problem and found the Whittle index policy. For the case with two sources and reliable channels, we were able to show that the Whittle index policy is exactly optimal. We also established structural properties of an optimal policy, for the case with reliable channels. These properties hint at why the performance of the Whittle index policy is close to optimal in general.
A possible direction of future work is to try and prove constant factor optimality of the Whittle index policy in general, using the structural properties developed in this work. Another interesting extension would be to consider sources with stochastic arrivals instead of active sources.
References
 [1] (2019) Ageofinformation vs. valueofinformation scheduling for cellular networked control systems. arXiv preprint arXiv:1903.05356. Cited by: §I.
 [2] (2019) Minimizing the age of information through queues. IEEE Transactions on Information Theory. Cited by: §I.
 [3] (2019) Performance characterization using aoi in a singleloop networked control system. arXiv preprint arXiv:1901.06694. Cited by: §I.
 [4] (2018) On the age of information in multisource multihop wireless status update networks. In 2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), pp. 1–5. Cited by: §I.
 [5] (1989) Multiarmed bandit allocation indices. Vol. 25, Wiley Online Library. Cited by: §III.
 [6] (2017) Scheduling algorithms for minimizing age of information in wireless broadcast networks with random arrivals: the nobuffer case. arXiv preprint arXiv:1712.07419. Cited by: §I, §I, §I, §IVA.
 [7] (2015) Optimizing ageofinformation in a multiclass queueing system. In 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1681–1685. Cited by: §I.
 [8] (2018) A general formula for the stationary distribution of the age of information and its application to singleserver queues. arXiv preprint arXiv:1804.06139. Cited by: §I.
 [9] (2018) Ageofinformation aware scheduling. In SPCOM, Cited by: §I, §I, §IVA.
 [10] (2018) Scheduling algorithms for optimizing age of information in wireless networks with throughput constraints. IEEE/ACM Transactions on Networking. Cited by: §I, §I.
 [11] (2018) Scheduling policies for minimizing age of information in broadcast wireless networks. IEEE/ACM Transactions on Networking (TON) 26 (6), pp. 2637–2650. Cited by: §I, §I, §I, §III, §III, §IV.
 [12] (2016) Minimizing the age of information in broadcast wireless networks. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 844–851. Cited by: §I, §I, §I, §III, §III, §IV, §V.
 [13] (2018) On the age of information with packet deadlines. IEEE Transactions on Information Theory 64 (9), pp. 6419–6428. Cited by: §I.
 [14] (2012) Realtime status: how often should one update?. In Proc. INFOCOM, pp. 2731–2735. Cited by: §I.
 [15] (2019) AoIpenalty minimization for networked control systems with packet loss. IEEE Infocom AoI Workshop. Cited by: §I, §IV.
 [16] (2017) Age and value of information: nonlinear age case. In 2017 IEEE International Symposium on Information Theory (ISIT), pp. 326–330. Cited by: §I.
 [17] (2018) The cost of delay in status updates and their value: nonlinear ageing. arXiv preprint arXiv:1812.09320. Cited by: §I.
 [18] (2017Nov.) Update or wait: how to keep your data fresh. IEEE Trans. Inf. Theory 63 (11), pp. 7492–7508. Cited by: §I, §I.
 [19] (2018) Optimizing information freshness in wireless networks under general interference constraints. In Proceedings of the Eighteenth ACM International Symposium on Mobile Ad Hoc Networking and Computing, pp. 61–70. Cited by: §I, §I.
 [20] (2017) Age of information in multisource systems. In GLOBECOM 20172017 IEEE Global Communications Conference, pp. 1–6. Cited by: §I, §I.
Comments
There are no comments yet.