K eeping information fresh is an essential requirement for a variety of control and communication tasks. Stale feedback information in networked control systems may reduce the gain or phase margin, which may, in turn, push the system towards the verge of instability ,. Real-time status updates are necessary for a plethora of communication tasks including effective traffic monitoring , online gaming , intrusion detection , environment sensing using IoT devices  etc. While designing routing and scheduling policies for maximizing the throughput region is well-understood , , designing optimal policies for maximizing the information freshness is currently an active area of research .
With the advent of the 5G technology, it is becoming increasingly common for the Base Stations (BS) to serve the following two different types of UEs at the same time - Type-I: Delay-constrained UEs (e.g., UEs with URLLC type of traffic , such as control information updates for autonomous vehicles), and Type-II: Throughput-constrained UEs (e.g., UEs with eMBB type of traffic, such as HD multimedia streaming). Moreover, in Network-Control applications where the delay-constrained UEs often perform a global task jointly (e.g., by sensing different parts of a sizeable cyber-physical system), it is critical to uniformly maximize the information freshness across all Type-I UEs to avoid information bottlenecks. In this paper, we study the problem of optimal joint scheduling of Type-I and Type-II UEs over wireless erasure channels.
We characterize the freshness of information at a UE by a metric called the Age of Information (AoI) , . In our context, the AoI for a UE at a time indicates the time elapsed since the UE received a new packet from the BS prior to time . The larger the value of AoI for a UE at a time, the more outdated the UE is at that time. In this short paper, we consider two related problems on minimizing the AoI - Problem (1): In the presence of only Type-I UEs, our goal is to design a scheduling policy which minimizes the long-term peak-AoI uniformly across all UEs, and Problem (2): When the Type-II UEs are to be scheduled simultaneously along with the Type-I UEs by the same BS, we consider the problem of minimizing the long-term peak-AoI, subject to throughput constraint to the Type-II UEs.
In a recent paper  Bedewy et al., consider the problem of optimal scheduling of status updates over an error-free delay channel. They showed that, in that setting, the greedy Max-Age First scheduling policy is an optimal policy for both peak-age and total age metrics. In the paper 
, He et al. consider the problem of link scheduling to transmit a fixed number of packets over a common interference-constrained channel such that the overall age is minimized. They proved the problem to be NP-hard and proposed an Integer Linear Program and a fast heuristic. The authors continued studying the previous problem in for minimizing the peak-age, and obtained similar results. See the monograph  for a detailed survey of the recent literature on Age of Information.
Closer to our work is the paper , which studies a similar problem with single-hop wireless erasure channel. However, contrary to this paper, the objective of  is to design a policy to minimize the long-term average AoI. Using Lyapunov-drift based methodology, the paper  designs an approximately optimal policy for this problem. Designing an optimal policy in this setting still remains elusive. As we argued before, with distributed sensing applications, where all sensors need to stay updated uniformly, a more suitable objective is to minimize the long-term peak-AoI across all users. In this paper, we design an exactly optimal policy for the problem (1) using MDP techniques. We also show that the proposed policy achieves the optimal large deviation exponent among all scheduling policies. Moreover, inspired by the analysis for the problem (1), we propose a heuristic policy for the problem (2), where we incorporate an additional throughput constraint for the eMBB UEs. Operating performances of these proposed policies have been compared extensively with other well-known scheduling policies through numerical simulations.
The rest of the paper is organized as follows. Section II outlines the system model. In Section III, we consider the problem of minimizing the long-term peak-AoI across all UEs when only Type-I UEs are present. In Section IV, we consider the problem of optimal joint scheduling in the presence of both Type-I and Type-II UEs. Section V presents numerical simulation results comparing the proposed policies with other well-known scheduling policies, such as Proportional Fair and Randomized Policies. Finally, we conclude the paper in Section VI.
Ii System Model
We consider the downlink UE scheduling problem where a Base Station (BS) serves wireless users, each with full-buffer traffic, meaning, each user is infinitely backlogged. The channel from the BS to the th
UE is modelled by a binary erasure channel with erasure probability, where . Time is slotted, and the BS can transmit to only one user per slot. If the BS transmits a packet to the th UE at slot , the packet is either successfully decoded by the UE with probability , or, the packet is permanently lost with probability , independently of everything else. Refer to Figure 1 for a schematic diagram of the model. The objective is to design suitable downlink UE scheduling policy optimizing a given metric. In this paper, we consider two related problems - (1) designing peak-AoI-optimal scheduler without any throughput constraints (Section III) and (2) Designing peak-AoI-optimal scheduler with throughput constraint for a UE (Section IV).
Iii Minimizing the Peak-Age-of-Information
In this section, we consider the problem of minimizing the long-term peak-AoI metric, which denotes the maximum Age of Information among all receivers associated with a base-station in a cellular network. With hard deadline constraints for each user in the case of URLLC traffic in 5G, minimizing the peak-AoI metric is more practically meaningful than the minimizing the average AoI , .
At a given slot , define to be the peak instantaneous Age of Information among all users. Our objective is to design a scheduling policy , which minimizes the time-averaged expected peak-AoI. More formally, we consider the following stochastic control problem :
subject to the constraint that at most one user may be scheduled at any slot. Define a greedy scheduling policy MA (Max-Age) which, at any given slot , schedules the user having the highest instantaneous age. More formally, at a slot , a user is scheduled which maximizes the metric (ties are broken arbitrarily). We establish the following theorem for MA:
Theorem 1 (Optimal Policy).
The greedy policy MA is an optimal policy for the problem (1). Moreover, the optimal long-term peak-AoI is given by .
Theorem 1 states that the greedy policy MA
is optimal for the peak-AoI metric. Interestingly, the optimal policy is independent of the channel statistics (the probability vector). This should be contrasted with the approximately optimal policy MW for minimizing the average-AoI metric proposed in .
We prove this theorem by proposing a closed-form candidate solution of the Bellman’s equation of the associated average-cost MDP and then verifying that the candidate solution indeed satisfies the Bellman’s equation.
The stochastic control problem under investigation is an instance of a countable-state average-cost MDP with a finite action space. The state of the system at a slot given by the current AoI vector of all users, i.e., . The per-stage cost at time is , which is unbounded, in general. Finally, the finite action space corresponds to the index of the user scheduled at a given slot.
Let the optimal cost for the problem be denoted by and the differential cost-to-go from the state be denoted by . Then, following the standard theory of average cost countable state MDP (Proposition 4.6.1 of ), we consider the following Bellman Eqn.
where the vector denotes the dimensional vector of all coordinates excepting the th coordinate and is a all-one vector.
The Bellman Equation (III) may be explained as follows. Suppose that the current AoI state is given by . If the scheduler schedules a transmission to the th user, the transmission is successful with probability and is unsuccessful with probability . If the transmission is successful, the AoI of all users, excepting the th user, is incremented by , and the AoI of the th UE is reduced to . This explains the first term. On the other hand, if the transmission to the th UE is unsuccessful, the AoI of all users are incremented by . This explains the second term within the bracket. Finally, the term denotes the current stage cost.
Solution to the Bellman Equation (Iii)
We verify that the following constitutes a solution to the Bellman Equation (III):
The optimality result now follows from . ∎ The following interesting features of the optimal scheduling policy MA should be noted:
The proof of optimality of the MA policy gives an explicit expression for the associated cost-to-go function and the optimal cost . This is one of the rare cases where the associated Bellman Equation of an MDP has an analytic solution.
Iii-a Large Deviation Rate Optimality and Stability
Although Theorem 1 establishes that the MA scheduling policy is optimal in terms of minimizing the long-term expected peak-AoI, for mission-critical URLLC applications, we need to additionally ensure that the peak-AoI metric stays within a bounded limit with high probability. The following Proposition 2 shows that the peak-Age process has an exponentially light tail under the action of the MA policy. This ensures high-probability delay guarantees to URLLC traffic having a strict latency requirement.
Under the action of the MA policy, there exists a constant such that, for any fixed time and any ,
Our objective in the rest of this subsection is to show that the large deviation bound (5) is asymptotically optimal in the sense that no other scheduling policy has lower probability exceeding a given sufficiently large AoI-threshold . Towards this end, in the next proposition, we establish a fundamental performance bound of the peak-AoI tail probability under the action of any scheduling policy.
Under the action of any arbitrary scheduling policy , at any slot and for all , we have
Let . Now,
where the inequality (a) follows from the fact that consecutive erasures just prior to time for UE (which takes place with probability ) ensures that the age of UE at time is at least . ∎
The MA policy achieves the optimal large-deviation exponent for the max-age metric and the value of the optimal exponent is given by
Hence, taking limit as , we obtain
Since the bound (8) is valid for any , for any fixed , we can let to obtain
Finally, since the bound (9) is valid for any , we can now let to obtain
Interestingly enough, although the time-average optimal performance depends on the statistical parameters of all UEs (Theorem 1), the optimal large-deviation exponent depends only on the parameter of the worst UE.
We conclude this Section with the following stability result of the Age-process under the MA policy.
The Markov Chain
The Markov Chainis Positive Recurrent under the action of the MA policy.
See Appendix VIII-B for the proof.
Iv Minimizing the Peak-Age-of-Information with Throughput Constraints
In this Section, we consider a generalization of the above system model, where, in addition to maintaining a small peak-AoI, there is also a Type-II UE (denoted by UE), which is interested in maximizing its throughput. This problem can be motivated by considering a residential subscriber who is running one high-throughput application (with eMBB-type traffic), such as, downloading an HD movie, while also using several smart home automation IoTs, which have URLLC-type traffic, and hence, require low-latency. In a small-cell residential network, all of these devices are served by a single BS typically located within the house .
Define a sequence of random variablessuch that if the UE did not successfully receive a packet at the end of slot and otherwise. Let be a non-negative tuning parameter. We are interested in finding a scheduling policy which solves the following problem :
The single-stage cost described above may be understood as follows: the first term denotes the usual maximum AoI across all UEs as in the previous Section. The second term imposes a penalty of if the UE1 does not receive a packet at the current slot. By suitably controlling the value of , a tradeoff between the peak-AoI and achievable throughput to UE1 (serving the eMBB traffic) may be obtained .
Similar to the problem of Eqn. (1), the problem is also an instance of an infinite-state average-cost MDP with an additional action-dependent additive per-stage cost term (). Arguing as before, and introducing an additional cost term arising due to the throughput constraint, the Bellman Equation for this problem may be written down as follows:
Here denotes the expected cost when the UE, which receives eMBB traffic, does not successfully receive a packet at slot . The above Bellman Equation (12) may be explained along similar line as the equation (III).
This yields the MATP (Max-Age with Throughput) scheduling policy, which minimizes the RHS of the Bellman Equation (12):
The MATP policy strikes a balance between minimizing the peak-AoI (through the first term ) while also ensuring sufficient throughput to the eMBB user (through the second term ). As is increased, it gradually dominates the AoI term, which, in turn, facilitates scheduling the eMBB user.
Analysis of Matp
Furthermore, since , we have
Hence, by taking , we see that under the action of the MATP policy, the sup-norm of the difference between the RHS and LHS of the Bellman Equation (12) is bounded by the constant . In other words, upon denoting the RHS of the Bellman operator of (12) by (see  for this operator notation), we have
It should be noted that unlike the MA policy, the MATP policy takes into account the channel statistics (the value of ) for the eMBB user. It is oblivious to the channel statistics of other URLLC UEs, however.
Following a similar line of argument as in the proof of Theorem 5, we can establish the result below.
The Markov Chain is Positive Recurrent under the action of the MATP policy.
V Numerical Simulation
In this section, we simulate the following five scheduling policies for the downlink wireless system described in Section II - 1) Max-Age Policy (MA) 2) Randomized Policy (RP) 3) Max-Weight Policy (MW) 4) Proportional Fair (PF) 5) Max-Age Policy with Throughput Constraints (MATP). The policy MA is described in Section III. The second policy RP chooses a UE randomly in each time slot. The third policy MW was proposed in  for approximately minimizing the long-term average-AoI metric. In every time slot , the policy MW chooses the UE which maximizes the metric amongst all UEs. The fourth policy is the well-known Proportional Fair policy ,  which at every slot selects the UE maximizing the metric . Here is the exponentially-smoothed average rate, which is updated at every time slot as: where is the instantaneous throughput to the UE at slot . The fifth policy (MATP) is described in Section IV of this paper.
We simulate a downlink wireless network with nodes, each with a binary erasure channel. The probability of successful transmission for the th channel
is sampled i.i.d. from a uniform distribution in. Each simulation is run for slots, and an average of simulations is taken for the plots. For the PF algorithm, the value of is set to .
In Figure 2, we have compared the performance of five different scheduling policies on the basis of long-term Max-Age in the set-up described in Section III. The number of Type-I UEs associated with the BS has been varied from to . For reference, we have also included the Theoretical Optimal value of AoI, given in Theorem 1. As expected, we see that the performance of the Max-Age (MA) policy matches with the optimal value. The Max-Weight policy performs slightly worse than the optimal MA policy. However, we find that the randomized and the PF policy performs very poorly in terms of the long-term peak-age metric. The bottom line is that a utility-maximizing policy (such as PF, which maximizes the summation of logarithmic rates of the UEs) may be far from optimality when maximizing freshness of information on the UE side.
Figures 3 and 4 pertain to the problem discussed in Section IV, where the objective is to minimize the long-term peak-AoI, while providing a certain throughput guarantee to UE1, which serves eMBB type of traffic. Consistent with the observation that the parameter amplifies the cost for throughput-loss to UE1 (viz. Eqn. 11), Figure 3 shows that, under the action of the MATP policy, UE1 receives more throughput as the parameter is increased. For very large value of the (), the throughput to UE1 saturates to ( in the Figure 3), which is the maximum-throughput obtainable for UE1 if UE1 is scheduled exclusively. Figure 4 compares the performance of various scheduling policies in terms of the metric given in Eqn. (11). Similar to Figure 2, we see that the MW and the Randomized Policies perform poorly in this case. However, the proposed approximately optimal policy MATP performs close to the theoretical bound and the performance of the MW policy is also not very far from that of the MATP policy.
In this paper, we have derived an optimal downlink scheduling policy for minimizing the long-term peak-Age-of-Information for UEs with URLLC type of traffic. We have also proposed a heuristic scheduling policy in the case when one of the UEs is throughput-constrained. Extensive numerical simulations have been carried out comparing the efficacy of different scheduling policies. Deriving an optimal scheduling policy for minimizing peak-AoI with throughput constraint is an interesting research direction which will be pursued in the future.
The second author would like to thank Prof. Eytan Modiano and Igor Kadota from MIT, for the useful discussions that led to this paper.
-  I. Kadota, A. Sinha, E. Uysal-Biyikoglu, R. Singh, and E. Modiano, “Scheduling policies for minimizing age of information in broadcast wireless networks,” IEEE/ACM Transactions on Networking, pp. 1–14, 2018.
-  R. Johari and D. K. H. Tan, “End-to-end congestion control for the internet: Delays and stability,” IEEE/ACM Trans. Netw., vol. 9, no. 6, pp. 818–832, Dec. 2001. [Online]. Available: http://dx.doi.org/10.1109/90.974534
-  J. Barman, F. Callier, and C. Desoer, “L2-stability and L2-instability of linear time-invariant distributed feedback systems perturbed by a small delay in the loop,” IEEE Transactions on Automatic Control, vol. 18, no. 5, pp. 479–484, October 1973.
N. J. Ferrier, S. Rowe, and A. Blake, “Real-time traffic monitoring,” in
Proceedings of 1994 IEEE Workshop on Applications of Computer Vision. IEEE, 1994, pp. 81–88.
-  C. Neumann, N. Prigent, M. Varvello, and K. Suh, “Challenges in peer-to-peer gaming,” ACM SIGCOMM Computer Communication Review, vol. 37, no. 1, pp. 79–82, 2007.
-  O. Dousse, C. Tavoularis, and P. Thiran, “Delay of intrusion detection in wireless sensor networks,” in Proceedings of the 7th ACM International Symposium on Mobile Ad Hoc Networking and Computing, ser. MobiHoc ’06. New York, NY, USA: ACM, 2006, pp. 155–165. [Online]. Available: http://doi.acm.org/10.1145/1132905.1132923
-  P. Schulz, M. Matthe, H. Klessig, M. Simsek, G. Fettweis, J. Ansari, S. A. Ashraf, B. Almeroth, J. Voigt, I. Riedel et al., “Latency critical iot applications in 5g: Perspective on the design of radio interface and network architecture,” IEEE Communications Magazine, vol. 55, no. 2, pp. 70–78, 2017.
-  L. Tassiulas and A. Ephremides, “Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks,” Automatic Control, IEEE Transactions on, vol. 37, no. 12, pp. 1936–1948, 1992.
-  A. Sinha and E. Modiano, “Optimal control for generalized network-flow problems,” IEEE/ACM Transactions on Networking, vol. 26, no. 1, pp. 506–519, Feb 2018.
-  C. Joo and A. Eryilmaz, “Wireless scheduling for information freshness and synchrony: Drift-based design and heavy-traffic analysis,” IEEE/ACM Transactions on Networking, vol. 26, no. 6, pp. 2556–2568, 2018.
-  P. Popovski, J. J. Nielsen, C. Stefanovic, E. de Carvalho, E. Ström, K. F. Trillingsgaard, A.-S. Bana, D. M. Kim, R. Kotaba, J. Park et al., “Ultra-reliable low-latency communication (urllc): principles and building blocks,” arXiv preprint arXiv:1708.07862, 2017.
-  S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in INFOCOM, 2012 Proceedings IEEE. IEEE, 2012, pp. 2731–2735.
-  A. Kosta, N. Pappas, V. Angelakis et al., “Age of information: A new concept, metric, and tool,” Foundations and Trends® in Networking, vol. 12, no. 3, pp. 162–259, 2017.
-  A. M. Bedewy, Y. Sun, S. Kompella, and N. B. Shroff, “Age-optimal sampling and transmission scheduling in multi-source systems,” arXiv preprint arXiv:1812.09463, 2018.
-  Q. He, D. Yuan, and A. Ephremides, “Optimizing freshness of information: On minimum age link scheduling in wireless systems,” in 2016 14th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), May 2016, pp. 1–8.
-  ——, “On optimal link scheduling with min-max peak age of information in wireless systems,” in 2016 IEEE International Conference on Communications (ICC), May 2016, pp. 1–7.
-  I. Kadota, A. Sinha, and E. Modiano, “Optimizing age of information in wireless networks with throughput constraints,” in IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 2018, pp. 1844–1852.
-  D. P. Bertsekas, Dynamic programming and optimal control. Athena scientific Belmont, MA, 2005, vol. 2, no. 3.
-  J. Weitzen, M. Li, E. Anderland, and V. Eyuboglu, “Large-scale deployment of residential small cells,” Proceedings of the IEEE, vol. 101, no. 11, pp. 2367–2380, 2013.
Constrained Markov decision processes. CRC Press, 1999, vol. 7.
-  A. L. Stolyar, “On the asymptotic optimality of the gradient scheduling algorithm for multiuser throughput allocation,” Operations research, vol. 53, no. 1, pp. 12–25, 2005.
-  H. Kim and Y. Han, “A proportional fair scheduling for multicarrier transmission systems,” IEEE Communications Letters, vol. 9, no. 3, pp. 210–212, March 2005.
-  B. Hajek, Random processes for engineers. Cambridge university press, 2015.
Viii-a Proof of Proposition 2
Using Union bound, we have
Next, for any UE , the event occurs iff at time , it has been at least slots since UEi received a packet successfully. Let and . Since the MA policy successfully serves other UEs exactly once between two consecutive successful service of UEi, it follows that, during the last slots prior to time , at most UEs have successfully received a packet. Thus,
where we have used the bound and defined 111In the case , we take .. The final result now follows from Eqn. (17).
Viii-B Proof of Theorem 5
It is clear that forms a countable-state Markov Chain under the action of the MA policy. To show the positive recurrence of the chain , we analyze the stochastic dynamics of the random variable , and choose it as our Lyapunov function for the subsequent drift analysis.
Let , where we break ties arbitrarily. Then, the MA policy transmits a fresh packet to the th user at time . Over the binary erasure channel that we consider, this packet transmission is successful with probability and is unsuccessful w.p. . In case the packet transmission is unsuccessful, the age of all users increase by . Thus,
On the other hand, in the case when the packet transmission
is successful, the age of the th user drops to , and the age of all other users increases by . Hence, we can write
Finally, note that, under the action of the MA policy, we have . Hence, from the above equation, we conclude that
where . The drift upper-bound (20) shows that if for any , we have . Thus, the one-slot conditional drift of the chosen Lyapunov function is strictly negative whenever the state lies outside the bounded -dimensional box . Finally, using the Foster-Lyapunov Theorem for stability of Markov Chains (Proposition 6.13 (b) of ), we conclude that the Markov Chain is Positive Recurrent.