Starting with Burnetas and Katehakis (1997), regret bounds for reinforcement learning have addressed the question of how difficult it is to learn optimal behavior in an unknown Markov decision process (MDP). Some of these bounds like the one derived in the mentioned Burnetas and Katehakis (1997) depend on particular properties of the underlying MDP, typically some kind of gap that specifies the distance between an optimal and a sub-optimal action or policy (see e.g. Ok et al. (2018) for a recent refinement of such bounds). The first so-called problem independent bounds that have no dependence on any gap-parameter were obtained in Jaksch et al. (2010). For MDPs with states, actions and diameter the regret of the UCRL algorithm was shown to be after any steps. A corresponding lower bound of left the open question of the true dependence of the regret on the parameters and . Recently, regret bounds of have been claimed in Agrawal et al. (2017), however there seems to be a gap in the proof, cf. Sec. 38.9 of Lattimore and Szepesvári (2019), so that the original bounds of Jaksch et al. (2010) are still the best known bounds.
In the simpler episodic setting, the gap between upper and lower bounds has been closed in Azar et al. (2017), showing that the regret is of order , where is the length of an episode. However, while bounds for the non-episodic setting can be easily transferred to the episodic setting, the reverse is not true. We also note that another kind of regret bounds that appears in the literature assumes an MDP sampled from some distribution (see e.g. Osband and Van Roy (2017) for a recent contribution). Regret bounds in this Bayesian setting cannot be turned into bounds for the worst case setting as considered here.
There is also quite some work on bounds on the number of samples from a generative model necessary to approximate the optimal policy by an error of at most . Obviously, having access to a generative model makes learning the optimal policy easier than in the online setting considered here. However, for ergodic MDPs it could be argued that any policy reaches any state so that in this case sample complexity bounds could in principle be turned into regret bounds. We first note that this seems difficult for bounds in the discounted setting, which make up the majority in the literature. Bounds in the discounted setting (see e.g. Azar et al. (2013) or Sidford (2018) for a more recent contribution obtaining near-optimal bounds) depend on the term , where is the discount factor, and it is not clear how this term translates into a mixing time parameter in the average reward case. For the few results in the average reward setting the best sample complexity bound we are aware of is the bound of of wang (2017), where is a mixing time parameter like ours (cf. below) and characterizes the range of stationary distributions across policies. Translated into respective regret bounds, these would have a worse (i.e., linear) dependence on the mixing time and would depend on the additional parameter , which does not appear in our bounds.
Starting with Kearns and Singh (2002); Brafman and Tennenholtz (2002) there are also sample complexity bounds in the literature that were derived for settings without generative sampling model. Although this is obviously harder, there are bounds for the discounted case where the dependence with respect to , , and is the same as for the case with a generative sampling model Szita (2010).
However, we are not aware of any such bounds for the undiscounted setting that would translate into online regret bounds optimal in , , and .
In this note, we present a simple algorithm that allows the derivation of regret bounds of for uniformly ergodic MDPs with mixing time , a parameter that measures how long it takes to approximate the stationary distribution induced by any policy. These bounds are optimal with respect to the parameters , , , and . The only possible improvement is a replacement of by a parameter that may be smaller for some MDPs, such as the diameter Jaksch et al. (2010) or the bias span Bartlett and Tewari (2009); Fruit et al. (2018). We note, however, that it is easy to give MDPs for which is basically of the same size as the mentioned alternative parameters.111See Jaksch et al. (2010); Bartlett and Tewari (2009) for a discussion of various transition parameters used in the literature. Accordingly, the obtained bound basically closes the gap between upper and lower bound on the regret for a subclass of MDPs.
Algorithmically, the algorithm we propose works like an optimistic bandit algorithm such as UCB Auer et al. (2002a). Such algorithms have been proposed before for MDP settings with a limited set of policies Azar et al. (2013). The main difference to the latter approach is that due to the re-use of samples we obtain regret bounds that do not scale with the number of policies but with the number of state-action pairs. We note however that as Azar et al. (2013) our algorithm needs to evaluate each policy independently, which makes it impractical. The proof of the regret bound is much simpler than for bounds achieved before and relies on concentration results for Markov chains.
We consider reinforcement learning in an average reward Markov decision process (MDP) with finite state space and finite action space . We assume that each stationary policy induces a uniformly ergodic222See Section 3 for definitions. Markov chain on the state space. In such MDPs, which we call uniformly ergodic, the chain induced by a policy has a unique stationary distribution , and the (state-independent) average reward can be written as , where and
are the (column) vectors for the stationary distribution and the average reward under, respectively. We assume that the reward distribution for each state-action pair has support in .
The maximal average reward is known (cf. Puterman (1994)) to be achieved by a stationary policy that gives average reward . We are interested in the regret accumulated by an algorithm after any number of steps defined as333Since we are only interested in upper bounds on this quantity we ignore the dependence on the initial state to keep things simpler. See Jaksch et al. (2010) for a discussion.
where are the (random) rewards collected by the algorithm at each step .
3 Preliminaries on Markov Chains
In this section, we give some definitions and results about Markov chain concentration that we will use in the following.
3.1 Mixing Times
For two distributions over the same state space with -algebra , let
be the total variational distance between and . A Markov chain with a transition kernel and a stationary distribution is said to be uniformly ergodic, if there are a and a finite such that
Furthermore, the mixing time of the Markov chain is defined as
For a uniformly ergodic MDP we set the mixing time of a policy to be the mixing time of the Markov chain induced by , and define the mixing time of the MDP to be .
3.2 McDiarmid’s Inequality for Markov Chains
Our results mainly rely on the following version of McDiarmid’s inequality for Markov chains from Paulin (2015).
(Corollary 2.10 and the following Remark 2.11 of Paulin (2015))
Consider a uniformly ergodic Markov chain with state space and mixing time . Let with
Lemma 1 can be used to obtain a concentration result for the empirical average reward of any policy in an MDP. This works analogously to the concentration bounds for the total variational distance between the empirical and the stationary distribution (Proposition 2.18 in Paulin (2015)).
Consider an MDP and a policy that induces a uniformly ergodic Markov chain with mixing time . Using (column) vector notation and for the stationary distribution and the reward function under , and writing for the empirical distribution after steps defined as , it holds that
Choosing the error probability to be
, we obtain the following confidence interval that will be used by our algorithm.
Using the same assumptions and notation of Corollary 1, with probability at least ,
3.3 Concentration of the Empirical Distribution
We will also need the following results on the concentration of the empirical state distribution of Markov chains from Paulin (2015). In the following, consider a uniformly ergodic Markov chain with a stationary distribution and a mixing time . Let be the empirical distribution after performing steps in the chain.
(Proposition 2.18 in Paulin (2015))
(Proposition 3.16 and following remark in Paulin (2015))
where is the pseudo-spectral gap444The pseudo-spectral gap is defined as , where is the transition kernel interpreted as linear operator, is the adjoint of , and is the spectral gap of the self-adjoint operator . For more details see Paulin (2015). Here we do not make direct use of this quantity and only use the bound given in Lemma 4. of the chain.
(Proposition 3.4 in Paulin (2015)) In uniformly ergodic Markov chains, the pseudo-spectral gap can be bounded via the mixing time as
We summarize these results in the following corollary.
With probability at least ,
At the core, the Osp algorithm we propose works like the UCB algorithm in the bandit setting. In our case, each policy corresponds to an arm, and the concentration results of the previous chapter are used to obtain suitable confidence intervals for the MDP setting.
Osp (shown in detail as Algorithm 1) does not evaluate the policies at each time step. Instead, it proceeds in phases555We emphasize that we consider non-episodic reinforcement learning and that these phases are internal to the algorithm. (cf. line 3 of Osp), where in each phase an optimistic policy is selected (line 8). This is done (cf. line 5) by first constructing for each policy a sample path from the observations so far. Accordingly, the algorithm keeps a record of all observations. That is, after choosing in a state an action , obtaining the reward , and observing a transition to the next state , the respective observation is appended to the sequence of observations (cf. line 10).
The sample path constructed from the observation sequence contains each observation from at most once. Further, the path is such that there is no unused observation in that could be used to extend the path by appending the observation. In the following, we say that such a path is non-extendible. Algorithm 2 provides an algorithm for constructing a non-extendible path. Alternative constructions could be used for obtaining non-extendible paths as well.
For each possible policy
the algorithm computes an estimate of the average rewardfrom the sample path and considers an optimistic upper confidence value (cf. line 6 of Osp) using the concentration results of Section 3. The policy with the maximal is chosen for use in phase . The length of phase , in which the chosen policy is used, depends on the length of the sample path . That is, is usually played for steps, but at least for steps (cf. line 9).
Note that at the beginning, all sample paths are empty in which case we set the confidence intervals to be , and the algorithm chooses an arbitrary policy. The initial state of the sample paths can be chosen to be the current state, but this is not necessary. Note that by the Markov property the outcomes of all samples are independent of each other. The way Algorithm 2 extracts observations from is analogous to when having access to a generative sampling model as e.g. assumed in work on sample complexity bounds like Azar et al. (2013). In both settings the algorithm can request a sample for a particular state-action pair . The only difference is that in our case at some point there are no suitable samples available anymore, when the construction of the sample path is terminated.
As the goal of this paper is to demonstrate an easy way to obtain optimal regret bounds, we do not elaborate in detail on computational aspects of the algorithm. A brief discussion is however in order. First, note that it is obviously not necessary to construct sample paths from scratch in each phase. It is sufficient to extend the path for each policy with new and previously unused samples. Further, while the algorithm as given is exponential in nature (as it loops over all policies), it may be possible to find the optimistic policy by some kind of optimistic policy gradient algorithm Lazaric (2018). We note that policies in ergodic MDPs exhibit a particular structure (see Section 3 of Ortner (2007)
) that could be exploited by such an algorithm. However, at the moment this is not more than an idea for future research and the details of such an algorithm are yet to be developed.
5 Regret Analysis
The following theorem is the main result of this note.
In uniformly ergodic MDPs, with probability at least the regret of Osp is bounded by
provided that , where .
The improvement with respect to previously known bounds can be achieved due to the fact that the confidence intervals for our algorithm are computed on the policy level and not on the level of rewards and transition probabilities as for UCRL Jaksch et al. (2010). This avoids the problem of having rectangular confidence intervals that lead to an additional factor of in the regret bounds for UCRL, cf. the discussion in Osband and Van Roy (2017).
To keep the exposition simple, we have chosen confidence intervals which give a high probability bound for each horizon . It is easy to adapt the confidence intervals to gain a high probability bound that holds for all simultaneously (cf. Jaksch et al. (2010)).
The mixing time parameter in our bounds is different from the transition parameters in the regret bounds of Auer and Ortner. (2006); Jaksch et al. (2010) or the bias span used in Bartlett and Tewari (2009); Fruit et al. (2018). We note however that for reversible Markov chains, is linearly bounded in the diameter (i.e., the hitting time) of the chain, cf. Section 10.5 of Levin et al. (2009). It follows from the lower bounds on the regret in Jaksch et al. (2010) that the upper bound of Theorem 1 is best possible with respect to the appearing parameters. Mixing times have also been used for sample complexity bounds in reinforcement learning Kearns and Singh (2002); Brafman and Tennenholtz (2002), however not for a fixed constant as in our case but with respect to the required accuracy. It would be desirable to replace the upper bound on all mixing times by the mixing time of the optimal policy like in Azar et al. (2013). However, the technique of Azar et al. (2013) comes at the price of an additional dependence on the number of considered policies, which in our case obviously would damage the bound.
The parameter can be guessed using a standard doubling scheme getting the same regret bounds with a slightly larger constant. Guessing is more costly. Using as a guess for , the additional regret is an additive constant exponential in . We note however, that it is an open problem whether it is possible to get regret bounds depending on a different parameter than the diameter (such as the bias span) without having a larger bound on the quantity, cf. the discussion in Appendix A of Fruit et al. (2018b).
5.1 Proof of Theorem 1
Recall that is the policy applied in phase for steps. The respective optimistic estimate has been computed from a sample path of length .
5.1.1 Estimates are optimistic
We start showing that the values computed by our algorithm from the sample paths of any policy are indeed optimistic. This holds in particular for the employed policies .
With probability at least , for all phases it holds that .
Let us first consider an arbitrary fixed policy and some time step . Using (column) vector notation and for the stationary distribution and the reward function under , and writing and for the respective estimated values at step , we have
with probability at least (using a union bound over all possible values for ). The second term of (2) can be written as
Since the sum is a martingale difference sequence, we obtain by Azuma-Hoeffding inequality (cf. Lemma A.7 in Cesa-Bianchi et al. (2006)) and another union bound that with probability
Accordingly, writing for the time step when phase starts and for the length of the sample path for policy at step ,
It follows that
5.1.2 Splitting regret into phases
Lemma 5 implies that in each phase with high probability
Accordingly, we can split and bound the regret as a sum over the single phases and obtain that with probability at least ,
Now we can distinguish between two kinds of phases: The length of most phases is . However, there there are also a few phases where the sample path for the chosen policy is shorter than , when the length is . Let be the set of these latter phases and set . The regret for each phase in is simply bounded by , while for phases we use666The final phase may be shorter than . to obtain from (5) that
with probability at least .
It remains to bound the number of phases (not) in . A bound on obviously gives a bound on the first term in (6), while a bound on the number of phases not in allows to bound the second term, as by Jensen’s inequality we have due to that
5.1.3 Bounding the number of phases
The following lemma gives a bound on the total number of phases that can be used as a bound on and to conclude the proof of Theorem 1.
With probability at least , the number of phases up to step is bounded by
provided that , where .
Let be the number of visits to before phase . Note that the sample path for each policy in general will not use all samples of , so that we also introduce the notation for the number of samples of used in the sample path of computed before phase . Note that by definition of the algorithm, sample paths are non-extendible, so that for each there is a state for which all samples are used,777In particular, this holds for the last state of the sample path. that is, . We write and for the empirical distributions of the policy in the sample path of and in phase , respectively.
Note that for each phase we have
each with probability at least by Corollary 3 and a union bound over all possible values of and , respectively. By another union bound over the at most phases, (8) and (9) hold for all phases with probability at least . In the following, we assume that the confidence intervals of (8) and (9) hold, so that all following results hold with probability .
Each phase has length at least . Consequently, if , then it is guaranteed by (9) that in each phase it holds that and therefore for each state
Now consider an arbitrary phase and let be the state for which , so that in particular . We are going to show that the number of visits to is increased by (at least) a factor in phase . By (8)–(10) and using that we have
so that abbreviating
Hence in each phase there is a state-action pair for which the number of visits is increased by a factor of . This can be used to show that the total number of phases within steps is upper bounded as
6 Discussion and Conclusion
While we were able to close the gap between lower and upper bound on the regret for uniformly ergodic MDPs, there are still quite a few open questions. First of all, the concentration results we use are only available for uniformly ergodic Markov chains, so a generalization of our approach to more general communicating MDPs seems not easy. An improvement over the parameter may be possible by considering more specific concentration results for Markov reward processes. These might depend not so much on the mixing time than the bias span Fruit et al. (2018). However, even if one achieves such bounds, the resulting regret bounds would depend on the maximum bias span over all policies. Obtaining a dependence on the bias span of the optimal policy instead seems not easily possible. Finally, another topic for future research is to develop an optimistic policy gradient algorithm that computes the optimistic policy more efficiently than by an iteration over all policies.
The author would like to thank Ronan Fruit, Alessandro Lazaric, and Matteo Pirotta for discussion as well as Sadegh Talebi and three anonymous reviewers for pointing out errors in a previous version of the paper.
- Agrawal et al.  S. Agrawal and R. Jia. Optimistic posterior sampling for reinforcement learning: worst-case regret bounds. In Adv. Neural Inf. Process. Syst. 30, NIPS 2017, pages 1184–1194. 2017.
- Auer and Ortner.  P. Auer and R. Ortner. Logarithmic Online Regret Bounds for Undiscounted Reinforcement Learning. In Adv. Neural Inf. Process. Syst. 19, NIPS 2006, pages 49–56. 2006.
- Auer et al. [2002a] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multi-armed bandit problem. Mach. Learn., 47:235–256, 2002a.
- Azar et al.  M. G. Azar, R. Munos, and H. J. Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model. Mach. Learn., 91(1):(3):325–349, 2013.
- Azar et al.  M. G. Azar, A. Lazaric, and E. Brunskill. Regret Bounds for Reinforcement Learning with Policy Advice. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2013, Proceedings, Part I, pages 97–112. 2013.
- Azar et al.  M. G. Azar, I. Osband, and R. Munos. Minimax Regret Bounds for Reinforcement Learning. In Proc. 34th International Conference on Machine Learning, ICML 2017, pages 263–272. 2007.
Bartlett and Tewari 
P. L. Bartlett and A. Tewari.
REGAL: A regularization based algorithm for reinforcement learning
in weakly communicating MDPs.
Proc. 25th Conference on Uncertainty in Artificial Intelligence, UAI 2009, pages 25–42, 2009.
- Brafman and Tennenholtz  R. I. Brafman and M. Tennenholtz. R-max – a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res., 3:213–231, 2002.
- Burnetas and Katehakis  A. N. Burnetas and M. N. Katehakis. Optimal adaptive policies for Markov decision processes. Math. Oper. Res., 22(1):222–255, 1997.
- Cesa-Bianchi et al.  N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.
- Fruit et al.  R. Fruit, M. Pirotta, A. Lazaric, and R. Ortner. Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning In Proc. 35th International Conference on Machine Learning. ICML 2018, pages 1573–1581. 2018.
- Fruit et al. [2018b] R. Fruit, M. Pirotta, and A. Lazaric. Near Optimal Exploration-Exploitation in Non-Communicating Markov Decision Processes. CoRR abs/1807.02373 (2018)
- Jaksch et al.  T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. J. Mach. Learn. Res., 11:1563–1600, 2010.
- Kearns and Singh  M. J. Kearns and S. P. Singh. Near-optimal reinforcement learning in polynomial time. Mach. Learn., 49:209–232, 2002.
- Lazaric  A. Lazaric. personal communication, August and October 2018.
- Levin et al.  D. A. Levin, Y. Peres, and E. L. Wilmer. Markov Chains and Mixing Times. American Mathematical Society, Providence, USA, 2009.
- Ok et al.  J. Ok, A. Proutière, D. Tranos. Exploration in Structured Reinforcement Learning. In Adv. Neural Inf. Process. Syst. 31, NeurIPS 2018, pages 8888–8896.
- Ortner  R. Ortner. Online regret bounds for Markov decision processes with deterministic transitions. Theor. Comput. Sci., 411(29–30):2684–2695, 2010.
- Ortner  R. Ortner. Linear dependence of stationary distributions in ergodic Markov decision processes. OR Letters, 35(5):619–626, 2007.
- Osband and Van Roy  Ian Osband, Benjamin Van Roy. Why is Posterior Sampling Better than Optimism for Reinforcement Learning? In Proc. 34th International Conference on Machine Learning. ICML 2017, pages 2701–2710.
- Paulin  D. Paulin. Concentration inequalities for Markov chains by Marton couplings and spectral methods. Electron. J. Probab., 20(79):1–32, 2015.
- Puterman  M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1994.
- Sidford  A. Sidford, M. Wang, X. Wu, L. Yang, Y. Ye: Near-Optimal Time and Sample Complexities for Solving Markov Decision Processes with a Generative Model. In Adv. Neural Inf. Process. Syst. 31, NeurIPS 2018, pages 5192–5202.
- Lattimore and Szepesvári  T. Lattimore and C. Szepesvári. Bandit Algorithms. Draft Revision 1631 from January 2019, available at http://banditalgs.com/
- Szita  I. Szita, C. Szepesvári: Model-based reinforcement learning with nearly tight exploration complexity bounds. In Proc. 27th International Conference on Machine Learning. ICML 2010, pages 1031–1038.
Tewari and Bartlett 
A. Tewari and P. Bartlett.
Optimistic linear programming gives logarithmic regret for irreducible MDPs.In Adv. Neural Inf. Process. Syst. 20, NIPS 2007, pages 1505–1512. 2008.
- wang  Mengdi Wang: Primal-Dual Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems. Draft available at https://arxiv.org/abs/1710.06100, 2017