Reinforcement Learning (RL) is a form of implicit stochastic adaptive control where the optimal control policy is estimated without directly estimating the underlying model. This paper considers reinforcement learning for an infinite horizon discounted cost continuous state Markov decision process. In a MDP, actions affect the Markovian state dynamics and result in rewards for the agent. The objective is to find a map from the states to actions, also known as policy, that results in the accumulation of largest expected return. There are many approaches to estimate a policy: policy iteration, learning [2, 3] (which operates in “value” space ), policy-gradients [5, 6] (that operate in policy space); see [7, 8].
Recently, policy-gradient algorithms have gained popularity due to their ability to address complex real-world RL problems with continuous state-action spaces. Given a parametrized policy space, usually designed to incorporate domain knowledge, policy-gradient algorithms update policy parameters along an estimated ascent direction of the expected return. Depending on whether the expected reward or the value function is convex or non-convex, the parameters converge to a minimum or a stationary point; for a comprehensive survey see [9, 10].
Typically, to compute the ascent direction in policy search , one employs the Policy Gradient Theorem  to write the gradient as the product of two factors: the function111function is also known as the state-action value function . It gives the expected return for a choice of action in a given state. and the score function (a likelihood ratio). This score function approach has yielded numerous viable policy search techniques [12, 13, 14, 8], although the resulting gradient estimates are afflicted with high variance: the score function is a martingale and so for a Markov process its variance is for measurements. In pursuit of reducing the variance, we propose replacing the score function with the Jordan decomposition of signed measures , similar to the method222Jordan decomposition (also known as Hahn-Jordan decomposition) of signed measures is a specific type of weak derivative form - this expresses the derivative of a measure as the weighted difference of orthogonal measures. For example, the gradient of gaussian policy  is written as a (scaled) difference of two Rayleigh policies. of Weak Derivatives in the finite state-action MDP literature; see [16, 17, 18].
, motivated by the fact that obtaining unbiased estimates of continuous state-actionfunction in the infinite horizon case is otherwise challenging. The product of these terms yields a valid estimate of the overall policy gradient, as in .
This paper considers reinforcement learning for the case when the underlying system can be simulated using statistically independent trials with different policies. Our main results are:
A policy gradient theorem using Jordan decomposition for the policy gradient. We establish that the resulting policy gradient algorithm, named Policy Gradient with Jordan Decomposition (PG-JD), yields unbiased estimates of the gradient of the reward function.
to establish that the PG-JD algorithm converges to a stationary point of the parametrized value function almost surely under decreasing step-sizes.
to derive the iteration (and sample333Iteration complexity is a measure of the number of changes of the unknown parameter. Sample complexity includes the additional simulations required to estimate the continuous state-action function using Monte Carlo roll-out with random path lengths.) complexity as , where is the time step. This shows that the convergence rate is similar to stochastic gradient method for non-convex settings.
to upper-bound the expected variance of the gradient estimates obtained using the PG-JD algorithm, which isshown to be lower than those generated by score function methods using Monte Carlo roll-outs with random path lengths, for common policy parametrizations.
The setup and problem formulation are discussed in Sec. II. The new policy gradient theorem using weak derivatives (Jordan decomposition) is derived in Sec. III. The algorithm to compute the stochastic gradient and the policy parameter update is given in Sec. IV. Convergence analysis of the stochastic gradient ascent algorithm and its statistical properties are derived in Sec. V. Numerical studies on OpenAI gym using the pendulum environment is discussed in Sec. VI.
Ii Problem Formulation and Policy Search
The problem of reinforcement learning is considered in the framework of Markov Decision Process, which is defined as a tuple consisting of the state space , a subset of Euclidean space with elements ; the action space , a subset of Euclidean space with elements ; the transition lawthat assigns a next-state upon taking action in state , where denotes the set of all probability measures on ; the reward function , a real valued function on the product space ; the discount , a parameter that scales the importance of future rewards.
A stochastic Markov policy is defined as a sequence of transition probabilities from to such that for each and . Here maps each to the set of all available actions . Let denote the class of stochastic Markov policies.
For an initial state and a stochastic Markov policy , define the expected reward function
For the infinite horizon problem (3), it is sufficient [22, 23, 21, 24] to restrict the class of policies to the class of stationary stochastic Markov policies. A stationary stochastic Markov policy is defined as the transition probability from to such that for each . In order to solve (3) we resort to direct policy search over the space of continuous stationary policies. It is convenient to parametrize the stationary policy as for for , and search over the space of . For example, consider Gaussian policy . Here the function is commonly referred to as the feature map and
denotes the standard deviation. With a slight abuse of notation, the problem (3
) can be reformulated in terms of the finding a parameter vectorto satisfy:
Here is the expectation with respect to the measure induced by the probability measure as in (2) with the policy and initial state .
Iii Policy Gradient Theorem via Hahn-Jordan
The foundation of any valid policy search technique is a valid ascent direction on the value function with respect to the policy parameters. Classically, one may derive that the policy gradient decomposes into two factors: the action-value (Q) function and the score function . Here we establish that one may obviate the need for the log trick that gives rise to the score function through measure-valued differentiation by employing the Jordan decomposition of signed measures . To begin doing so, define the function as
The weak derivative of the signed measure using Jordan decomposition 444
Result 1. [Hahn Decomposition] Let be a finite signed measure on the measurable space . There exists a disjoint partition of the set into and such that , , and .
Result 2. [Jordan Decomposition] Every finite signed measure has a unique decomposition into a difference of two finite non-negative measures and such that for any Hahn decomposition of , we have for that if and if .
Here the decomposed positive and negative component measures and are orthogonal in (see Example 1 below). The ergodic measure associated with the transition kernel and policy is . The induced measures on by and are defined as and . Using this measure (weak) derivative representation of the policy, we can write the gradient of the value function with respect to policy parameters in an unusual way which is given in the following theorem.
(Jordan Decomposition for Policy Gradients) The policy gradient using Jordan decomposition takes the form
where is a normalizing constant to ensure and are valid measures.
Discussion: A proof is included in the Appendix. Theorem 1 is the policy gradient theorem using weak derivatives, specifically Jordan decomposition. In Theorem 1, note that the functions in the expectations are the same, indicating that the model is unaffected by the measure decomposition; only the induced measures are different. The expression for the gradient in (7) contains a difference of two expectations. Unlike, the method of score functions, the expectation obviates the need for a score function
term. Intuitively, this allows us to avoid computing the logarithm of the policy which may amplify useless parts of the state-action space and cause variance to needlessly be increased, and instead yield a sharp “perceptron-like” behavior. In subsequent sections, we indeed establish that this representation may reduce variance but this reduction intrinsically depends on the policy parameterization. Note thatfor a given parameter and state , is a constant, which makes the stochastic gradient easier to compute in Algorithm 2. Before continuing, we present a representative example.
Consider a gaussian policy , where the mean of the gaussian distribution is modulated by the optimization parameter. The Jordan decomposition of the gaussian policy can be derived as follows:
, where the mean of the gaussian distribution is modulated by the optimization parameter. The Jordan decomposition of the gaussian policy can be derived as follows:
Here we may glean the normalizing constant and the positive and negative component measures are
Observe that and define the Rayleigh555The probability density function corresponding to Rayleigh distribution is: , . policy. They are orthogonal in the sense that is defined on 666 denotes the indicator function. and is defined over .
Iv Policy Search via Jordan Decomposition
Iv-a Estimating the Action-Value
The estimation of the function is carried out using Monte Carlo roll-outs of random path lengths, similar to 
. Here the random length is a geometric random variable with parameter, the discount factor in the reinforcement learning problem. Specifically, we simulate and then simulate state-action pairs according to the positive and negative induced policies and . For this time horizon, we collect rewards for the two different trajectories.
More specifically, from a given starting state , a (real) trajectory is simulated to update the policy parameters
. At each epochof the parameter update , the simulator (modeled as ) is called two times to simulate two different (phantom777Here the word “phantom” is used to refer to the actions on the simulator.) trajectories. These trajectories correspond to the random Monte-Carlo roll-outs used to estimate the functions with two different policies, the positive and negative policy measure, and hence the stochastic gradient of the expected reward function. Let
denote a geometrically distributed random variable:where is the discount factor. Let the path-wise cost be defined by .
Discussion: Algorithm 2 with Algorithm 1 is the stochastic gradient algorithm that is used to update the policy parameters. The simulation consists of a single simulation (real trajectory) to update the parameters and multiple phantom simulations to estimate the gradient of the expected reward function. The two phantom trajectories correspond to different polices and not different models, starting from the system’s state represented by the state corresponding to the real trajectory. The stochastic gradient computation is summarized in three steps: For a fixed initial state– (i) Simulate two phantom initial actions from the measures obtained using Jordan decomposition, i.e, and . (ii) Simulate a geometric random variable , and (iii) Perform Monte Carlo roll-outs of length (i.e, simulate and feed actions to the simulator and collect the rewards) using the system policy derived from old parameters, i.e using and .
The merit of using these random horizons for estimation of the Q function, as summarized in Algorithm 1, is that one may establish that it is an unbiased estimate in the infinite-horizon discounted case, as we summarize in the following theorem.
For a geometric r.v , let the approximate state-action value function (Q-function) be defined by . Let denote a geometrically distributed random variable. Then,
A proof is included in the Appendix. Now that we may obtain unbiased samples of the action-value function, we shift focus to how to compute the stochastic gradients needed for policy search based on Jordan decomposition (Theorem 1).
Iv-B Stochastic Gradient Algorithm
With the estimation of the action-value function addressed, we now discuss how we can sample the former factor: the signed measure gradients. Specifically, Theorem 1 can be used to effectively compute the gradient given access to an oracle/simulator that may generate state-action-reward triples. It is well known that one only needs to compute estimates of the gradient that are unbiased in expectation to ensure convergence of the iterates to a stationary point . This results in a modification of the gradient expression as in REINFORCE algorithm [12, 8], which is a stochastic gradient, for computing the optimal policy of the reinforcement learning problem. Let denote the expectation with respect to the geometric distribution.
Here the initial state simulated from the ergodic measure is , and the policies that simulate the two trajectories are: and . Here the initial actions are simulated from the decomposed measures and the parametrized policy is used for the remainder of the trajectory simulation. Here (15) is the (stochastic) gradient estimate for a random path length and (16) is the (stochastic) gradient estimate using a realization . Using the estimates (16) that are computable using Algorithm 1 to estimate the Q function with respect to the signed measures, then, we may write out an iterative stochastic gradient method to optimize with respect to the value function as
The overall policy search routine is summarized as Algorithm 2. Its convergence and variance properties are discussed in the following section.
V Convergence, Complexity, Variance Analysis
In this section, we discuss a few properties of the stochastic gradient ascent algorithm derived using weak derivatives, namely, convergence, the iteration complexity, sample complexity, and the variance of the resulting gradient estimates.
V-a Convergence Analysis
We now analyze the convergence of the PG-JD algorithm (Algorithm 2 ), establishing that the stochastic gradient estimates obtained from the algorithm are unbiased estimates of the true gradient, and that the parameter sequence (17) converges almost surely to a stationary point of the value function (4). To do so, some assumptions are required which we state next.
The reward function888Let the product space be equipped with the taxi-cab norm:
For , the transition law is irreducible, positive Harris recurrent, and geometrically ergodic.
The continuous policy is Lipschitz, i.e,
The stochastic gradient
for all , and .
Assumptions 1 - 3 are model assumptions, whereas Assumptions 4 - 6 impose restrictions on how the algorithm behaves. Assumption 1 is standard, and tied to learnability of the problem. Assumption 2 is a continuity assumption on the transition law that is easily satisfied by most physical systems. Assumption 3 makes sure that for every policy
, there exists a unique invariant (stationary) measure and the Markov chain reaches stationarity geometrically fast; see. All the results hold without the transition law being geometrically ergodic. Assuming geometric ergodicity makes simulating from the ergodic measure (in Algorithm2, Sec.IV) more meaningful. Regarding the algorithmic conditions: Assumptions 4-5 are standard in stochastic gradient methods; see . Assumption 6 says that the stochastic gradient is always bounded by the true gradient, which can grow unbounded with . This assumption makes sure that the martingale noise in the stochastic gradient algorithm is bounded by the true gradient; see .
The following result makes sure that the stochastic gradient estimates so obtained are representative of the true gradient.
The stochastic gradient obtained in (16) is an unbiased estimate of the true gradient , i.e,
Discussion: A proof is included in the Appendix. Theorem 3 says that the estimates of the stochastic gradient are unbiased in expectation. This is required to ensure the almost sure convergence of the iterates to a stationary point .
Discussion: A proof is included in the Appendix. The expected cost function , under model assumptions, is continuous and Lipschitz; see [Chapter 7]  and . Theorem 4 says that the sequence of iterates converges to with probability one, and since is a continuous function, converges to with probability one. The gradient (which can be unbounded) at iterates is such that with probability one.
V-B Sample Complexity
In this section, we consider the convergence rate analysis of the PG-JD algorithm. We choose the stepsize to be for some parameter . Since the optimization of is generally non-convex, we consider the convergence rate in terms of a metric of non-stationarity, i.e., the norm of the gradient . The following theorem considers a diminishing step-size and establishes a rate for the decrement of the expected gradient norm square .
Let be the sequence of parameters of the policy generated by Algorithm 2. Let the stepsize be for and for some . Let
denote the number of iteration steps for the norm of the expected cost to come within the error neighbourhood. Then,
where optimizing the complexity bound over , we have . Therefore, .
Discussion: A proof is included in the Appendix. Theorem 5 characterizes the iteration complexity, which is a measure of the number of iteration steps of the algorithm are required to settle down on a stationary point of the value function. The iteration complexity is showing that the convergence rate is similar to the stochastic gradient methods for convex settings.
Let denote the discount factor and denote the iteration complexity. The average sample complexity using Algorithm 2 is given as:
Discussion: A proof is included in the Appendix. Corollary 6 characterizes the sample complexity, which is a measure of the number of the expected total number of actions and states realized. Higher the discount factor , longer the two (random) Monte-Carlo roll-outs (trajectories) that need to simulated, and hence higher the sample complexity. Together the complexity results, Theorem 5 and Corollary 6, provide an estimate of the duration and expected number of simulations to learn a stationary solution for the reinforcement learning task considered.
V-C Variance Analysis
In this section, we provide an analysis of the variance of the stochastic gradient estimates obtained using weak derivatives and score function approaches. Since the function estimation in the computation of the gradient is performed using random Monte Carlo roll-outs as in , the stochastic gradient obtained is a function of the geometric random variable that characterizes the roll-out (trajectory) length. To obtain a comparison of the different methods – weak derivatives and score function – we consider the expected variance of the gradient estimates. A proof of Theorem 7 is given in the Appendix. The proof of Theorem 8 is similar and hence omitted.
The expected variance of the gradient estimates obtained using weak derivatives is given as:
The expected variance of the gradient estimates , if score function is used instead of weak derivatives, is given as:
For the Gaussian policy , we have
Hence, the maximum expected variance of the gradient estimates using weak derivatives is smaller than those obtained using the score function method.
Vi Numerical Studies
In this section, we present a simple experiment using PG-JD algorithm on the Pendulum environment in OpenAI gym . The performance is compared with Monte Carlo Policy Gradient using Score Function (PG-SF) which is akin to REINFORCE  with random roll-out horizons; see Fig.1. In the simulation environment, the pendulum starts at a random position, and the goal is to swing it up so that it stays upright. The environment state is a vector of dimension three, i.e., , where is the angle between the pendulum and the upright direction, and is the derivative of . The action is a one-dimensional scalar modified using a -function, and represents the joint effort.
The received reward is given as
which lies in , is normalized between and lies in . The transition dynamics are determined according to Newton’s Second Law of Motion. We use Gaussian policy , which is parameterized as , where and being the feature vector. The policy is a stationary policy (time-homogeneous) as it is well known  to be sufficient for infinite or random horizon discounted MDP problems. Observe that the discounted return is higher on average using PG-JD as opposed to PG-SF, which may attributable to the variance-reduced properties of the policy gradient estimates using signed measures as compared with the score function.
Remark: It is noted that for common parametrizations of the mean of the Gaussian policy , for example like linear – , the score function is unbounded with respect to with the expression being . This results in convergence issues in policy gradient algorithms for unbounded and unbounded state spaces. However, using Jordan decomposition, even with linear parametrization and unboundedness, the convergence of the policy gradient algorithm is ensured due to the absence of explicit function of .
-  R. Bellman, Dynamic Programming, 1st ed. Princeton, NJ, USA: Princeton University Press, 1957.
-  C. Watkins and J. C. Hellaby, “Learning from delayed rewards,” Ph.D. dissertation, King’s College, Cambridge, UK, May 1989.
-  E. Tolstaya, A. Koppel, E. Stump, and A. Ribeiro, “Nonparametric stochastic compositional gradient descent for Q-learning in continuous markov decision problems,” in 2018 Annual American Control Conference (ACC). IEEE, 2018, pp. 6608–6615.
-  A. Koppel, G. Warnell, E. Stump, and A. Ribeiro, “Policy evaluation in continuous MDPs with efficient kernelized gradient temporal difference,” 2017.
-  R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in neural information processing systems, 2000, pp. 1057–1063.
-  K. Zhang, A. Koppel, H. Zhu, and T. Basar, “Global Convergence of Policy Gradient Methods: A Nonconvex Optimization Perspective,” SIAM Journal on control and Optimization (under review), 2019.
-  D. P. Bertsekas, Dynamic Programming and Optimal Control, 2005, vol. 1, no. 3.
-  R. S. Sutton, A. G. Barto et al., Reinforcement Learning: An Introduction, 2nd ed., 2017.
-  I. Grondman, L. Busoniu, G. A. Lopes, and R. Babuska, “A survey of actor-critic reinforcement learning: Standard and natural policy gradients,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 1291–1307, 2012.
-  M. P. Deisenroth, G. Neumann, J. Peters et al., “A survey on policy search for robotics,” Foundations and Trends® in Robotics, vol. 2, no. 1–2, pp. 1–142, 2013.
-  D. Silver, “Reinforcement learning and simulation-based search,” Doctor of philosophy, University of Alberta, 2009.
-  R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3-4, pp. 229–256, 1992.
-  K. Doya, “Reinforcement learning in continuous time and space,” Neural Computation, vol. 12, no. 1, pp. 219–245, 2000.
-  E. Greensmith, P. L. Bartlett, and J. Baxter, “Variance reduction techniques for gradient estimates in reinforcement learning,” Journal of Machine Learning Research, vol. 5, no. Nov, pp. 1471–1530, 2004.
-  P. Billingsley, Probability and measure. John Wiley & Sons, 2008.
-  P. Glasserman and Y.-C. Ho, Gradient estimation via perturbation analysis. Springer Science & Business Media, 1991, vol. 116.
-  V. Krishnamurthy and F. J. V. Abad, “Gradient based policy optimization of constrained markov decision processes,” in Stochastic Processes, Finance and Control: A Festschrift in Honor of Robert J Elliott. World Scientific, 2012, pp. 503–547.
-  V. Krishnamurthy, Partially Observed Markov Decision Processes. Cambridge University Press, 2016.
-  S. Paternain, “Stochastic Control Foundations of Autonomous Behavior,” Ph.D. dissertation, University of Pennsylvania, 2018.
-  J. Neveu, Mathematical foundations of the calculus of probability. Holden-day, 1965.
-  O. Hernández-Lerma and J. B. Lasserre, Discrete-time Markov control processes: basic optimality criteria. Springer Science & Business Media, 2012, vol. 30.
-  D. Blackwell, “Discounted dynamic programming,” The Annals of Mathematical Statistics, vol. 36, no. 1, pp. 226–235, 1965.
-  D. P. Bertsekas and S. E. Shreve, Stochastic optimal control: the discrete-time case. Academic Press Inc.[Harcourt Brace Jovanovich Publishers], New York, 1978.
-  E. A. Feinberg, “On measurability and representation of strategic measures in Markov decision processes,” Lecture Notes-Monograph Series, pp. 29–43, 1996.
-  S. Bhatt, A. Koppel, and V. Krishnamurthy, “Policy Gradient using Weak Derivatives for Reinforcement Learning,” U.S. Army Research Laboratory/ Cornell University-Technical Report, 2019., https://koppel.netlify.com/assets/papers/2019_report_sujay_etal.pdf.
-  V. I. Bogachev, Measure theory. Springer Science & Business Media, 2007, vol. 1.
-  K. Hinderer, “Lipschitz continuity of value functions in Markovian decision processes,” Mathematical Methods of Operations Research, vol. 62, no. 1, pp. 3–22, 2005.
-  O. Hernández-Lerma and J. B. Lasserre, Markov chains and invariant probabilities. Birkhäuser, 2012, vol. 211.
-  D. P. Bertsekas and J. N. Tsitsiklis, “Gradient convergence in gradient methods with errors,” SIAM Journal on Optimization, vol. 10, no. 3, pp. 627–642, 2000.
-  N. Bäuerle and U. Rieder, Markov decision processes with applications to finance. Springer Science & Business Media, 2011.
-  G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016.
-  R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in neural information processing systems, 2000, pp. 1057–1063.
Appendix A Proofs
A-a Proof of Theorem 1
We begin by computing the derivative of the value function with respect to policy , assuming that the policy is parameterized by a vector .
By Hahn-Jordan decomposition () for the policy gradient,
where we defime the ergodic measure  , the positive induced measure and the negative induced measure .
A-B Proof of Theorem 2
By Linearity of Expectation with bounded rewards,
|by virtue of geometric distribution.|
A-C Proof of Theorem 3
Consider the stochastic gradient in (16) when used with the two simulator policies associated with the positive and negative induced measures,