I Introduction
Reinforcement Learning (RL) is a form of implicit stochastic adaptive control where the optimal control policy is estimated without directly estimating the underlying model. This paper considers reinforcement learning for an infinite horizon discounted cost continuous state Markov decision process. In a MDP, actions affect the Markovian state dynamics and result in rewards for the agent. The objective is to find a map from the states to actions, also known as policy, that results in the accumulation of largest expected return
[1]. There are many approaches to estimate a policy: policy iteration, learning [2, 3] (which operates in “value” space [4]), policygradients [5, 6] (that operate in policy space); see [7, 8].Recently, policygradient algorithms have gained popularity due to their ability to address complex realworld RL problems with continuous stateaction spaces. Given a parametrized policy space, usually designed to incorporate domain knowledge, policygradient algorithms update policy parameters along an estimated ascent direction of the expected return. Depending on whether the expected reward or the value function is convex or nonconvex, the parameters converge to a minimum or a stationary point; for a comprehensive survey see [9, 10].
Typically, to compute the ascent direction in policy search [11], one employs the Policy Gradient Theorem [8] to write the gradient as the product of two factors: the function^{1}^{1}1function is also known as the stateaction value function [8]. It gives the expected return for a choice of action in a given state. and the score function (a likelihood ratio). This score function approach has yielded numerous viable policy search techniques [12, 13, 14, 8], although the resulting gradient estimates are afflicted with high variance: the score function is a martingale and so for a Markov process its variance is for measurements. In pursuit of reducing the variance, we propose replacing the score function with the Jordan decomposition of signed measures [15], similar to the method^{2}^{2}2Jordan decomposition (also known as HahnJordan decomposition) of signed measures is a specific type of weak derivative form  this expresses the derivative of a measure as the weighted difference of orthogonal measures. For example, the gradient of gaussian policy [13] is written as a (scaled) difference of two Rayleigh policies. of Weak Derivatives in the finite stateaction MDP literature; see [16, 17, 18].
To estimate the function in the policy gradient [8], we use Monte Carlo rollouts with random path lengths akin to [19]
, motivated by the fact that obtaining unbiased estimates of continuous stateaction
function in the infinite horizon case is otherwise challenging. The product of these terms yields a valid estimate of the overall policy gradient, as in [8].This paper considers reinforcement learning for the case when the underlying system can be simulated using statistically independent trials with different policies. Our main results are:

A policy gradient theorem using Jordan decomposition for the policy gradient. We establish that the resulting policy gradient algorithm, named Policy Gradient with Jordan Decomposition (PGJD), yields unbiased estimates of the gradient of the reward function.

to establish that the PGJD algorithm converges to a stationary point of the parametrized value function almost surely under decreasing stepsizes.

to derive the iteration (and sample^{3}^{3}3Iteration complexity is a measure of the number of changes of the unknown parameter. Sample complexity includes the additional simulations required to estimate the continuous stateaction function using Monte Carlo rollout with random path lengths.) complexity as , where is the time step. This shows that the convergence rate is similar to stochastic gradient method for nonconvex settings.

to upperbound the expected variance of the gradient estimates obtained using the PGJD algorithm, which isshown to be lower than those generated by score function methods using Monte Carlo rollouts with random path lengths, for common policy parametrizations.
The setup and problem formulation are discussed in Sec. II. The new policy gradient theorem using weak derivatives (Jordan decomposition) is derived in Sec. III. The algorithm to compute the stochastic gradient and the policy parameter update is given in Sec. IV. Convergence analysis of the stochastic gradient ascent algorithm and its statistical properties are derived in Sec. V. Numerical studies on OpenAI gym using the pendulum environment is discussed in Sec. VI.
Ii Problem Formulation and Policy Search
The problem of reinforcement learning is considered in the framework of Markov Decision Process, which is defined as a tuple consisting of the state space , a subset of Euclidean space with elements ; the action space , a subset of Euclidean space with elements ; the transition law
, a probability density function
that assigns a nextstate upon taking action in state , where denotes the set of all probability measures on ; the reward function , a real valued function on the product space ; the discount , a parameter that scales the importance of future rewards.A stochastic Markov policy is defined as a sequence of transition probabilities from to such that for each and . Here maps each to the set of all available actions . Let denote the class of stochastic Markov policies.
For an initial state and a stochastic Markov policy , define the expected reward function
(1) 
For an initial state and a Markov policy , using Ionescu Tulcea theorem [20, 21], define as
(2)  
Here is an atomic measure with . The expectation in (1) is with respect to in (2). Our goal is to find the policy that maximizes the longterm reward accumulation, or value:
(3) 
For the infinite horizon problem (3), it is sufficient [22, 23, 21, 24] to restrict the class of policies to the class of stationary stochastic Markov policies. A stationary stochastic Markov policy is defined as the transition probability from to such that for each . In order to solve (3) we resort to direct policy search over the space of continuous stationary policies. It is convenient to parametrize the stationary policy as for for , and search over the space of . For example, consider Gaussian policy . Here the function is commonly referred to as the feature map and
denotes the standard deviation. With a slight abuse of notation, the problem (
3) can be reformulated in terms of the finding a parameter vector
to satisfy:(4)  
Here is the expectation with respect to the measure induced by the probability measure as in (2) with the policy and initial state .
Iii Policy Gradient Theorem via HahnJordan
The foundation of any valid policy search technique is a valid ascent direction on the value function with respect to the policy parameters. Classically, one may derive that the policy gradient decomposes into two factors: the actionvalue (Q) function and the score function [5]. Here we establish that one may obviate the need for the log trick that gives rise to the score function through measurevalued differentiation by employing the Jordan decomposition of signed measures [15]. To begin doing so, define the function as
(5) 
The weak derivative of the signed measure using Jordan decomposition ^{4}^{4}4
Result 1.
[15] [Hahn Decomposition] Let be a finite signed measure on the measurable space . There exists a disjoint partition of the set into and such that , , and .Result 2.
[15] [Jordan Decomposition] Every finite signed measure has a unique decomposition into a difference of two finite nonnegative measures and such that for any Hahn decomposition of , we have for that if and if .(6) 
Here the decomposed positive and negative component measures and are orthogonal in (see Example 1 below). The ergodic measure associated with the transition kernel and policy is . The induced measures on by and are defined as and . Using this measure (weak) derivative representation of the policy, we can write the gradient of the value function with respect to policy parameters in an unusual way which is given in the following theorem.
Theorem 1.
(Jordan Decomposition for Policy Gradients) The policy gradient using Jordan decomposition takes the form
(7)  
where is a normalizing constant to ensure and are valid measures.
Discussion: A proof is included in the Appendix. Theorem 1 is the policy gradient theorem using weak derivatives, specifically Jordan decomposition. In Theorem 1, note that the functions in the expectations are the same, indicating that the model is unaffected by the measure decomposition; only the induced measures are different. The expression for the gradient in (7) contains a difference of two expectations. Unlike, the method of score functions, the expectation obviates the need for a score function
term. Intuitively, this allows us to avoid computing the logarithm of the policy which may amplify useless parts of the stateaction space and cause variance to needlessly be increased, and instead yield a sharp “perceptronlike” behavior. In subsequent sections, we indeed establish that this representation may reduce variance but this reduction intrinsically depends on the policy parameterization. Note that
for a given parameter and state , is a constant, which makes the stochastic gradient easier to compute in Algorithm 2. Before continuing, we present a representative example.Example 1.
Consider a gaussian policy
, where the mean of the gaussian distribution is modulated by the optimization parameter. The Jordan decomposition of the gaussian policy can be derived as follows:
(8)  
(9) 
Here we may glean the normalizing constant and the positive and negative component measures are
(10)  
(11) 
Observe that and define the Rayleigh^{5}^{5}5The probability density function corresponding to Rayleigh distribution is: , . policy. They are orthogonal in the sense that is defined on ^{6}^{6}6 denotes the indicator function. and is defined over .
Iv Policy Search via Jordan Decomposition
In order to develop a policy search method based on Theorem 1, we need samples of both factors inside the expectation in (7) which are unbiased. We first focus on the later factor, the function.
Iva Estimating the ActionValue
The estimation of the function is carried out using Monte Carlo rollouts of random path lengths, similar to [19]
. Here the random length is a geometric random variable with parameter
, the discount factor in the reinforcement learning problem. Specifically, we simulate and then simulate stateaction pairs according to the positive and negative induced policies and . For this time horizon, we collect rewards for the two different trajectories.More specifically, from a given starting state , a (real) trajectory is simulated to update the policy parameters
. At each epoch
of the parameter update , the simulator (modeled as ) is called two times to simulate two different (phantom^{7}^{7}7Here the word “phantom” is used to refer to the actions on the simulator.) trajectories. These trajectories correspond to the random MonteCarlo rollouts used to estimate the functions with two different policies, the positive and negative policy measure, and hence the stochastic gradient of the expected reward function. Letdenote a geometrically distributed random variable:
where is the discount factor. Let the pathwise cost be defined by .Discussion: Algorithm 2 with Algorithm 1 is the stochastic gradient algorithm that is used to update the policy parameters. The simulation consists of a single simulation (real trajectory) to update the parameters and multiple phantom simulations to estimate the gradient of the expected reward function. The two phantom trajectories correspond to different polices and not different models, starting from the system’s state represented by the state corresponding to the real trajectory. The stochastic gradient computation is summarized in three steps: For a fixed initial state– (i) Simulate two phantom initial actions from the measures obtained using Jordan decomposition, i.e, and . (ii) Simulate a geometric random variable , and (iii) Perform Monte Carlo rollouts of length (i.e, simulate and feed actions to the simulator and collect the rewards) using the system policy derived from old parameters, i.e using and .
The merit of using these random horizons for estimation of the Q function, as summarized in Algorithm 1, is that one may establish that it is an unbiased estimate in the infinitehorizon discounted case, as we summarize in the following theorem.
Theorem 2.
For a geometric r.v , let the approximate stateaction value function (Qfunction) be defined by . Let denote a geometrically distributed random variable. Then,
(12)  
(13) 
A proof is included in the Appendix. Now that we may obtain unbiased samples of the actionvalue function, we shift focus to how to compute the stochastic gradients needed for policy search based on Jordan decomposition (Theorem 1).
IvB Stochastic Gradient Algorithm
With the estimation of the actionvalue function addressed, we now discuss how we can sample the former factor: the signed measure gradients. Specifically, Theorem 1 can be used to effectively compute the gradient given access to an oracle/simulator that may generate stateactionreward triples. It is well known that one only needs to compute estimates of the gradient that are unbiased in expectation to ensure convergence of the iterates to a stationary point [8]. This results in a modification of the gradient expression as in REINFORCE algorithm [12, 8], which is a stochastic gradient, for computing the optimal policy of the reinforcement learning problem. Let denote the expectation with respect to the geometric distribution.
Using Theorem 2 and Fubini’s Theorem [26], the gradient in (7) can be rewritten to make it implementable on a simulator:
(14) 
We have from Theorem 2 and (14),
(15)  
(16) 
Here the initial state simulated from the ergodic measure is , and the policies that simulate the two trajectories are: and . Here the initial actions are simulated from the decomposed measures and the parametrized policy is used for the remainder of the trajectory simulation. Here (15) is the (stochastic) gradient estimate for a random path length and (16) is the (stochastic) gradient estimate using a realization . Using the estimates (16) that are computable using Algorithm 1 to estimate the Q function with respect to the signed measures, then, we may write out an iterative stochastic gradient method to optimize with respect to the value function as
(17) 
The overall policy search routine is summarized as Algorithm 2. Its convergence and variance properties are discussed in the following section.
V Convergence, Complexity, Variance Analysis
In this section, we discuss a few properties of the stochastic gradient ascent algorithm derived using weak derivatives, namely, convergence, the iteration complexity, sample complexity, and the variance of the resulting gradient estimates.
Va Convergence Analysis
We now analyze the convergence of the PGJD algorithm (Algorithm 2 ), establishing that the stochastic gradient estimates obtained from the algorithm are unbiased estimates of the true gradient, and that the parameter sequence (17) converges almost surely to a stationary point of the value function (4). To do so, some assumptions are required which we state next.
VA1 Assumptions

[label=()]

The reward function^{8}^{8}8Let the product space be equipped with the taxicab norm:

The transition law^{9}^{9}9As in [27],
denotes the Kantorovich distance between probability distributions
and . It is given by: 
For , the transition law is irreducible, positive Harris recurrent, and geometrically ergodic.

The continuous policy is Lipschitz, i.e,

and .

The stochastic gradient
for all , and .
Assumptions 1  3 are model assumptions, whereas Assumptions 4  6 impose restrictions on how the algorithm behaves. Assumption 1 is standard, and tied to learnability of the problem. Assumption 2 is a continuity assumption on the transition law that is easily satisfied by most physical systems. Assumption 3 makes sure that for every policy
, there exists a unique invariant (stationary) measure and the Markov chain reaches stationarity geometrically fast; see
[28]. All the results hold without the transition law being geometrically ergodic. Assuming geometric ergodicity makes simulating from the ergodic measure (in Algorithm2, Sec.IV) more meaningful. Regarding the algorithmic conditions: Assumptions 45 are standard in stochastic gradient methods; see [29]. Assumption 6 says that the stochastic gradient is always bounded by the true gradient, which can grow unbounded with . This assumption makes sure that the martingale noise in the stochastic gradient algorithm is bounded by the true gradient; see [29].Proposition 1.
The following result makes sure that the stochastic gradient estimates so obtained are representative of the true gradient.
Theorem 3.
The stochastic gradient obtained in (16) is an unbiased estimate of the true gradient , i.e,
(19) 
Discussion: A proof is included in the Appendix. Theorem 3 says that the estimates of the stochastic gradient are unbiased in expectation. This is required to ensure the almost sure convergence of the iterates to a stationary point [8].
Theorem 4.
Discussion: A proof is included in the Appendix. The expected cost function , under model assumptions, is continuous and Lipschitz; see [Chapter 7] [30] and [27]. Theorem 4 says that the sequence of iterates converges to with probability one, and since is a continuous function, converges to with probability one. The gradient (which can be unbounded) at iterates is such that with probability one.
VB Sample Complexity
In this section, we consider the convergence rate analysis of the PGJD algorithm. We choose the stepsize to be for some parameter . Since the optimization of is generally nonconvex, we consider the convergence rate in terms of a metric of nonstationarity, i.e., the norm of the gradient . The following theorem considers a diminishing stepsize and establishes a rate for the decrement of the expected gradient norm square .
Theorem 5.
Let be the sequence of parameters of the policy generated by Algorithm 2. Let the stepsize be for and for some . Let
(21) 
denote the number of iteration steps for the norm of the expected cost to come within the error neighbourhood. Then,
(22) 
where optimizing the complexity bound over , we have . Therefore, .
Discussion: A proof is included in the Appendix. Theorem 5 characterizes the iteration complexity, which is a measure of the number of iteration steps of the algorithm are required to settle down on a stationary point of the value function. The iteration complexity is showing that the convergence rate is similar to the stochastic gradient methods for convex settings.
Corollary 6.
Let denote the discount factor and denote the iteration complexity. The average sample complexity using Algorithm 2 is given as:
(23) 
Discussion: A proof is included in the Appendix. Corollary 6 characterizes the sample complexity, which is a measure of the number of the expected total number of actions and states realized. Higher the discount factor , longer the two (random) MonteCarlo rollouts (trajectories) that need to simulated, and hence higher the sample complexity. Together the complexity results, Theorem 5 and Corollary 6, provide an estimate of the duration and expected number of simulations to learn a stationary solution for the reinforcement learning task considered.
VC Variance Analysis
In this section, we provide an analysis of the variance of the stochastic gradient estimates obtained using weak derivatives and score function approaches. Since the function estimation in the computation of the gradient is performed using random Monte Carlo rollouts as in [19], the stochastic gradient obtained is a function of the geometric random variable that characterizes the rollout (trajectory) length. To obtain a comparison of the different methods – weak derivatives and score function – we consider the expected variance of the gradient estimates. A proof of Theorem 7 is given in the Appendix. The proof of Theorem 8 is similar and hence omitted.
Theorem 7.
The expected variance of the gradient estimates obtained using weak derivatives is given as:
(24) 
where .
Theorem 8.
The expected variance of the gradient estimates , if score function is used instead of weak derivatives, is given as:
(25) 
where .
Corollary 9.
For the Gaussian policy , we have
(26) 
Hence, the maximum expected variance of the gradient estimates using weak derivatives is smaller than those obtained using the score function method.
Vi Numerical Studies
In this section, we present a simple experiment using PGJD algorithm on the Pendulum environment in OpenAI gym [31]. The performance is compared with Monte Carlo Policy Gradient using Score Function (PGSF) which is akin to REINFORCE [32] with random rollout horizons; see Fig.1. In the simulation environment, the pendulum starts at a random position, and the goal is to swing it up so that it stays upright. The environment state is a vector of dimension three, i.e., , where is the angle between the pendulum and the upright direction, and is the derivative of . The action is a onedimensional scalar modified using a function, and represents the joint effort.
The received reward is given as
(27) 
which lies in , is normalized between and lies in . The transition dynamics are determined according to Newton’s Second Law of Motion. We use Gaussian policy , which is parameterized as , where and being the feature vector. The policy is a stationary policy (timehomogeneous) as it is well known [7] to be sufficient for infinite or random horizon discounted MDP problems. Observe that the discounted return is higher on average using PGJD as opposed to PGSF, which may attributable to the variancereduced properties of the policy gradient estimates using signed measures as compared with the score function.
Remark: It is noted that for common parametrizations of the mean of the Gaussian policy [13], for example like linear – , the score function is unbounded with respect to with the expression being . This results in convergence issues in policy gradient algorithms for unbounded and unbounded state spaces. However, using Jordan decomposition, even with linear parametrization and unboundedness, the convergence of the policy gradient algorithm is ensured due to the absence of explicit function of .
References
 [1] R. Bellman, Dynamic Programming, 1st ed. Princeton, NJ, USA: Princeton University Press, 1957.
 [2] C. Watkins and J. C. Hellaby, “Learning from delayed rewards,” Ph.D. dissertation, King’s College, Cambridge, UK, May 1989.
 [3] E. Tolstaya, A. Koppel, E. Stump, and A. Ribeiro, “Nonparametric stochastic compositional gradient descent for Qlearning in continuous markov decision problems,” in 2018 Annual American Control Conference (ACC). IEEE, 2018, pp. 6608–6615.
 [4] A. Koppel, G. Warnell, E. Stump, and A. Ribeiro, “Policy evaluation in continuous MDPs with efficient kernelized gradient temporal difference,” 2017.
 [5] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in neural information processing systems, 2000, pp. 1057–1063.
 [6] K. Zhang, A. Koppel, H. Zhu, and T. Basar, “Global Convergence of Policy Gradient Methods: A Nonconvex Optimization Perspective,” SIAM Journal on control and Optimization (under review), 2019.
 [7] D. P. Bertsekas, Dynamic Programming and Optimal Control, 2005, vol. 1, no. 3.
 [8] R. S. Sutton, A. G. Barto et al., Reinforcement Learning: An Introduction, 2nd ed., 2017.
 [9] I. Grondman, L. Busoniu, G. A. Lopes, and R. Babuska, “A survey of actorcritic reinforcement learning: Standard and natural policy gradients,” IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), vol. 42, no. 6, pp. 1291–1307, 2012.
 [10] M. P. Deisenroth, G. Neumann, J. Peters et al., “A survey on policy search for robotics,” Foundations and Trends® in Robotics, vol. 2, no. 1–2, pp. 1–142, 2013.
 [11] D. Silver, “Reinforcement learning and simulationbased search,” Doctor of philosophy, University of Alberta, 2009.
 [12] R. J. Williams, “Simple statistical gradientfollowing algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 34, pp. 229–256, 1992.
 [13] K. Doya, “Reinforcement learning in continuous time and space,” Neural Computation, vol. 12, no. 1, pp. 219–245, 2000.
 [14] E. Greensmith, P. L. Bartlett, and J. Baxter, “Variance reduction techniques for gradient estimates in reinforcement learning,” Journal of Machine Learning Research, vol. 5, no. Nov, pp. 1471–1530, 2004.
 [15] P. Billingsley, Probability and measure. John Wiley & Sons, 2008.
 [16] P. Glasserman and Y.C. Ho, Gradient estimation via perturbation analysis. Springer Science & Business Media, 1991, vol. 116.
 [17] V. Krishnamurthy and F. J. V. Abad, “Gradient based policy optimization of constrained markov decision processes,” in Stochastic Processes, Finance and Control: A Festschrift in Honor of Robert J Elliott. World Scientific, 2012, pp. 503–547.
 [18] V. Krishnamurthy, Partially Observed Markov Decision Processes. Cambridge University Press, 2016.
 [19] S. Paternain, “Stochastic Control Foundations of Autonomous Behavior,” Ph.D. dissertation, University of Pennsylvania, 2018.
 [20] J. Neveu, Mathematical foundations of the calculus of probability. Holdenday, 1965.
 [21] O. HernándezLerma and J. B. Lasserre, Discretetime Markov control processes: basic optimality criteria. Springer Science & Business Media, 2012, vol. 30.
 [22] D. Blackwell, “Discounted dynamic programming,” The Annals of Mathematical Statistics, vol. 36, no. 1, pp. 226–235, 1965.
 [23] D. P. Bertsekas and S. E. Shreve, Stochastic optimal control: the discretetime case. Academic Press Inc.[Harcourt Brace Jovanovich Publishers], New York, 1978.
 [24] E. A. Feinberg, “On measurability and representation of strategic measures in Markov decision processes,” Lecture NotesMonograph Series, pp. 29–43, 1996.
 [25] S. Bhatt, A. Koppel, and V. Krishnamurthy, “Policy Gradient using Weak Derivatives for Reinforcement Learning,” U.S. Army Research Laboratory/ Cornell UniversityTechnical Report, 2019., https://koppel.netlify.com/assets/papers/2019_report_sujay_etal.pdf.
 [26] V. I. Bogachev, Measure theory. Springer Science & Business Media, 2007, vol. 1.
 [27] K. Hinderer, “Lipschitz continuity of value functions in Markovian decision processes,” Mathematical Methods of Operations Research, vol. 62, no. 1, pp. 3–22, 2005.
 [28] O. HernándezLerma and J. B. Lasserre, Markov chains and invariant probabilities. Birkhäuser, 2012, vol. 211.
 [29] D. P. Bertsekas and J. N. Tsitsiklis, “Gradient convergence in gradient methods with errors,” SIAM Journal on Optimization, vol. 10, no. 3, pp. 627–642, 2000.
 [30] N. Bäuerle and U. Rieder, Markov decision processes with applications to finance. Springer Science & Business Media, 2011.
 [31] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016.
 [32] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Advances in neural information processing systems, 2000, pp. 1057–1063.
Appendix A Proofs
Aa Proof of Theorem 1
We begin by computing the derivative of the value function with respect to policy , assuming that the policy is parameterized by a vector .
By HahnJordan decomposition ([15]) for the policy gradient,
(28)  
(29) 
where we defime the ergodic measure [8] , the positive induced measure and the negative induced measure .
AB Proof of Theorem 2
Equation (12) follows by definition of the pathwise cost. We will prove (13) below. Here is a geometric random variable.
(30)  
By boundedness of rewards and using Fubini’s Theorem [26], (AB) can be written as:
By Linearity of Expectation with bounded rewards,
by virtue of geometric distribution. 
Therefore,
AC Proof of Theorem 3
Consider the stochastic gradient in (16) when used with the two simulator policies associated with the positive and negative induced measures,
(31)  
Using Ionescu Tulcea theorem [20, 21], define the induced probability measures (as in (2)) and for some initial state . Let the expectation operator
(32) 
indicate the expectation with respect to measures and respectively. By Fubini’s Theorem [26], result (13), and by the property of the expectation and we have
Comments
There are no comments yet.