Many learning algorithms include a stochastic updating schedule, often based on a Markov chain. Studying the performance of these processes can be carried out using the asynchronous stochastic approximation framework. However, the previous work in this area has focused on continuous, single-valued updates as discussed in the literature (see for example, , , , ). Furthermore some of the assumptions which are typically used are challenging to verify. In this work we expand the asymptotic pseudo-trajectory approach of Benaïm, Hofbauer and Sorin 
to asynchronous stochastic approximations with set-valued mean fields. We incorporate the asynchronicity into the mean field to give a differential inclusion which will characterise the limiting behaviour of the associated learning process.
Consider an iterative process where and denote the component of as where and is finite. A typical stochastic approximation (SA) is of the form
where is a positive, decreasing sequence, is a zero-mean martingale noise sequence, is a bounded sequence which converges to zero and is a Lipschitz continuous mean field. Standard arguments (e.g. ) are then used to show that the limiting behaviour of the iterative process in (1.1
) can be studied through the ordinary differential equation (ODE)
Commonly known as the ODE method of stochastic approximation, originally proposed by Ljung , this technique has been extended by numerous authors, for example Benaïm , Benaïm, Hofbauer and Sorin , Borkar , Kushner and Clark  and Kushner and Yin , . In particular Benaïm, Hofbauer and Sorin  have developed the approach so that under some weak criteria can be updated via a set-valued mean field, . This allows for the limiting behaviour to be studied using the associated differential inclusion.
Standard stochastic approximations are not always applicable; an example which we examine in this paper is when learning action values in a Markov decision process (MDP) and this is also discussed by Konda and Tsitsiklis , Tsitsiklis  and Singh et al. . In a MDP updates are made to a single random component at each iteration. Therefore we have a stochastic, asynchronous updating pattern, where a subset of an iterative process similar to (1.1) can be updated many times before the remaining components are selected for a single update. Based on this idea extensions to the standard theory have been examined such as those by Kushner and Yin , . Here however we follow the extension to asynchronous stochastic approximation provided by Borkar  and Konda and Borkar . They show that when the iterative updates have a Lipschitz continuous mean field then, similarly to a standard stochastic approximation, the limiting behaviour can be studied via the associated differential equation,
where is a diagonal matrix and the diagonal elements of lie in the set for all . This early work on asynchronous stochastic approximations has certain restrictions which limit its usability. In particular, many of the assumptions made in the work of Borkar  and Konda and Borkar  are given in implicit form and are difficult to verify in specific situations.
As with the initial results for a standard stochastic approximation the results of Borkar  are limited to the case when the mean field, , is a Lipschitz continuous function. The subsequent work by Benaïm, Hofbauer and Sorin  on set-valued mean fields leaves the natural question of whether similar results are possible for asynchronous stochastic approximations when a set-valued mean field is used. In addition, the ODE in (1.3) is non-autonomous and the scaling matrix is not explicitly defined. This makes analysis of the limiting behaviour more difficult to study, although some methods for verifying global convergence are outlined by Borkar .
Borkar  originally extended the stochastic approximation framework to two-timescales. Since then Leslie and Collins  extended this idea to multiple timescales and Konda and Borkar  provide a first venture into the two-timescale asynchronous stochastic approximation. However, all of these only consider stochastic approximations when the mean field is Lipschitz continuous.
The aim of this work is to combine and generalise the results by Borkar , , Konda and Borkar  and Benaïm, Hofbauer and Sorin  to create a framework for single and two-timescale asynchronous stochastic approximations which is straightforward to use in practical applications. In this paper we show that, under a set of verifiable assumptions, the diagonal elements of lie in the closed set , for some . The set can be combined with the mean field to form a set-valued mean field, , whose limiting behaviour can be studied via the associated differential inclusion using the results of Benaïm, Hofbauer and Sorin . A natural benefit of using the differential inclusion framework is that can be set-valued as this does not alter the analysis.
This paper is organised in the following manner: Section 2 reviews some previous results on stochastic approximation with differential inclusions and asynchronous stochastic approximations. In Section 3 we focus on the single-timescale asynchronous stochastic approximation. We state the main theorem before presenting the weak convergence results required for the proof. Section 4 examines the extension to a two-timescale asynchronous stochastic approximation process. Large parts of this section follow directly from the results in Section 3. In Section 5 we present an example of a learning algorithm for discounted reward Markov decision processes and obtain convergence results by applying the method shown in Section 4. This illustrates the ease in which this framework can be used. Finally, the paper concludes with a summary of the work. Throughout this paper many of the proofs are omitted from the main flow of text and are instead presented in an appendix.
Throughout this paper we use two main ideas from the stochastic approximation literature. The first relates to the work by Benaïm, Hofbauer and Sorin  on stochastic approximation with differential inclusions, and the second concerns the asynchronous stochastic approximation framework introduced by Borkar . We take the opportunity to review the pertinent features of their work in this section.
In what is to follow we use the standard concept of set multiplication: if is a set of matrices and are two closed, convex sets then let the multiplication of these sets be defined as,
Note that is also closed and convex. This definition is still used if either or both of the sets and are single valued. We also use the same concept when multiplying a constant by a set. That is, if is a constant then define
However, in this latter case we often drop the ‘’ notation for convenience.
2.1 Stochastic Approximation with Differential Inclusions
We begin by outlining the current convergence results for stochastic approximations with set-valued maps proved by Benaïm, Hofbauer and Sorin . These results are heavily used in Section 3, most notably to prove our main result. Initially we provide a definition which outlines the class of set-valued mean fields we are able to use for stochastic approximation. These criteria are taken directly from the original work on stochastic approximations with differential inclusions by Benaïm et al. .
Call a stochastic approximation map if it satisfies the following
is a closed set-valued map. That is,
is a closed set. Equivalently, is an upper semi-continuous set-valued map.
For all , is a non-empty, compact, convex subset of .
There exists a such that for all ,
Take as a stochastic approximation map; a typical differential inclusion is in the form,
and a solution to (2.1) is an absolutely continuous mapping such that and for almost every ,
The flow induced by (2.1) is defined by,
Definition 2.2 (Benaïm et al. ).
A continuous function is an asymptotic pseudo-trajectory to if
for any and where is a distance measure on .
Many important properties of a dynamical system and the asymptotic pseudo-trajectories of the systems are discussed by Benaïm, Hofbauer and Sorin . Most important for the work here is that an asymptotic pseudo-trajectory to (2.1) will behave in a similar manner to the solutions of the differential inclusion and hence the limiting behaviour will be closely related.
We conclude this section by considering a standard iterative process in the form of (1.1) where the mean field
is a stochastic approximation map. The following theorem states that under four assumptions a linear interpolation of(a function defined precisely in Section 3.1) is an asymptotic pseudo-trajectory to the differential inclusion (2.1). Hence the limiting behaviour of can be studied via this differential inclusion.
This is a slight modification to a result stated by Benaïm, Hofbauer and Sorin [4, Proposition 1.3] to include the terms. It is trivial to verify that this will not alter any of the asymptotic results of the original work.
2.2 Asynchronous Stochastic Approximations
Now we fully introduce the asynchronous stochastic approximation notation used. A typical asynchronous stochastic approximation such as those studied by Borkar  fits the following framework. If is the power set of all possible updating combinations in then let be the components of the iterative process updated at iteration . Using a counter for state ,
we consider processes in which no component, , in the asynchronous process needs to know the global counter, , merely its own counter, . Let , , and be the component of , , and respectively, for . We directly extend the notation used in (1.1) for an asynchronous stochastic approximation; let be a stochastic approximation map, then for let
Define the asynchronous step sizes, , and the relative step sizes, , to be
The asynchronous step sizes, , are random step sizes (in contrast with the deterministic terms) whilst the relative step sizes, are zero whenever the component of the iterative process is not updated. Clearly . By letting be the diagonal matrix of the terms we can express the previous asynchronous stochastic approximation (2.3) in the more concise form
This is a more familiar form for a stochastic approximation with a set-valued mean field. If is a stochastic approximation map in (1.1) then (2.4) differs from (1.1) only in that the step sizes in (2.4) are random and the addition of the coefficient. Instead of thinking of as a coefficient of the step sizes we combine it with the mean field. Convergence of the error term, , will be unaffected and, under a set of assumptions in Section 3.2, the noise term will still satisfy the Kushner and Clark noise condition (2.2). Combining the term and the mean field term into a single set provides an intuitive method of rephrasing the stochastic approximation and leads to a set-valued mean field.
Proceeding with this intuition is not immediately straightforward since is time varying and can be zero infinitely often, in which case the mean field could be zero even when the original update term, , is not. This would mean the limiting behaviour of the differential inclusion in the asynchronous stochastic approximation could be different to the synchronous case, where ultimately we wish to say that the two behave in the same manner in the limit. To avoid this scenario we follow Borkar  and consider the weak limit of the interpolations of , which will always be bounded away from zero under some verifiable assumptions, given in Section 3.2.
3 Asynchronous SA with Differential Inclusions
We begin by presenting the main result of this paper which concerns the limiting behaviour of the asynchronous stochastic approximation in (2.4), before outlining the results required to prove this in the remainder of the section.
3.1 Main Result
Assume that is a stochastic approximation map and for all define by its component parts, , , such that
Notice that if then we can select any . Then we can write the iterative process in (2.4) as
Now by letting and we get,
For general let
If is Lipschitz continuous direct comparisons can be made between the mean field, , and the analogous mean field from (1.3) which is used by Borkar  and Konda and Borkar . This provides the key insight into the new approach we take. Under the assumptions used in Section 3.2 the equivalent values almost surely lie in . By combining this with we produce a differential inclusion which is more straightforward to study than a non-autonomous differential equation and naturally fits the stochastic approximation framework of Benaïm, Hofbauer and Sorin . In addition, this idea naturally lends itself to examining a similar process for a set-valued mean field as we proceed to do in this paper.
Let , be the timescale for the asynchronous updates. To allow this process to be analysed in continuous time consider an interpolated version of the stochastic approximation (3.6) so that this process can be considered in continuous time,
Under the assumptions (A1)-(A5), presented in Section 3.2, we show in Section 3.3 that a sequence, , can be defined such that and satisfy the Kushner and Clark noise condition in (2.2). By invoking Theorem 2.3 we obtain our main result, which is proved in Section 3.4.
Under the assumptions (A1)-(A5), with probability 1,
Under the assumptions (A1)-(A5), with probability 1,is an asymptotic pseudo-trajectory to the differential inclusion,
Throughout this section we study the convergence properties of the iterative process (2.4). We make reference to the following assumptions, (A1)-(A5), all of which are either standard requirements for a stochastic approximation or can be verified prior to running the asynchronous stochastic approximation process. This is in contrast with the previous work on asynchronous stochastic approximations by Borkar  and Konda and Borkar .
For a compact set, , for all .
is a bounded sequence such that as .
Let satisfy the following criteria,
and as ,
For , , where , means the “integer part of”. In addition, for all , .
is a stochastic approximation map.
(A1)(a) is a slight strengthening of the standard stochastic approximation boundedness assumption; however this is still a relatively mild condition. Methods to ensure that it is satisfied are discussed elsewhere, for example ,  or . A basic restriction is placed on in (A1)(b); in this form the sequence does not affect the asymptotic behaviour of the process. (A2)(a) is a standard assumption required for stochastic approximation, and (A2)(b) is a mild technical condition required to deal with the asynchronicity, which is also used by Borkar . We have dropped the additional restriction on the step-sizes used by Borkar which severely restricts the possible choices of . (A3) ensures that we can use the convergence results presented in Theorem 2.3 and is a standard assumption for stochastic approximations with a set-valued mean field.
Define as the set of all the possible combinations which have positive probability of occurring. As an example, if every element of gets updated and it is known that is a singleton for each , then .
Let be a sigma algebra containing all the information up to and including the iteration. That is .
For all , ,
For all the transition probabilities form an aperiodic, irreducible, positive recurrent Markov chain over and for all there exists an such that .
The map is Lipschitz continuous.
(A4)(a) assumes that the transitions between the updated elements in are part of a controlled Markov chain. (A4)(b) is a straightforward assumption on this controlled Markov chain which can be verified prior to implementation which allows us to negate the need for some of the original technical assumptions made by Konda and Borkar . In this previous work Konda and Borkar assume that every state is updated at a comparable rate in the limit which cannot directly be verified prior to running the process. (A4)(c) is a condition which is required later to use a result from Ma et al.  on the convergence of stochastic approximation with Markovian Noise.
Take independent of for and independent of given for all . Let . Then there exists a positive such that for all ,
for each .
We say that (A5) holds if either (A5)(a) or (A5)(b) is true.
An assumption similar to (A5) is used by Benaïm, Hofbauer and Sorin  to verify a condition for noise convergence and is similar to that used by Kushner and Clark . We use this assumption only to show the noise term still satisfies the Kushner-Clark condition with the convergence given in Lemma 3.3; the proof is presented via two lemmas in Appendix A.2.
Assume that (A2)(b), (A4) and (A5) hold. Then with probability 1, for all ,
where , and .
3.3 Weak Convergence of Asynchronous Updates
As discussed in the introduction, the key issue with asynchronous stochastic approximations is how to handle the interaction of the relative step sizes and the mean field. It is important to be able to bound the limit of the relative step sizes, , away from zero for all in order to produce an asynchronous stochastic approximation mean field, which will behave similarly to the synchronous mean field, . However the relative step size of a state is zero whenever that state is not updated, hence it is not immediately clear that this is even possible. Despite this it is sufficient that for any an ‘average’ of over length in the continuous time interpolation converges to a value which is bounded away from zero. In this section we prove that under (A2)(b) and (A4) this is indeed the case.
For the space is the set of measurable functions such that,
for all . Hence is a space of trajectories. This means that for any map defined on convergence to a limit point will be in the weak sense, along a subsequence. That is a sequence of maps such that for all is said to possess a limit point if for fixed there exists a subsequence , such that for any ,
Now we extend the relative step sizes, , to continuous time; for all let for and let . For all and define .
Under (A2)(b) and (A4), for all and for any , converges along a subsequence to a limit point such that for some and any ,
See Appendix A.3. ∎
For any let be a limit point of in , then under (A2)(b) and (A4) there exists an such that for all and any such that ,
See Appendix A.3. ∎
We now expand upon the discussions in Sections 2.2 and 3.1 on producing a sequence of matrices . In order to use the differential inclusions framework described in Section 2.1 we need to define a sequence of diagonal matrices, with diagonal entries which are always in the set , for some , and such that the terms converge to the same limit as the terms of . Recall that is a diagonal matrix containing the terms.
Fix taken from Lemma 3.4 and define a new function such that
For all let . Corollary 3.5 shows that, with respect to the topology of , in the limit for almost every and similarly for all . From this it is clear and have the same limit point in . That is, if is a limit point of then it is also a limit point of . Hence for any there exists a subsequence such that for ,
However, the key interest here is in the convergence of and . Following the reasoning of Borkar  and Konda and Borkar , it does not matter whether and converge directly or via a subsequence as this does not affect the convergence of the continuous processes and . Hence we can say that converge weakly to a limit point , or equivalently, if is any bounded, continuous function then for all ,
Define to be the diagonal matrix of the terms and let .
Almost surely under assumptions (A2)(b) and (A4),
3.4 Proof of Theorem 3.1
Fix . Then,
Using Lemma 3.3 and Lemma 3.6 immediately gives that (3.12) and (3.13) converge to zero a.s., and hence this verifies that property holds. Assumption (A1)(a) directly gives that holds. Lastly, it is straightforward to verify that, under (A1)-(A5), is a stochastic approximation map which verifies condition , and (A1)(b) is equivalent to .
4 Two-timescale Asynchronous Stochastic Approximation
A useful extension of standard stochastic approximations is to two-timescales. This concept was originally introduced by Borkar  and has later been used by Leslie and Collins  for multiple timescales and Konda and Borkar  for two-timescales asynchronous stochastic approximation. If we have a coupled pair of stochastic approximations where one system can be seen to update more aggressively than the other then the aggressive process is always fully adjusted to the value of the other process. This is all controlled through the user’s choice of step sizes in the stochastic approximation. The main result in this section is Corollary 4.8, which comes from combining Theorem 3.1 with the previous work of Konda and Borkar .
In what is to follow we consider the extension of Theorem 3.1 to the two-timescales setting, with updates and on different timescales. Let be the set of individual elements of the process as in Section 3, and define similarly for the process. Let and so that for all , and . As in Section 3 let be the set containing all combinations of elements in which have a positive probability of being part of the asynchronous update, and define in the same manner for the process. At iteration let and be the updated components of each timescale respectively. Let each component of the two processes have a counter for the number of times it has been selected to be updated defined by,
Here is as in Section 3 and has an analogous definition for the process. Let , be martingale noise processes defined on and respectively, and , as similarly defined on and respectively. Let be component of and , and similarly let be component of and . As in the previous sections , and now , are positive, deceasing sequences of step sizes. Similar restrictions to those in (A2) will be placed on , with an additional requirement for the two-timescale arguments to be valid; this will be made precise in Section 4.2. Finally, and are set-valued maps, where is the value of and similarly for . For all and consider the following coupled process,
Notice that the only change to the first process from Sections 2 and 3 is that the mean field now depends on as well as . It follows that the asynchronous and relative step sizes retain the same form. Recall these definitions and extend them for the process:
Finally, define the two timescales; let , , , . The division of time on the ‘slow’ timescale is given by the increments and similarly for on the ‘fast’ timescale. In a similar manner to the previous sections let , and .
We state the assumptions (B1)-(B6) used for the convergence results of the two-timescale algorithm (4.2). These are exactly analogous to (A1)-(A5) and are simply extended to accommodate the two-timescales framework. The exceptions to this are (B2)(c) and (B6) and the slight adaptations to (B3), which are in line with those used by Borkar  and Konda and Borkar . In (B4) we have produced a single combined Markov chain instead of one for each of the and processes to present a clearer assumption.
For compact sets, , , , for all .
and are bounded sequences such that as .
The following must be true for and
and as ,
For , . In addition, for all , . .
For all , is upper semi-continuous and for all , is a stochastic approximation map.
For all , is a stochastic approximation map
The first and second assumptions are direct extensions of (A1) and (A2) to two-timescales with the addition of (B2)(c) which is a standard two-timescale assumption used by Borkar . Condition (B3)(a) is similar to (A3) for the ‘slow’ timescale, however this must hold for all values of the ‘fast’ timescale. (B3)(b) is a similar condition for the ‘fast’ timescale.
Define such that if and then if and only if and have a positive probability of occurring simultaneously (at the same iteration). This means that is the combination of elements across which have positive probability of being updated at any particular iteration. At iteration is taken to be the updated components across and . In addition, let and be a sigma algebra containing all the information up to and including the iteration. That is