We deal with the following optimal control problem with state constraints.
Let be an open bounded subset of , we consider the following system of controlled differential equations
Here and the control belongs to the set of admissible control functions , typically the set of measurable control functions with values in , a compact subset of . We impose a state constraint on (1.1) requiring that the state remains in for all . As a consequence, we will consider admissible (with respect to the state constraint) only control functions guaranteeing that the corresponding trajectory never leaves . We will denote by this subset of , then for any
where denotes the solution trajectory starting at .
Given a cost functional , the problem is to determine the value function
and possibly an optimal control (at least approximate). We will use the notion of viscosity solution to the Hamilton-Jacobi-Bellman equation, introduced by Crandall and Lions in  (see also ), and in particular its extension to the notion of constrained viscosity solution given by Soner  in order to treat problems with state constraints. This definition combines the standard definition on with an appropriate inequality to be satisfied on (see also  for further developments of this notion). This condition can be applied to other hamiltonians coming from various optimal control problems.
In the first part of this paper we will consider for simplicity only convex constraints for the infinite horizon problem. As we will see later, similar arguments can be applied to other control problems and non convex constraints although our result does not cover this case. Dealing with the infinite horizon problem, Soner has shown that, whenever the value function is continuous, it is the unique constrained viscosity solution of the following Hamilton-Jacobi-Bellman equation
where is a positive real parameter, the discount rate.
We should also mention that several results have been obtained for the existence of trajectories of (1.1) satisfying the state constraints (the so called viable trajectories) using the theory of multivalued differential inclusions (see Aubin–Cellina ). Essentially, we know that a viable solution exists if for any there exists at least one control such that the corresponding velocity belongs to the tangent cone to at (see Section 2 for a precise result in the convex case due to Haddad ). We recall that several extensions have been proposed for more general constraints using appropriate definitions of tangent cones (see  for an extensive presentation of this theory). These results gives necessary and sufficient conditions for the existence of viable trajectories so that one can determine the minimum set of assumptions guaranteeing that the optimal control problem can have a solution.
Several papers have been written on optimal control problems with state constraints starting from
the seminal paper . We can mention the interesting contributions by Ishii-Koike , Bokanowski-Zidani and co-authors [8, 4] and Motta  for different ways to deal with state constraints still having a well posed problem. We also mention the recent contribution by Kim, Tran and Tu  dealing with constrained problems on nested domains.
From the point of view of the numerical approximation a classical grid approach has been developed by Camilli-Falcone  and Bokanowski-Forcadel-Zidani .
In this respect they represent an extension to constrained problems of the numerical approximation developed by Capuzzo Dolcetta , Falcone  (see also the survey paper  and the book  for other numerical schemes related to optimal control problems via the Dynamic Programming approach). We end this short presentation mentioning that also viability tools have been applied to construct numerical methods for optimal control problems with state constraints, see e.g. .
Although convergence results are available for every dimension, numerical methods based on fixed space grids are difficult to apply for high-dimensional problems since they suffer for the well known ’curse of dimensionality’. This is why a renewed effort has been made in recent years to find other methods which can tackle high-dimensional optimal control problems. A list of references for other approaches dealing with high-dimensional problems is presented and discussed in.
In the first part of this paper we propose a novel formulation of the time discrete infinite horizon problem that is close to the formulation presented in 
for the continuous problem and we prove a convergence result for the value function for a convex constraint. The proof is based on a mixture of tools coming from multivalued analysis and viscosity solutions. As we said, we want to develop a fast approximation scheme for the value function using the characterization in terms of the Hamilton-Jacobi-Bellman equation. To this end we will also use some tools of the viability theory to establish a precise convergence result (see Sections 2 and 3). The scheme is build having in mind a ”heuristic” representation of the value function which comes out coupling the viability results with standard dynamic programming arguments. Although we present our convergence result for the infinite horizon problem focusing on the treatment of boundary conditions for the stationary problem, similar arguments can be applied also to other optimal control problems such as the finite horizon and the optimal stopping problem (see Remark3.1).
The second part of the paper is devoted to the construction of an efficient algorithm for a time discrete approximation of the value function that avoids the construction of a fixed grid in space and allows to apply the dynamic programming principle on a Tree Structure (TS), the main results on this approach have been presented [1, 2, 3]. Our contribution here is the extension of the TS Algorithm (TSA) to problems with state constraints and the feedback reconstruction using scattered data interpolation.
The outline of our paper is the following.
In Section 2 we introduce our basic assumptions and state some previous results about the characterization of the value function in terms of the Hamilton-Jacobi-Bellman equation. We present some results in the viability theory that are useful for the problem at hand and discuss a different way to write the equation. We continue introducing our time discretization and prove some properties of the discrete value function showing that the discretized equation (2.27) has a unique solution . We establish our main convergence result for the infinite horizon problem in Section 3 proving that converges to the value uniformly on the constraint , provided the state constraint is convex. In Section 4 we introduce the TSA for the finite horizon problem with state constraints and discuss some of its features. Finally, the last section is devoted to numerical experiments where we show the TSA is faster than the classical grid approximation. Moreover, some of the tests show that the method can also solve problems with non convex space constraints, overcoming the limits of our convergence result.
2. The infinite horizon problem with state constraints.
We will denote by the position at time of the solution trajectory of (1.1) corresponding to the control . Whenever this will be possible without ambiguity we adopt the simplified notations or instead of . The cost functional related to the infinite horizon problem is given by
where is the running cost. As we said in the introduction we want to minimize with respect to the controls in so we need at least the assumption that
It is important to note that in general is not continuous on even when (2.2) is satisfied. This is due to the structure of the multivalued map .
Soner has shown in  that the value function is continuous (and then uniformly continuous) on if the following boundary condition on the vectorfield is satisfied
where is the outward normal to at the point .
We will make the following assumptions:
A0. is a bounded, open convex subset of ;
A1. , compact;
A2. , is continuous and
A3. is continuous and .
Clearly, there exist two positive constants , such that
for any . Notice that under the above assumptions the value function is bounded in by as can be easily checked.
Using the Dynamic Programming Principle, Soner has shown that is the unique viscosity solution of (1.4). This means that satisfies
and the above inequalities should be understood in the viscosity sense (see  for the precise definition). A function satisfying (2.5) (respectively (2.6)) is be called a constrained viscosity subsolution (respectively supersolution) of .
Necessary and sufficient conditions.
Condition (2.3) is known to be only a sufficient condition for the existence of trajectories living in . However, necessary and sufficient condition for the existence of solutions in have been extensively studied in viability theory (see ).
Let be an open convex subset of . A trajectory is called viable when
Let be a multivalued map which is lower semicontinuous and has compact convex images (we refer to  for the theory and the definitions related to multivalued maps). Let us define the tangent cone to a compact convex set at the point , as
A result due to Haddad  shows that the condition
is necessary and sufficient to have viable trajectories in for the multivalued Cauchy problem
This result has been also extended to more general sets (also non convex) introducing more general tangent cones (see  for a general presentation of the viability theory).
2.1. The time-discrete scheme for the constrained problem
In order to build a discretization of (1.4) we start using the standard discretization in time of (1.1), (2.1). We fix a positive parameter , the time step, and consider the following approximation scheme for (1.1) and (2.1)
where , and .
For every the corresponding value function is
The above definition is meaningful only provided there exists a step such that . We look for conditions guaranteeing the existence of viable discrete trajectories. Let us introduce the multivalued map
representing the subset of admissible (i.e. satisfying the constraint) controls for the discrete dynamics. Clearly if and only if for any . Due to the regularity assumptions on , is open and is bounded since is always contained in .
where is the interior of the tangent cone to at , i.e.
In fact, if , then , which implies . Note that is not empty since .
The dependence of from is such that
In fact, if then and (2.19) follows by the convexity of .
The following proposition gives necessary and sufficient conditions for the existence of a time step , such that for any and therefore guarantees .
Let be an open bounded convex subset of . Assume that is continuous. Then, there exists such that
if and only if the following assumption holds,
Moreover, (2.22) is also valid for every positive by the convexity of . Since is bounded, (2.22) is satisfied for any and will not depend on . By the continuity of there will be an and a neighbourhood of such that
at least for .
Note that when all the directions are allowed provided is sufficiently small and the restrictions apply only for . The family is an open covering of from which we can extract a finite covering . We will have then for any setting . ∎
Under the same assumptions of Proposition 2.1 there exists such that
Let us remark that condition (2.21) is more general than the boundary condition (2.3) since does not require the regularity of . In fact for a closed convex subset , we can define the normal cone to at as
When the tangent cone will be the whole space and the normal cone will be empty. For these are real convex cones. Now assume that
has a regular boundary, the tangent cone is an hyperplane and the normal cone is reduced to, . Then (2.3) implies that
The proof of the following result can be obtained by standard arguments so it will not be given here (see  for details).
for any and , where is the trajectory with the the con sequence .
We will refer to (2.26) as the Discrete Dynamic Programming Principle (DDPP). For ,it gives the following discrete version of
Note that for the constrained problem the infimum is taken on the variable control set . In the next section we will see how to handle this dependency.
Let . Then, for any there exists a unique solution of .
Moreover, the following estimates hold true:
. Moreover, the following estimates hold true:
where is the modulus of continuity of .
The solution of (2.27) is the fixed point of the operator
Let and . By (3.16) for any , there exists such that
Reversing the role of e we get
Note that if is such that , we have
Then, recalling the definition of , implies
We can conclude that, for any , is a contraction mapping in so that there will be a unique bounded solution of .
Now we prove that . We show first that if then . Let , for any there exists which satisfies (2.31). Since is open and is continuous, there will be a neighbourhood of such that
then and we have
By the arbitrariness of , we conclude
Since and are arbitrary, we can determine such that
whenever . By (2.36) we get
and by the uniform continuity of
Since , the constant is strictly positive and one can easily check that
for any such that . Then the recursion sequence
starting at a such that and converges to the unique solution of (2.27). By (3.19) satisfies (3.15). Since is decreasing in , we get
and we can conclude the proof of the theorem. ∎
3. A convergence result
The main result of this section is that the solution of the discrete–time equation converges to . In order to prove this convergence we need some preliminary lemmas on the regularity of with respect to .
For any fixed , the multivalued map , , is lower semicontinuous in the sense of multivalued maps
Let and . Recalling the definition of l.s.c. maps ( (see ), we have to show that there exists a neighborhood of such that
where is the unit ball in . Since is open and is continuous, we can determine and such that
Let and consider the sequence of sets , . Let per , then
Let , we have to prove that , i.e. that for any , there exists an index such that
Since and is bounded, there exists such that
By a compactness argument we can choose independently of . The continuity of then implies that there exists such that
Moreover, there exists an index such that
then by the convexity of also
so . To end the proof it suffices to choose such that (3.5) holds. ∎
Using the above propositions, we can prove our main convergence result.
Let , then uniformly in , for .
Since is uniformly bounded and equicontinuous, by the Ascoli–Arzelà theorem, there exist for and a function such that
for sufficiently small. Then, for large enough, there exists such that has a local maximum point at and converges to . Note that for any control the point belongs to , and for large enough it belongs to . The above remarks imply
Since , it follows that there exists such that by the above inequality we get
Let be such that and . We can choose such that for any
and by (3) we have
Let and , we define the real function ,
where (note that is continuous in both variables). Let us define
and converges to , for any , there exists such that (4.11) and (4.12) hold true. By the lower semicontinuity of and the arbitrariness of we get
The inequality (3.15) is verified for any . We show that
It suffices to prove that
In fact, for any , we can find a sequence , such that for and
then passing to the limit for , by the continuity of we have
Proposition 3.1 implies that
b) Now we prove that is a viscosity supersolution of (1.4) in .
Let and , be a strict maximum point for in . We can use the same arguments that we used for (4.9) in the first part of this theorem (just replace by ), so we get
By (3.16) for any there exists such that