The theory of the Hamiltonian system is known as one of the dominant tools for the description of dynamic phenomenons in the field of physics and economics . For example, in physics, the mechanical and electrical systems are usually represented as energy functions, which are at the same time Hamiltonian systems. Actually, the Hamiltonian system could reflect the laws of energy conservation and dissipation [1, 2].
A determined Hamiltonian system can be given as
where is a given real function called the Hamiltonian, and are partial derivatives of with respect to and , respectively. When considering a terminal condition for a given function , (1.1) becomes a boundary problem.
For more complex environments where the physical system can not be represented with deterministic form, the Hamiltonian system is usually combined with a stochastic process, and the form of a stochastic Hamiltonian system with boundary condition can be given as,
which is essentially a fully coupled forward-backward stochastic differential equation (FBSDE in short). Many research work have studied the solutions of FBSDEs and the eigenvalue of the Hamiltonian systems[3, 4, 5, 6, 7, 8, 9, 10, 11]. The significance of studying this kind of Hamiltonian system is that on the one hand it can be applied in solving the stochastic optimal control problems via the well-known stochastic maximum principle [12, 13]
; on the other hand, it helps to obtain the solutions of nonlinear partial differential equations (PDEs in short) according to the connection between the FBSDEs and the PDEs.
In most cases, it is difficult to obtain the explicit solution of the Hamiltonian system (1.2), thus numerical methods should be studied. As (1.2) is essentially a FBSDE, an intuitive way is to solve (1.2) from the perspective of FBSDEs. Therefore, numerical methods for solving the FBSDEs can be applied [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 8]
, such as the PDE methods, the probabilistic methods, etc. However, most of the traditional numerical methods can not deal with high-dimensional problems. Moreover, it is worth to point out that solving the fully coupled FBSDEs numerically has been a much more challenging problem than the general FBSDEs, even for low dimensional cases. Recently, with the application of the deep learning technique in a wide range of areas, numerical methods based on deep neural networks have been proposed to solve high-dimensional Backward Stochastic Differential Equations (BSDEs in short) and FBSDEs and achieved remarkable success. Among them, a breakthrough work was developed by[24, 25], the main idea is to reformulate the BSDE into a stochastic optimal control problem by rewriting the backward process into a forward form and taking the terminal error as the cost functional, then the solution of the BSDE is approximated by deep neural network. Other different deep learning algorithms are proposed to solve the BSDEs and related PDEs [26, 27, 28, 29], where they also focus on approximating the solution of the BSDE (or PDE) with the deep neural network. For solving coupled and fully coupled FBSDEs [30, 31] developed numerical algorithms which are also inspired by [24, 25].
In this paper, we propose a novel method to solve the Hamiltonian system (1.2) via deep learning. As equation (1.2) is at the same time a fully coupled FBSDE, this method is also suitable for solving fully coupled FBSDEs. However, different from the above mentioned deep learning methods which aim to solve the FBSDEs directly, we first look for the corresponding stochastic optimal control problem of the Hamiltonian system, such that the Hamiltonian system of the stochastic control problem is exactly what we need to solve. Then we approximate the optimal control with deep neural networks. In order to solve the optimal control problem, two different cases are considered which correspond to two different algorithms. The first algorithm (Algorithm 1) deals with the case where the function defined in (2.5) has an explicit form. For the case that cannot be expressed explicitly, the original control problem is transformed to a double objective optimization problem and we develop the second algorithm (Algorithm 2) to solve it. Finally, the numerical solutions of (1.2) are obtained by calculating the solution of the extended Hamiltonian system for the optimal control according to the stochastic maximum principle.
We also compare the results of our novel proposed algorithms with that of the algorithm mentioned as Algorithm 1 in our previous work  ( which is called the Deep FBSDE method here), which can be used to solve the Hamiltonian system from the view of the FBSDEs. Comparing with the Deep FBSDE method, our proposed algorithms have two advantages. The first advantage is that less numbers of iteration steps are required to achieve convergent results. When the Deep FBSDE method converges, it needs more iterations to achieve a convergent result. The second advantage is that our proposed algorithms have more stable convergences. For some Hamiltonian systems, the Deep FBSDE method is easier to diverge with the same piecewise decay learning rate as our two proposed algorithms. The details can be referred to the numerical results in section 4.
This paper is organized as follows. In section 2, we describe the Hamiltonian system that we aim to solve, and introduce its corresponding stochastic optimal control problem. In section 3, we introduce two schemes to solve the stochastic control problem according to whether the function defined in (2.5) has an explicit form, and then give the corresponding neural network architectures. The numerical results for different examples are shown in section 4, and a brief conclusion is made in section 5.
2 Problem formulation
In this section, we first describe a kind of stochastic Hamiltonian systems, then we show that solving this kind of stochastic Hamiltonian systems is equivalent to solving a stochastic optimal control problem.
2.1 The stochastic Hamiltonian system
be a filtered probability space, in whichis a -dimensional standard -Brownian motion; is the natural filtration generated by the Brownian motion Suppose that is complete, contains all the -null sets in and is right continuous.
For , define and . The space of all mean square-integrable -adapted and -valued processes will be denoted by , which is a Hilbert space with the norm
be a real function of , called a Hamiltonian and let
be a real function of . In our context, unless otherwise stated, we always assume that the Hamiltonian is strictly convex with respect to and .
Consider the following stochastic Hamiltonian system:
where , , are gradients of with respect to , , , respectively. It is worth to pointing out that the above system is essentially a special kind of FBSDEs.
For any and ,
there exists a constant , such that
there exists a constant , such that the following monotonic conditions hold.
Theorem 1 (Theorem 3.1 in ).
Recently, numerical algorithms for solving the BSDEs and FBSDEs with deep learning method [24, 25, 30, 31] have been proposed and demonstrated remarkable performance. The main idea is to reformulate the BSDE into a stochastic optimal control problem, where the solution of the BSDE is regarded as a control and approximated with deep neural network, and the terminal error is taken as the cost functional. Other different numerical algorithms have also been developed for solving the FBSDEs and the related PDEs [26, 27, 28, 29], which also regard the solution of the FBSDE ( or
) as a control and approximate it with appropriate loss function.
2.2 A novel method to solve the stochastic Hamiltonian system
As noted in the previous sections, the stochastic Hamiltonian system (2.3) is essentially a fully coupled FBSDE and can be solved through the methods for solving the FBSDEs. In this paper, we develop a novel method for solving the Hamiltonian system(2.3). Different from the above mentioned algorithms for solving the BSDEs or FBSDEs, our main idea is to find the corresponding stochastic optimal control problem of the stochastic Hamiltonian system, and then directly apply the deep learning method to solve the control problem.
Here is the Legendre-Fenchel transform of with respect to . Due to the differentiability and strict convexity of the Hamiltonian , is also differential and strictly convex with respect to and .
Consider the following control system,
and the cost functional is given as
where the controls and belong to and , respectively. The set of all admissible controls is denoted by . Any satisfying
is called an optimal control. The corresponding state trajectory is called an optimal trajectory and the corresponding triple is called an optimal triple.
We need the following assumption.
is continuously differentiable with respect to , , , and
for some given .
Let be a given real function and strictly convex with respect to and . The derivatives of and satisfy Assumption 1; satisfies Assumption 2. Suppose that is the optimal triple of the optimal control problem (2.6)-(2.7). Then uniquely solves the Hamiltonian system (2.3), where can be given as
and can be solved by this following BSDE
Under our assumptions, for any optimal triple of the optimal control problem (2.6)-(2.7), we have the following extended stochastic Hamiltonian system through the well-known stochastic maximum principle (SMP in short) ( e.g. Theorem 4.1 in ),
Because is strictly concave in , it yields that the maximum point of (2.13) is uniquely determined by due to the implicit function existence theorem:
and are differentiable functions. By (2.14),
which leads to
Thus, the derivatives of with respect to are
It can be verified that
The following proposition can help us to construct our algorithms in the next section.
By the definition of and the SMP,
Then, we have
The definition of shows that
where is the minimum point.
Notice that is strictly concave with respect to . If , then
In Theorem 2 and Proposition 3, we choose the stochastic optimal control problem (2.6)-(2.7), whose diffusion term and drift term of the state equation are simple and . In fact, to simplify the linear terms of and in the Hamiltonian , we can also choose other forms of and , such as and , which are linear with respect to and . In this cases, the transformations (2.5) and (2.14) also hold. We show an example in the numerical result, in subsection 4.2.
Besides, we can still solve the Hamiltonian system (2.3) even if the coefficients do not satisfy the monotonic conditions (Assumption 1 (ii)) in Theorem 2 and Proposition 3. For example, the articles [8, 9] studied the solvability of FBSDEs under relatively loose conditions. In this situation, as long as the optimal controls reach the optimal values, the Hamiltonian system (2.3) can be solved, however the solution is not necessarily unique.
3 Numerical method for solving Hamiltonian systems
In Section 2, we present the idea of the stochastic optimal control method to solve the Hamiltonian system (2.3). According to Theorem 2, we only need to find the optimal control triple of the stochastic control problem (2.6) and (2.7), then the solution can be obtained by taking the conditional expectation on the backward SDE of (2.10). Therefore, effective approximation method should be used to obtain the optimal triple of (2.6)-(2.7), especially for high dimensional cases.
Deep neural networks are usually used to approximate functions defined on finite-dimensional space, and the approximation relies on the composition of layers with simple functions. On the basis of the universal approximation theorem [33, 34], the neural networks have shown to be an effective tool and gained great successes in numerous practical applications. In this paper, inspired by , we simulate the stochastic optimal control problem (2.6)-(2.7) from a direct way with the deep neural network and develop two different numerical algorithms suitable for different cases.
Let be a partition of the time interval, of . Define and , where , for . We also denote
which is small enough. Then the Euler-Maruyama scheme of the state equation (2.6) can be written as
and the corresponding cost functional is given as
where represents the number of Monte Carlo samples.
We introduce a feedforward neural network of the form
is a positive integer specifying the depth of the neural network,
We approximate the controls with two different neural networks, which can be represented with (3.3) and denoted as and , respectively:
The two neural networks have the same input dimension but different output dimensions. Figure 1 shows an example of the whole network architecture for . In this paper, we use the common parameters of the neural networks for all the time points, i.e. a single network is developed for simulating each of the control, and the time point is regarded as an input of the neural network.
3.1 Case 1: the function has an explicit form
which is also the loss function we need to minimize in the whole neural network, and are the outputs of the two neural networks at time . Both the neural networks approximating contain one -dim input layer, three -dim hidden layers. The network of has an -dim output layer and that of has a -dim output layer. In order to simplify the representation, here we use to represent the training parameters for both of the neural networks.
To minimize the loss function (3.5
) and learn the optimal parameters, some basic optimization algorithms, such as stochastic gradient descent (SGD), AdaGrad, RMSProp, and Adam which are already implemented in TensorFlow can be used. In this paper, the Adam method is adopted as the optimizer.