DeepAI AI Chat
Log In Sign Up

Distributional Offline Continuous-Time Reinforcement Learning with Neural Physics-Informed PDEs (SciPhy RL for DOCTR-L)

by   Igor Halperin, et al.

This paper addresses distributional offline continuous-time reinforcement learning (DOCTR-L) with stochastic policies for high-dimensional optimal control. A soft distributional version of the classical Hamilton-Jacobi-Bellman (HJB) equation is given by a semilinear partial differential equation (PDE). This `soft HJB equation' can be learned from offline data without assuming that the latter correspond to a previous optimal or near-optimal policy. A data-driven solution of the soft HJB equation uses methods of Neural PDEs and Physics-Informed Neural Networks developed in the field of Scientific Machine Learning (SciML). The suggested approach, dubbed `SciPhy RL', thus reduces DOCTR-L to solving neural PDEs from data. Our algorithm called Deep DOCTR-L converts offline high-dimensional data into an optimal policy in one step by reducing it to supervised learning, instead of relying on value iteration or policy iteration methods. The method enables a computable approach to the quality control of obtained policies in terms of both their expected returns and uncertainties about their values.


page 1

page 2

page 3

page 4


Continuous-Time Fitted Value Iteration for Robust Policies

Solving the Hamilton-Jacobi-Bellman equation is important in many domain...

Dynamically optimal treatment allocation using Reinforcement Learning

Consider a situation wherein a stream of individuals arrive sequentially...

Q-learning for Optimal Control of Continuous-time Systems

In this paper, two Q-learning (QL) methods are proposed and their conver...

Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning

Continuous-time reinforcement learning offers an appealing formalism for...

Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control

Recent work has shown that reinforcement learning (RL) is a promising ap...

Optimal Reinforcement Learning for Gaussian Systems

The exploration-exploitation trade-off is among the central challenges o...

Off-policy reinforcement learning for H_∞ control design

The H_∞ control design problem is considered for nonlinear systems with ...