Reconnaissance and Planning algorithm for constrained MDP

09/20/2019
by   Shin-ichi Maeda, et al.
5

Practical reinforcement learning problems are often formulated as constrained Markov decision process (CMDP) problems, in which the agent has to maximize the expected return while satisfying a set of prescribed safety constraints. In this study, we propose a novel simulator-based method to approximately solve a CMDP problem without making any compromise on the safety constraints. We achieve this by decomposing the CMDP into a pair of MDPs; reconnaissance MDP and planning MDP. The purpose of reconnaissance MDP is to evaluate the set of actions that are safe, and the purpose of planning MDP is to maximize the return while using the actions authorized by reconnaissance MDP. RMDP can define a set of safe policies for any given set of safety constraint, and this set of safe policies can be used to solve another CMDP problem with different reward. Our method is not only computationally less demanding than the previous simulator-based approaches to CMDP, but also capable of finding a competitive reward-seeking policy in a high dimensional environment, including those involving multiple moving obstacles.

READ FULL TEXT

page 3

page 4

page 5

page 6

page 11

page 12

page 13

page 14

research
06/16/2021

Safe Reinforcement Learning Using Advantage-Based Intervention

Many sequential decision problems involve finding a policy that maximize...
research
01/02/2022

Reinforcement Learning for Task Specifications with Action-Constraints

In this paper, we use concepts from supervisory control theory of discre...
research
04/06/2023

Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effects

In safe MDP planning, a cost function based on the current state and act...
research
07/09/2021

Safe Exploration by Solving Early Terminated MDP

Safe exploration is crucial for the real-world application of reinforcem...
research
10/02/2022

Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation

We address the problem of safe reinforcement learning from pixel observa...
research
02/10/2023

Hierarchical Motion Planning under Probabilistic Temporal Tasks and Safe-Return Constraints

Safety is crucial for robotic missions within an uncertain environment. ...
research
10/02/2019

Formal Language Constraints for Markov Decision Processes

In order to satisfy safety conditions, a reinforcement learned (RL) agen...

Please sign up or login with your details

Forgot password? Click here to reset