Conditioning Hierarchical Reinforcement Learning on Flexible Constraints

02/21/2023
by   Yuxiao Lu, et al.
0

Safety in goal directed Reinforcement Learning (RL) settings has typically been handled through constraints over trajectories and have demonstrated good performance in primarily short horizon tasks (goal is not too far away). In this paper, we are specifically interested in the problem of solving temporally extended decision making problems such as (1) robots that have to clean different areas in a house while avoiding slippery and unsafe areas (e.g., stairs) and retaining enough charge to move to a charging dock; (2) autonomous electric vehicles that have to reach a far away destination while having to optimize charging locations along the way; in the presence of complex safety constraints. Our key contribution is a (safety) Constrained Planning with Reinforcement Learning (CoP-RL) mechanism that combines a high-level constrained planning agent (which computes a reward maximizing path from a given start to a far away goal state while satisfying cost constraints) with a low-level goal conditioned RL agent (which estimates cost and reward values to move between nearby states). A major advantage of CoP-RL is that it can handle constraints on the cost value distribution (e.g., on Conditional Value at Risk, CVaR, and also on expected value). We perform extensive experiments with different types of safety constraints to demonstrate the utility of our approach over leading best approaches in constrained and hierarchical RL.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
01/27/2023

Solving Constrained Reinforcement Learning through Augmented State and Reward Penalties

Constrained Reinforcement Learning has been employed to enforce safety c...
research
03/23/2021

Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework

Reinforcement learning (RL) agents with pre-specified reward functions c...
research
07/12/2021

A Simple Reward-free Approach to Constrained Reinforcement Learning

In constrained reinforcement learning (RL), a learning agent seeks to no...
research
07/29/2021

Lyapunov-based uncertainty-aware safe reinforcement learning

Reinforcement learning (RL) has shown a promising performance in learnin...
research
06/21/2019

Reinforcement Learning with Convex Constraints

In standard reinforcement learning (RL), a learning agent seeks to optim...
research
09/21/2018

Constrained Exploration and Recovery from Experience Shaping

We consider the problem of reinforcement learning under safety requireme...
research
10/10/2016

Situational Awareness by Risk-Conscious Skills

Hierarchical Reinforcement Learning has been previously shown to speed u...

Please sign up or login with your details

Forgot password? Click here to reset