Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

10/20/2020 ∙ by Daniel J. Mankowitz, et al. ∙ 0 ∙ share

Many real-world physical control systems are required to satisfy constraints upon deployment. Furthermore, real-world systems are often subject to effects such as non-stationarity, wear-and-tear, uncalibrated sensors and so on. Such effects effectively perturb the system dynamics and can cause a policy trained successfully in one domain to perform poorly when deployed to a perturbed version of the same domain. This can affect a policy's ability to maximize future rewards as well as the extent to which it satisfies constraints. We refer to this as constrained model misspecification. We present an algorithm with theoretical guarantees that mitigates this form of misspecification, and showcase its performance in multiple Mujoco tasks from the Real World Reinforcement Learning (RWRL) suite.

READ FULL TEXT

Constrained episodic reinforcement learning in concave-convex and knapsack settings

We propose an algorithm for tabular episodic reinforcement learning with...
06/09/2020 ∙ by Kianté Brantley, et al. ∙ 8 ∙ share

read it
SafeLife 1.0: Exploring Side Effects in Complex Environments

We present SafeLife, a publicly available reinforcement learning environ...
12/03/2019 ∙ by Carroll L. Wainwright, et al. ∙ 0 ∙ share

read it
An empirical investigation of the challenges of real-world reinforcement learning

Reinforcement learning (RL) has proven its worth in a series of artifici...
03/24/2020 ∙ by Gabriel Dulac-Arnold, et al. ∙ 9 ∙ share

read it
Self-Supervised Policy Adaptation during Deployment

In most real world scenarios, a policy trained by reinforcement learning...
07/08/2020 ∙ by Nicklas Hansen, et al. ∙ 4 ∙ share

read it
Value constrained model-free continuous control

The naive application of Reinforcement Learning algorithms to continuous...
02/12/2019 ∙ by Steven Bohez, et al. ∙ 0 ∙ share

read it
OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World

While deep reinforcement learning techniques have recently produced cons...
09/22/2017 ∙ by Tu-Hoa Pham, et al. ∙ 0 ∙ share

read it
Batch Policy Learning under Constraints

When learning policies for real-world domains, two important questions a...
03/20/2019 ∙ by Hoang M. Le, et al. ∙ 26 ∙ share

read it

Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification

Authors

Constrained episodic reinforcement learning in concave-convex and knapsack settings

SafeLife 1.0: Exploring Side Effects in Complex Environments

An empirical investigation of the challenges of real-world reinforcement learning

Self-Supervised Policy Adaptation during Deployment

Value constrained model-free continuous control

OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World

Batch Policy Learning under Constraints

This week in AI