DeepAI AI Chat
Log In Sign Up

Learning Robust Options

by   Daniel J. Mankowitz, et al.

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally abstract actions, in the framework of options. We present a Robust Options Policy Iteration (ROPI) algorithm with convergence guarantees, which learns options that are robust to model uncertainty. We utilize ROPI to learn robust options with the Robust Options Deep Q Network (RO-DQN) that solves multiple tasks and mitigates model misspecification due to model uncertainty. We present experimental results which suggest that policy iteration with linear features may have an inherent form of robustness when using coarse feature representations. In addition, we present experimental results which demonstrate that robustness helps policy iteration implemented on top of deep neural networks to generalize over a much broader range of dynamics than non-robust policy iteration.


page 1

page 2

page 3

page 4


Learning Robust Options by Conditional Value at Risk Optimization

Options are generally learned by using an inaccurate environment model (...

Policy Optimization with Robustness Certificates

We present a policy optimization framework in which the learned policy c...

Temporally Extended Successor Representations

We present a temporally extended variation of the successor representati...

Finding Options that Minimize Planning Time

While adding temporally abstract actions, or options, to an agent's acti...

Successor Options: An Option Discovery Framework for Reinforcement Learning

The options framework in reinforcement learning models the notion of a s...

Situationally Aware Options

Hierarchical abstractions, also known as options -- a type of temporally...

Provably efficient reconstruction of policy networks

Recent research has shown that learning poli-cies parametrized by large ...