Learning Transferable Domain Priors for Safe Exploration in Reinforcement Learning

Prior access to domain knowledge could significantly improve the performance of a reinforcement learning agent. In particular, it could help agents avoid potentially catastrophic exploratory actions, which would otherwise have to be experienced during learning. In this work, we identify consistently undesirable actions in a set of previously learned tasks, and use pseudo-rewards associated with them to learn a prior policy. In addition to enabling safe exploratory behaviors in subsequent tasks in the domain, these priors are transferable to similar environments, and can be learned off-policy and in parallel with the learning of other tasks in the domain. We compare our approach to established, state-of-the-art algorithms in a grid-world navigation environment, and demonstrate that it exhibits a superior performance with respect to avoiding unsafe actions while learning to perform arbitrary tasks in the domain. We also present some theoretical analysis to support these results, and discuss the implications and some alternative formulations of this approach, which could also be useful to accelerate learning in certain scenarios.


page 10

page 11

page 14

page 17

page 18


TempoRL: Temporal Priors for Exploration in Off-Policy Reinforcement Learning

Efficient exploration is a crucial challenge in deep reinforcement learn...

Identification and Off-Policy Learning of Multiple Objectives Using Adaptive Clustering

In this work, we present a methodology that enables an agent to make eff...

Curiosity Killed the Cat and the Asymptotically Optimal Agent

Reinforcement learners are agents that learn to pick actions that lead t...

Safe Reinforcement Learning through Meta-learned Instincts

An important goal in reinforcement learning is to create agents that can...

Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistake

Agents that operate in an unknown environment are bound to make mistakes...

Self-Organizing Maps as a Storage and Transfer Mechanism in Reinforcement Learning

The idea of reusing information from previously learned tasks (source ta...

Towards real-world navigation with deep differentiable planners

We train embodied neural networks to plan and navigate unseen complex 3D...

Please sign up or login with your details

Forgot password? Click here to reset