DeepAI
Log In Sign Up

Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization

11/17/2020
by   Sreejith Balakrishnan, et al.
7

The problem of inverse reinforcement learning (IRL) is relevant to a variety of tasks including value alignment and robot learning from demonstration. Despite significant algorithmic contributions in recent years, IRL remains an ill-posed problem at its core; multiple reward functions coincide with the observed behavior and the actual reward function is not identifiable without prior knowledge or supplementary information. This paper presents an IRL framework called Bayesian optimization-IRL (BO-IRL) which identifies multiple solutions that are consistent with the expert demonstrations by efficiently exploring the reward function space. BO-IRL achieves this by utilizing Bayesian Optimization along with our newly proposed kernel that (a) projects the parameters of policy invariant reward functions to a single point in a latent space and (b) ensures nearby points in the latent space correspond to reward functions yielding similar likelihoods. This projection allows the use of standard stationary kernels in the latent space to capture the correlations present across the reward function space. Empirical results on synthetic and real-world environments (model-free and model-based) show that BO-IRL discovers multiple reward functions while minimizing the number of expensive exact policy optimizations.

READ FULL TEXT

page 2

page 4

page 6

page 7

page 8

page 9

page 19

06/07/2021

Identifiability in inverse reinforcement learning

Inverse reinforcement learning attempts to reconstruct the reward functi...
11/09/2020

Reward Conditioned Neural Movement Primitives for Population Based Variational Policy Optimization

The aim of this paper is to study the reward based policy exploration pr...
06/01/2022

Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble

Inverse reinforcement learning (IRL) recovers the underlying reward func...
03/16/2022

Learning Representation for Bayesian Optimization with Collision-free Regularization

Bayesian optimization has been challenged by datasets with large-scale, ...
09/09/2021

OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching

Inverse Reinforcement Learning (IRL) is attractive in scenarios where re...
09/10/2021

Potential-based Reward Shaping in Sokoban

Learning to solve sparse-reward reinforcement learning problems is diffi...
07/07/2021

Learning Time-Invariant Reward Functions through Model-Based Inverse Reinforcement Learning

Inverse reinforcement learning is a paradigm motivated by the goal of le...

Code Repositories

BOIRL

Accompanying code of BO-IRL published in Neurips 2020


view repo