Useful Policy Invariant Shaping from Arbitrary Advice

11/02/2020
by   Paniz Behboudian, et al.
0

Reinforcement learning is a powerful learning paradigm in which agents can learn to maximize sparse and delayed reward signals. Although RL has had many impressive successes in complex domains, learning can take hours, days, or even years of training data. A major challenge of contemporary RL research is to discover how to learn with less data. Previous work has shown that domain information can be successfully used to shape the reward; by adding additional reward information, the agent can learn with much less data. Furthermore, if the reward is constructed from a potential function, the optimal policy is guaranteed to be unaltered. While such potential-based reward shaping (PBRS) holds promise, it is limited by the need for a well-defined potential function. Ideally, we would like to be able to take arbitrary advice from a human or other agent and improve performance without affecting the optimal policy. The recently introduced dynamic potential based advice (DPBA) method tackles this challenge by admitting arbitrary advice from a human or other agent and improves performance without affecting the optimal policy. The main contribution of this paper is to expose, theoretically and empirically, a flaw in DPBA. Alternatively, to achieve the ideal goals, we present a simple method called policy invariant explicit shaping (PIES) and show theoretically and empirically that PIES succeeds where DPBA fails.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/16/2020

Learning to Run with Potential-Based Reward Shaping and Demonstrations from Video Data

Learning to produce efficient movement behaviour for humanoid robots fro...
research
07/20/2019

Potential-Based Advice for Stochastic Policy Learning

This paper augments the reward received by a reinforcement learning agen...
research
05/04/2020

Formal Policy Synthesis for Continuous-Space Systems via Reinforcement Learning

This paper studies data-driven techniques for satisfying temporal proper...
research
04/18/2019

Improving Interactive Reinforcement Agent Planning with Human Demonstration

TAMER has proven to be a powerful interactive reinforcement learning met...
research
05/13/2021

Intelligence and Unambitiousness Using Algorithmic Information Theory

Algorithmic Information Theory has inspired intractable constructions of...
research
07/19/2023

Benchmarking Potential Based Rewards for Learning Humanoid Locomotion

The main challenge in developing effective reinforcement learning (RL) p...
research
06/15/2020

Pessimism About Unknown Unknowns Inspires Conservatism

If we could define the set of all bad outcomes, we could hard-code an ag...

Please sign up or login with your details

Forgot password? Click here to reset