Pitfalls of learning a reward function online

by   Stuart Armstrong, et al.

In some agent designs like inverse reinforcement learning an agent needs to learn its own reward function. Learning the reward function and optimising for it are typically two different processes, usually performed at different stages. We consider a continual (“one life”) learning approach where the agent both learns the reward function and optimises for it at the same time. We show that this comes with a number of pitfalls, such as deliberately manipulating the learning process in one direction, refusing to learn, “learning” facts already known to the agent, and making decisions that are strictly dominated (for all relevant reward functions). We formally introduce two desirable properties: the first is `unriggability', which prevents the agent from steering the learning process in the direction of a reward function that is easier to optimise. The second is `uninfluenceability', whereby the reward-function learning process operates by learning facts about the environment. We show that an uninfluenceable process is automatically unriggable, and if the set of possible environments is sufficiently rich, the converse is true too.



page 7


Generative Inverse Deep Reinforcement Learning for Online Recommendation

Deep reinforcement learning enables an agent to capture user's interest ...

Deep Reinforcement Learning with Stage Incentive Mechanism for Robotic Trajectory Planning

To improve the efficiency of deep reinforcement learning (DRL) based met...

Scalable agent alignment via reward modeling: a research direction

One obstacle to applying reinforcement learning algorithms to real-world...

Simplifying Reward Design through Divide-and-Conquer

Designing a good reward function is essential to robot planning and rein...

Making the most of your day: online learning for optimal allocation of time

We study online learning for optimal allocation when the resource to be ...

On the Expressivity of Markov Reward

Reward is the driving force for reinforcement-learning agents. This pape...

Potential-based Reward Shaping in Sokoban

Learning to solve sparse-reward reinforcement learning problems is diffi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.