On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference

06/23/2019
by   Rohin Shah, et al.
0

Our goal is for agents to optimize the right reward function, despite how difficult it is for us to specify what that is. Inverse Reinforcement Learning (IRL) enables us to infer reward functions from demonstrations, but it usually assumes that the expert is noisily optimal. Real people, on the other hand, often have systematic biases: risk-aversion, myopia, etc. One option is to try to characterize these biases and account for them explicitly during learning. But in the era of deep learning, a natural suggestion researchers make is to avoid mathematical models of human behavior that are fraught with specific assumptions, and instead use a purely data-driven approach. We decided to put this to the test -- rather than relying on assumptions about which specific bias the demonstrator has when planning, we instead learn the demonstrator's planning algorithm that they use to generate demonstrations, as a differentiable planner. Our exploration yielded mixed findings: on the one hand, learning the planner can lead to better reward inference than relying on the wrong assumption; on the other hand, this benefit is dwarfed by the loss we incur by going from an exact to a differentiable planner. This suggest that at least for the foreseeable future, agents need a middle ground between the flexibility of data-driven methods and the useful bias of known human biases. Code is available at https://tinyurl.com/learningbiases.

READ FULL TEXT

page 3

page 7

page 8

research
11/18/2020

Inverse Reinforcement Learning via Matching of Optimality Profiles

The goal of inverse reinforcement learning (IRL) is to infer a reward fu...
research
02/25/2022

Context-Hierarchy Inverse Reinforcement Learning

An inverse reinforcement learning (IRL) agent learns to act intelligentl...
research
01/02/2020

Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation

Reinforcement learning (RL) has achieved tremendous success as a general...
research
11/08/2017

Inverse Reward Design

Autonomous agents optimize the reward function we give them. What they d...
research
02/15/2022

Zero-Shot Assistance in Novel Decision Problems

We consider the problem of creating assistants that can help agents - of...
research
12/09/2022

On the Sensitivity of Reward Inference to Misspecified Human Models

Inferring reward functions from human behavior is at the center of value...
research
11/12/2021

Human irrationality: both bad and good for reward inference

Assuming humans are (approximately) rational enables robots to infer rew...

Please sign up or login with your details

Forgot password? Click here to reset