Inducing Structure in Reward Learning by Learning Features

01/18/2022
by   Andreea Bobu, et al.
0

Reward learning enables robots to learn adaptable behaviors from human input. Traditional methods model the reward as a linear function of hand-crafted features, but that requires specifying all the relevant features a priori, which is impossible for real-world tasks. To get around this issue, recent deep Inverse Reinforcement Learning (IRL) methods learn rewards directly from the raw state but this is challenging because the robot has to implicitly learn the features that are important and how to combine them, simultaneously. Instead, we propose a divide and conquer approach: focus human input specifically on learning the features separately, and only then learn how to combine them into a reward. We introduce a novel type of human input for teaching features and an algorithm that utilizes it to learn complex features from the raw state space. The robot can then learn how to combine them into a reward using demonstrations, corrections, or other reward learning frameworks. We demonstrate our method in settings where all features have to be learned from scratch, as well as where some of the features are known. By first focusing human input specifically on the feature(s), our method decreases sample complexity and improves generalization of the learned reward over a deepIRL baseline. We show this in experiments with a physical 7DOF robot manipulator, as well as in a user study conducted in a simulated environment.

READ FULL TEXT

page 2

page 9

page 10

page 12

page 14

page 16

page 21

page 23

research
06/23/2020

Feature Expansive Reward Learning: Rethinking Human Input

In collaborative human-robot scenarios, when a person is not satisfied w...
research
01/02/2023

SIRL: Similarity-based Implicit Representation Learning

When robots learn reward functions using high capacity models that take ...
research
06/12/2020

Recursion and evolution: Part II

We examine the question of whether it is possible for a diagonalizing sy...
research
07/07/2022

Energy-based Legged Robots Terrain Traversability Modeling via Deep Inverse Reinforcement Learning

This work reports on developing a deep inverse reinforcement learning me...
research
04/11/2023

Diagnosing and Augmenting Feature Representations in Correctional Inverse Reinforcement Learning

Robots have been increasingly better at doing tasks for humans by learni...
research
04/10/2018

Binary Space Partitioning as Intrinsic Reward

An autonomous agent embodied in a humanoid robot, in order to learn from...
research
03/02/2022

InsertionNet 2.0: Minimal Contact Multi-Step Insertion Using Multimodal Multiview Sensory Input

We address the problem of devising the means for a robot to rapidly and ...

Please sign up or login with your details

Forgot password? Click here to reset