Avoiding Side Effects in Complex Environments

06/11/2020
by   Alexander Matt Turner, et al.
0

Reward function specification can be difficult, even in simple environments. Realistic environments contain millions of states. Rewarding the agent for making a widget may be easy, but penalizing the multitude of possible negative side effects is hard. In toy environments, Attainable Utility Preservation (AUP) avoids side effects by penalizing shifts in the ability to achieve randomly generated goals. We scale this approach to large, randomly generated environments based on Conway's Game of Life. By preserving optimal value for a single randomly generated reward function, AUP incurs modest overhead, completes the specified task, and avoids side effects.

READ FULL TEXT

page 3

page 4

page 6

page 7

research
10/15/2020

Avoiding Side Effects By Considering Future Tasks

Designing reward functions is difficult: the designer has to specify wha...
research
08/30/2023

Iterative Reward Shaping using Human Feedback for Correcting Reward Misspecification

A well-defined reward function is crucial for successful training of an ...
research
05/02/2021

Curious Exploration and Return-based Memory Restoration for Deep Reinforcement Learning

Reward engineering and designing an incentive reward function are non-tr...
research
02/26/2019

Conservative Agency via Attainable Utility Preservation

Reward functions are often misspecified. An agent optimizing an incorrec...
research
01/25/2016

Towards Resolving Unidentifiability in Inverse Reinforcement Learning

We consider a setting for Inverse Reinforcement Learning (IRL) where the...
research
01/29/2021

Challenges for Using Impact Regularizers to Avoid Negative Side Effects

Designing reward functions for reinforcement learning is difficult: besi...
research
04/17/2018

Terrain RL Simulator

We provide 89 challenging simulation environments that range in difficul...

Please sign up or login with your details

Forgot password? Click here to reset