RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments

02/27/2020
by   Roberta Raileanu, et al.
0

Exploration in sparse reward environments remains one of the key challenges of model-free reinforcement learning. Instead of solely relying on extrinsic rewards provided by the environment, many state-of-the-art methods use intrinsic rewards to encourage exploration. However, we show that existing methods fall short in procedurally-generated environments where an agent is unlikely to visit a state more than once. We propose a novel type of intrinsic reward which encourages the agent to take actions that lead to significant changes in its learned state representation. We evaluate our method on multiple challenging procedurally-generated tasks in MiniGrid, as well as on tasks with high-dimensional observations used in prior work. Our experiments demonstrate that this approach is more sample efficient than existing exploration methods, particularly for procedurally-generated MiniGrid environments. Furthermore, we analyze the learned behavior as well as the intrinsic reward received by our agent. In contrast to previous approaches, our intrinsic reward does not diminish during the course of training and it rewards the agent substantially more for interacting with objects that it can control.

READ FULL TEXT

page 5

page 18

research
05/24/2023

Successor-Predecessor Intrinsic Exploration

Exploration is essential in reinforcement learning, particularly in envi...
research
01/20/2021

Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments

Exploration under sparse reward is a long-standing challenge of model-fr...
research
08/30/2023

Cyclophobic Reinforcement Learning

In environments with sparse rewards, finding a good inductive bias for e...
research
03/18/2019

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Exploration in sparse reward reinforcement learning remains a difficult ...
research
09/14/2021

Focus on Impact: Indoor Exploration with Intrinsic Motivation

Exploration of indoor environments has recently experienced a significan...
research
02/17/2022

Improving Intrinsic Exploration with Language Abstractions

Reinforcement learning (RL) agents are particularly hard to train when r...
research
10/11/2022

Exploration via Elliptical Episodic Bonuses

In recent years, a number of reinforcement learning (RL) methods have be...

Please sign up or login with your details

Forgot password? Click here to reset