Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments

07/14/2021
by   Francesco Massari, et al.
0

Reinforcement Learning (RL) is known to be often unsuccessful in environments with sparse extrinsic rewards. A possible countermeasure is to endow RL agents with an intrinsic reward function, or 'intrinsic motivation', which rewards the agent based on certain features of the current sensor state. An intrinsic reward function based on the principle of empowerment assigns rewards proportional to the amount of control the agent has over its own sensors. We implemented a variation on a recently proposed intrinsically motivated agent, which we refer to as the 'curious' agent, and an empowerment-inspired agent. The former leverages sensor state encoding with a variational autoencoder, while the latter predicts the next sensor state via a variational information bottleneck. We compared the performance of both agents to that of an advantage actor-critic baseline in four sparse reward grid worlds. Both the empowerment agent and its curious competitor seem to benefit to similar extents from their intrinsic rewards. This provides some experimental support to the conjecture that empowerment can be used to drive exploration.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/31/2019

VASE: Variational Assorted Surprise Exploration for Reinforcement Learning

Exploration in environments with continuous control and sparse rewards r...
research
10/07/2022

Generative Augmented Flow Networks

The Generative Flow Network is a probabilistic framework where an agent ...
research
05/11/2020

Maximizing Information Gain in Partially Observable Environments via Prediction Reward

Information gathering in a partially observable environment can be formu...
research
02/08/2020

Capsule Network Performance with Autonomous Navigation

Capsule Networks (CapsNets) have been proposed as an alternative to Conv...
research
12/18/2017

'Indifference' methods for managing agent rewards

Indifference is a class of methods that are used to control a reward bas...
research
06/18/2018

A unified strategy for implementing curiosity and empowerment driven reinforcement learning

Although there are many approaches to implement intrinsically motivated ...
research
12/01/2019

Affect-based Intrinsic Rewards for Learning General Representations

Positive affect has been linked to increased interest, curiosity and sat...

Please sign up or login with your details

Forgot password? Click here to reset