A Policy Gradient Method for Task-Agnostic Exploration

07/09/2020
by   Mirco Mutti, et al.
0

In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy? In this paper, we argue that the entropy of the state distribution induced by limited-horizon trajectories is a sensible target. Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a non-parametric, k-nearest neighbors estimate of the state distribution entropy. In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics. Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning a variety of meaningful reward-based tasks downstream.

READ FULL TEXT

page 7

page 20

page 23

research
02/07/2022

The Importance of Non-Markovianity in Maximum State Entropy Exploration

In the maximum state entropy exploration framework, an agent interacts w...
research
06/11/2020

Exploration by Maximizing Rényi Entropy for Zero-Shot Meta RL

Exploring the transition dynamics is essential to the success of reinfor...
research
09/22/2020

Learning Task-Agnostic Action Spaces for Movement Optimization

We propose a novel method for exploring the dynamics of physically based...
research
04/28/2019

Learning walk and trot from the same objective using different types of exploration

In quadruped gait learning, policy search methods that scale high dimens...
research
06/29/2020

End-Effect Exploration Drive for Effective Motor Learning

End-effect drives are proposed here as an effective way to implement goa...
research
12/02/2021

Reward-Free Attacks in Multi-Agent Reinforcement Learning

We investigate how effective an attacker can be when it only learns from...
research
07/16/2020

PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning

Direct policy gradient methods for reinforcement learning are a successf...

Please sign up or login with your details

Forgot password? Click here to reset