DeepAI AI Chat
Log In Sign Up

Fast active learning for pure exploration in reinforcement learning

07/27/2020
by   Pierre Ménard, et al.
10

Realistic environments often provide agents with very limited feedback. When the environment is initially unknown, the feedback, in the beginning, can be completely absent, and the agents may first choose to devote all their effort on exploring efficiently. The exploration remains a challenge while it has been addressed with many hand-tuned heuristics with different levels of generality on one side, and a few theoretically-backed exploration strategies on the other. Many of them are incarnated by intrinsic motivation and in particular explorations bonuses. A common rule of thumb for exploration bonuses is to use 1/√(n) bonus that is added to the empirical estimates of the reward, where n is a number of times this particular state (or a state-action pair) was visited. We show that, surprisingly, for a pure-exploration objective of reward-free exploration, bonuses that scale with 1/n bring faster learning rates, improving the known upper bounds with respect to the dependence on the horizon H. Furthermore, we show that with an improved analysis of the stopping time, we can improve by a factor H the sample complexity in the best-policy identification setting, which is another pure-exploration objective, where the environment provides rewards but the agent is not penalized for its behavior during the exploration phase.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/18/2019

Scheduled Intrinsic Drive: A Hierarchical Take on Intrinsically Motivated Exploration

Exploration in sparse reward reinforcement learning remains a difficult ...
06/14/2020

Combinatorial Pure Exploration with Partial or Full-Bandit Linear Feedback

In this paper, we propose the novel model of combinatorial pure explorat...
10/21/2019

Exploration via Sample-Efficient Subgoal Design

The problem of exploration in unknown environments continues to pose a c...
03/10/2020

Exploring Unknown States with Action Balance

Exploration is a key problem in reinforcement learning. Recently bonus-b...
06/11/2020

Adaptive Reward-Free Exploration

Reward-free exploration is a reinforcement learning setting recently stu...
07/29/2018

Sidekick Policy Learning for Active Visual Exploration

We consider an active visual exploration scenario, where an agent must i...
02/19/2021

Learning to Stop with Surprisingly Few Samples

We consider a discounted infinite horizon optimal stopping problem. If t...

Code Repositories