Identifying Critical States by the Action-Based Variance of Expected Return

08/26/2020
by   Izumi Karino, et al.
0

The balance of exploration and exploitation plays a crucial role in accelerating reinforcement learning (RL). To deploy an RL agent in human society, its explainability is also essential. However, basic RL approaches have difficulties in deciding when to choose exploitation as well as in extracting useful points for a brief explanation of its operation. One reason for the difficulties is that these approaches treat all states the same way. Here, we show that identifying critical states and treating them specially is commonly beneficial to both problems. These critical states are the states at which the action selection changes the potential of success and failure substantially. We propose to identify the critical states using the variance in the Q-function for the actions and to perform exploitation with high probability on the identified states. These simple methods accelerate RL in a grid world with cliffs and two baseline tasks of deep RL. Our results also demonstrate that the identified critical states are intuitively interpretable regarding the crucial nature of the action selection. Furthermore, our analysis of the relationship between the timing of the identification of especially critical states and the rapid progress of learning suggests there are a few especially critical states that have important information for accelerating RL rapidly.

READ FULL TEXT

page 7

page 8

research
12/31/2011

T-Learning

Traditional Reinforcement Learning (RL) has focused on problems involvin...
research
10/24/2022

Causal Explanation for Reinforcement Learning: Quantifying State and Temporal Importance

Explainability plays an increasingly important role in machine learning....
research
05/30/2022

SEREN: Knowing When to Explore and When to Exploit

Efficient reinforcement learning (RL) involves a trade-off between "expl...
research
01/03/2023

Towards Deployable RL – What's Broken with RL Research and a Potential Fix

Reinforcement learning (RL) has demonstrated great potential, but is cur...
research
01/03/2020

Making Sense of Reinforcement Learning and Probabilistic Inference

Reinforcement learning (RL) combines a control problem with statistical ...
research
08/07/2023

Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations

Offline reinforcement learning (RL) methods strike a balance between exp...

Please sign up or login with your details

Forgot password? Click here to reset