Evolution of Q Values for Deep Q Learning in Stable Baselines

04/24/2020
by   Matthew Andrews, et al.
0

We investigate the evolution of the Q values for the implementation of Deep Q Learning (DQL) in the Stable Baselines library. Stable Baselines incorporates the latest Reinforcement Learning techniques and achieves superhuman performance in many game environments. However, for some simple non-game environments, the DQL in Stable Baselines can struggle to find the correct actions. In this paper we aim to understand the types of environment where this suboptimal behavior can happen, and also investigate the corresponding evolution of the Q values for individual states. We compare a smart TrafficLight environment (where performance is poor) with the AI Gym FrozenLake environment (where performance is perfect). We observe that DQL struggles with TrafficLight because actions are reversible and hence the Q values in a given state are closer than in FrozenLake. We then investigate the evolution of the Q values using a recent decomposition technique of Achiam et al.. We observe that for TrafficLight, the function approximation error and the complex relationships between the states lead to a situation where some Q values meander far from optimal.

READ FULL TEXT
research
08/08/2017

Investigating Reinforcement Learning Agents for Continuous State Space Environments

Given an environment with continuous state spaces and discrete actions, ...
research
10/14/2019

On the Reduction of Variance and Overestimation of Deep Q-Learning

The breakthrough of deep Q-Learning on different types of environments r...
research
11/07/2016

Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

Instability and variability of Deep Reinforcement Learning (DRL) algorit...
research
01/05/2020

Universal Successor Features for Transfer Reinforcement Learning

Transfer in Reinforcement Learning (RL) refers to the idea of applying k...
research
06/17/2020

Delta Schema Network in Model-based Reinforcement Learning

This work is devoted to unresolved problems of Artificial General Intell...
research
06/28/2020

Image Classification by Reinforcement Learning with Two-State Q-Learning

In this paper, a simple and efficient Hybrid Classifier is presented whi...
research
07/02/2019

On Conforming and Conflicting Values

Values are things that are important to us. Actions activate values - th...

Please sign up or login with your details

Forgot password? Click here to reset