Measuring and Characterizing Generalization in Deep Reinforcement Learning

12/07/2018
by   Sam Witty, et al.
12

Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.

READ FULL TEXT

page 2

page 4

research
03/11/2021

Analyzing the Hidden Activations of Deep Policy Networks: Why Representation Matters

We analyze the hidden activations of neural network policies of deep rei...
research
05/18/2022

Generating Explanations from Deep Reinforcement Learning Using Episodic Memory

Deep Reinforcement Learning (RL) involves the use of Deep Neural Network...
research
06/09/2021

TempoRL: Learning When to Act

Reinforcement learning is a powerful approach to learn behaviour through...
research
10/26/2022

ERL-Re^2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation

Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithm (EA) ar...
research
06/04/2023

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

In the field of reinforcement learning (RL), representation learning is ...
research
11/12/2022

Deep Reinforcement Learning with Vector Quantized Encoding

Human decision-making often involves combining similar states into categ...
research
05/16/2020

Concept Learning in Deep Reinforcement Learning

Deep reinforcement learning techniques have shown to be a promising path...

Please sign up or login with your details

Forgot password? Click here to reset