On the Generalization of Representations in Reinforcement Learning

03/01/2022
by   Charline Le Lan, et al.
66

In reinforcement learning, state representations are used to tractably deal with large problem spaces. State representations serve both to approximate the value function with few parameters, but also to generalize to newly encountered states. Their features may be learned implicitly (as part of a neural network) or explicitly (for example, the successor representation of <cit.>). While the approximation properties of representations are reasonably well-understood, a precise characterization of how and when these representations generalize is lacking. In this work, we address this gap and provide an informative bound on the generalization error arising from a specific state representation. This bound is based on the notion of effective dimension which measures the degree to which knowing the value at one state informs the value at other states. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well. We complement our theoretical results with an empirical survey of classic representation learning methods from the literature and results on the Arcade Learning Environment, and find that the generalization behaviour of learned representations is well-explained by their effective dimension.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/12/2012

The Thing That We Tried Didn't Work Very Well : Deictic Representation in Reinforcement Learning

Most reinforcement learning methods operate on propositional representat...
research
10/27/2021

Towards Robust Bisimulation Metric Learning

Learned representations in deep reinforcement learning (DRL) have to ext...
research
06/16/2023

Bootstrapped Representations in Reinforcement Learning

In reinforcement learning (RL), state representations are key to dealing...
research
01/31/2019

A Geometric Perspective on Optimal Representations for Reinforcement Learning

This paper proposes a new approach to representation learning based on g...
research
01/13/2021

Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Reinforcement learning methods trained on few environments rarely learn ...
research
02/02/2021

Metrics and continuity in reinforcement learning

In most practical applications of reinforcement learning, it is untenabl...
research
07/06/2021

A Short Note on the Relationship of Information Gain and Eluder Dimension

Eluder dimension and information gain are two widely used methods of com...

Please sign up or login with your details

Forgot password? Click here to reset