An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

03/10/2021
by   Dilip Arumugam, et al.
0

How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the information sparsity. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/18/2019

Credit Assignment as a Proxy for Transfer in Reinforcement Learning

The ability to transfer representations to novel environments and tasks ...
research
03/07/2022

On Credit Assignment in Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning (HRL) has held longstanding promise ...
research
02/09/2021

Pairwise Weights for Temporal Credit Assignment

How much credit (or blame) should an action taken in a state get for a f...
research
05/14/2023

Theta sequences as eligibility traces: a biological solution to credit assignment

Credit assignment problems, for example policy evaluation in RL, often r...
research
12/02/2022

Credit Assignment for Trained Neural Networks Based on Koopman Operator Theory

Credit assignment problem of neural networks refers to evaluating the cr...
research
11/18/2020

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Credit assignment in reinforcement learning is the problem of measuring ...
research
06/01/2022

Predecessor Features

Any reinforcement learning system must be able to identify which past ev...

Please sign up or login with your details

Forgot password? Click here to reset