An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

03/10/2021
by   Dilip Arumugam, et al.
0

How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the information sparsity. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/18/2019

Credit Assignment as a Proxy for Transfer in Reinforcement Learning

The ability to transfer representations to novel environments and tasks ...
03/07/2022

On Credit Assignment in Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning (HRL) has held longstanding promise ...
02/09/2021

Pairwise Weights for Temporal Credit Assignment

How much credit (or blame) should an action taken in a state get for a f...
11/18/2020

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Credit assignment in reinforcement learning is the problem of measuring ...
06/01/2022

Predecessor Features

Any reinforcement learning system must be able to identify which past ev...
10/26/2020

Forethought and Hindsight in Credit Assignment

We address the problem of credit assignment in reinforcement learning an...
05/28/2019

Learning distant cause and effect using only local and immediate credit assignment

We present a recurrent neural network memory that uses sparse coding to ...