An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

by   Dilip Arumugam, et al.

How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the information sparsity. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.


page 1

page 2

page 3

page 4


Credit Assignment as a Proxy for Transfer in Reinforcement Learning

The ability to transfer representations to novel environments and tasks ...

On Credit Assignment in Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning (HRL) has held longstanding promise ...

Pairwise Weights for Temporal Credit Assignment

How much credit (or blame) should an action taken in a state get for a f...

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Credit assignment in reinforcement learning is the problem of measuring ...

Predecessor Features

Any reinforcement learning system must be able to identify which past ev...

Forethought and Hindsight in Credit Assignment

We address the problem of credit assignment in reinforcement learning an...

Learning distant cause and effect using only local and immediate credit assignment

We present a recurrent neural network memory that uses sparse coding to ...