Predecessor Features

06/01/2022
by   Duncan Bailey, et al.
0

Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we propose "Predecessor Features", an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, our algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of predecessor states than conventional methods, greatly improving learning speed. Our algorithm can also be naturally extended from tabular state representation to feature representations allowing for increased performance on a wide range of environments. We demonstrate several use cases for Predecessor Features and contrast its performance with other similar approaches.

READ FULL TEXT
research
12/05/2019

Hindsight Credit Assignment

We consider the problem of efficient credit assignment in reinforcement ...
research
02/20/2022

Selective Credit Assignment

Efficient credit assignment is essential for reinforcement learning algo...
research
03/10/2021

An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

How do we formalize the challenge of credit assignment in reinforcement ...
research
07/18/2019

Credit Assignment as a Proxy for Transfer in Reinforcement Learning

The ability to transfer representations to novel environments and tasks ...
research
11/19/2019

Variance Reduced Advantage Estimation with δ Hindsight Credit Assignment

Hindsight Credit Assignment (HCA) refers to a recently proposed family o...
research
05/14/2023

Theta sequences as eligibility traces: a biological solution to credit assignment

Credit assignment problems, for example policy evaluation in RL, often r...
research
02/09/2022

A new perspective on classification: optimally allocating limited resources to uncertain tasks

A central problem in business concerns the optimal allocation of limited...

Please sign up or login with your details

Forgot password? Click here to reset