Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

by   Alexander Meulemans, et al.

To make reinforcement learning more sample efficient, we need better credit assignment methods that measure an action's influence on future rewards. Building upon Hindsight Credit Assignment (HCA), we introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards, by quantifying a counterfactual query: "Would the agent still have reached this reward if it had taken another action?". We show that measuring contributions w.r.t. rewarding states, as is done in HCA, results in spurious estimates of contributions, causing HCA to degrade towards the high-variance REINFORCE estimator in many relevant environments. Instead, we measure contributions w.r.t. rewards or learned representations of the rewarding objects, resulting in gradient estimates with lower variance. We run experiments on a suite of problems specifically designed to evaluate long-term credit assignment capabilities. By using dynamic programming, we measure ground-truth policy gradients and show that the improved performance of our new model-based credit assignment methods is due to lower bias and variance compared to HCA and common baselines. Our results demonstrate how modeling action contributions towards rewarding outcomes can be leveraged for credit assignment, opening a new path towards sample-efficient reinforcement learning.


page 1

page 2

page 3

page 4


Counterfactual Credit Assignment in Model-Free Reinforcement Learning

Credit assignment in reinforcement learning is the problem of measuring ...

Towards Causal Credit Assignment

Adequately assigning credit to actions for future outcomes based on thei...

Synthetic Returns for Long-Term Credit Assignment

Since the earliest days of reinforcement learning, the workhorse method ...

Credit Assignment Techniques in Stochastic Computation Graphs

Stochastic computation graphs (SCGs) provide a formalism to represent st...

Variance Reduced Advantage Estimation with δ Hindsight Credit Assignment

Hindsight Credit Assignment (HCA) refers to a recently proposed family o...

Potential-based Credit Assignment for Cooperative RL-based Testing of Autonomous Vehicles

While autonomous vehicles (AVs) may perform remarkably well in generic r...

Pairwise Weights for Temporal Credit Assignment

How much credit (or blame) should an action taken in a state get for a f...

Please sign up or login with your details

Forgot password? Click here to reset