InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problem

05/02/2021
by   Markel Sanz Ausin, et al.
6

The temporal Credit Assignment Problem (CAP) is a well-known and challenging task in AI. While Reinforcement Learning (RL), especially Deep RL, works well when immediate rewards are available, it can fail when only delayed rewards are available or when the reward function is noisy. In this work, we propose delegating the CAP to a Neural Network-based algorithm named InferNet that explicitly learns to infer the immediate rewards from the delayed rewards. The effectiveness of InferNet was evaluated on two online RL tasks: a simple GridWorld and 40 Atari games; and two offline RL tasks: GridWorld and a real-life Sepsis treatment task. For all tasks, the effectiveness of using the InferNet inferred rewards is compared against the immediate and the delayed rewards with two settings: with noisy rewards and without noise. Overall, our results show that the effectiveness of InferNet is robust against noisy reward functions and is an effective add-on mechanism for solving temporal CAP in a wide range of RL tasks, from classic RL simulation environments to a real-world RL problem and for both online and offline learning.

READ FULL TEXT

page 2

page 3

page 4

page 6

page 8

page 9

page 11

page 12

research
02/08/2023

Temporal Video-Language Alignment Network for Reward Shaping in Reinforcement Learning

Designing appropriate reward functions for Reinforcement Learning (RL) a...
research
05/28/2023

Potential-based Credit Assignment for Cooperative RL-based Testing of Autonomous Vehicles

While autonomous vehicles (AVs) may perform remarkably well in generic r...
research
06/04/2018

TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning

Our understanding of reinforcement learning (RL) has been shaped by theo...
research
10/02/2018

Reinforcement Learning with Perturbed Rewards

Recent studies have shown the vulnerability of reinforcement learning (R...
research
02/03/2023

Mind the Gap: Offline Policy Optimization for Imperfect Rewards

Reward function is essential in reinforcement learning (RL), serving as ...
research
10/12/2022

Contrastive introspection (ConSpec) to rapidly identify invariant steps for success

Reinforcement learning (RL) algorithms have achieved notable success in ...
research
10/28/2021

Extracting Clinician's Goals by What-if Interpretable Modeling

Although reinforcement learning (RL) has tremendous success in many fiel...

Please sign up or login with your details

Forgot password? Click here to reset