Probabilistic Successor Representations with Kalman Temporal Differences

10/06/2019
by   Jesse P. Geerts, et al.
0

The effectiveness of Reinforcement Learning (RL) depends on an animal's ability to assign credit for rewards to the appropriate preceding stimuli. One aspect of understanding the neural underpinnings of this process involves understanding what sorts of stimulus representations support generalisation. The Successor Representation (SR), which enforces generalisation over states that predict similar outcomes, has become an increasingly popular model in this space of inquiries. Another dimension of credit assignment involves understanding how animals handle uncertainty about learned associations, using probabilistic methods such as Kalman Temporal Differences (KTD). Combining these approaches, we propose using KTD to estimate a distribution over the SR. KTD-SR captures uncertainty about the estimated SR as well as covariances between different long-term predictions. We show that because of this, KTD-SR exhibits partial transition revaluation as humans do in this experiment without additional replay, unlike the standard TD-SR algorithm. We conclude by discussing future applications of the KTD-SR as a model of the interaction between predictive and probabilistic animal reasoning.

READ FULL TEXT

page 1

page 3

research
02/24/2021

Synthetic Returns for Long-Term Credit Assignment

Since the earliest days of reinforcement learning, the workhorse method ...
research
02/08/2019

Source Traces for Temporal Difference Learning

This paper motivates and develops source traces for temporal difference ...
research
11/03/2021

The effect of synaptic weight initialization in feature-based successor representation learning

After discovering place cells, the idea of the hippocampal (HPC) functio...
research
03/31/2022

AKF-SR: Adaptive Kalman Filtering-based Successor Representation

Recent studies in neuroscience suggest that Successor Representation (SR...
research
04/22/2023

Sequential Recommendation with Probabilistic Logical Reasoning

Deep learning and symbolic learning are two frequently employed methods ...
research
12/30/2021

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Distributed Multi-Agent Reinforcement Learning (MARL) algorithms has att...
research
09/07/2023

A State Representation for Diminishing Rewards

A common setting in multitask reinforcement learning (RL) demands that a...

Please sign up or login with your details

Forgot password? Click here to reset