DeepAI AI Chat
Log In Sign Up

Explanation-Aware Experience Replay in Rule-Dense Environments

09/29/2021
by   Francesco Sovrano, et al.
University of Cambridge
University of Bologna
3

Human environments are often regulated by explicit and complex rulesets. Integrating Reinforcement Learning (RL) agents into such environments motivates the development of learning mechanisms that perform well in rule-dense and exception-ridden environments such as autonomous driving on regulated roads. In this paper, we propose a method for organising experience by means of partitioning the experience buffer into clusters labelled on a per-explanation basis. We present discrete and continuous navigation environments compatible with modular rulesets and 9 learning tasks. For environments with explainable rulesets, we convert rule-based explanations into case-based explanations by allocating state-transitions into clusters labelled with explanations. This allows us to sample experiences in a curricular and task-oriented manner, focusing on the rarity, importance, and meaning of events. We label this concept Explanation-Awareness (XA). We perform XA experience replay (XAER) with intra and inter-cluster prioritisation, and introduce XA-compatible versions of DQN, TD3, and SAC. Performance is consistently superior with XA versions of those algorithms, compared to traditional Prioritised Experience Replay baselines, indicating that explanation engineering can be used in lieu of reward engineering for environments with explainable features.

READ FULL TEXT

page 1

page 7

07/14/2020

Learning to Sample with Local and Global Contexts in Experience Replay Buffer

Experience replay, which enables the agents to remember and reuse experi...
03/04/2020

Dynamic Experience Replay

We present a novel technique called Dynamic Experience Replay (DER) that...
09/06/2018

ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay

Experience replay is an important technique for addressing sample-ineffi...
07/03/2022

USHER: Unbiased Sampling for Hindsight Experience Replay

Dealing with sparse rewards is a long-standing challenge in reinforcemen...
02/05/2021

Revisiting Prioritized Experience Replay: A Value Perspective

Experience replay enables off-policy reinforcement learning (RL) agents ...
09/18/2022

A Computational Model of Learning Flexible Navigation in a Maze by Layout-Conforming Replay of Place Cells

Recent experimental observations have shown that the reactivation of hip...