DeepAI AI Chat
Log In Sign Up

Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations

by   Pedro Sequeira, et al.
SRI International

We propose an explainable reinforcement learning (XRL) framework that analyzes an agent's history of interaction with the environment to extract interestingness elements that help explain its behavior. The framework relies on data readily available from standard RL algorithms, augmented with data that can easily be collected by the agent while learning. We describe how to create visual explanations of an agent's behavior in the form of short video-clips highlighting key interaction moments, based on the proposed elements. We also report on a user study where we evaluated the ability of humans in correctly perceiving the aptitude of agents with different characteristics, including their capabilities and limitations, given explanations automatically generated by our framework. The results show that the diversity of aspects captured by the different interestingness elements is crucial to help humans correctly identify the agents' aptitude in the task, and determine when they might need adjustments to improve their performance.


page 10

page 13

page 16

page 24


Culture-Based Explainable Human-Agent Deconfliction

Law codes and regulations help organise societies for centuries, and as ...

Explainable Reinforcement Learning via Model Transforms

Understanding emerging behaviors of reinforcement learning (RL) agents m...

A Framework for Constrained and Adaptive Behavior-Based Agents

Behavior Trees are commonly used to model agents for robotics and games,...

Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

In recent years, advances in deep learning have resulted in a plethora o...

Decisions that Explain Themselves: A User-Centric Deep Reinforcement Learning Explanation System

With deep reinforcement learning (RL) systems like autonomous driving be...

First-Person Perceptual Guidance Behavior Decomposition using Active Constraint Classification

Humans exhibit a wide range of adaptive and robust dynamic motion behavi...

Code Repositories


A python library for eXplainable Reinforcement Learning (XRL) based on the concept of interestingness elements.

view repo