DeepAI AI Chat
Log In Sign Up

Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

by   Pedro Sequeira, et al.
SRI International

In recent years, advances in deep learning have resulted in a plethora of successes in the use of reinforcement learning (RL) to solve complex sequential decision tasks with high-dimensional inputs. However, existing systems lack the necessary mechanisms to provide humans with a holistic view of their competence, presenting an impediment to their adoption, particularly in critical applications where the decisions an agent makes can have significant consequences. Yet, existing RL-based systems are essentially competency-unaware in that they lack the necessary interpretation mechanisms to allow human operators to have an insightful, holistic view of their competency. In this paper, we extend a recently-proposed framework for explainable RL that is based on analyses of "interestingness." Our new framework provides various measures of RL agent competence stemming from interestingness analysis and is applicable to a wide range of RL algorithms. We also propose novel mechanisms for assessing RL agents' competencies that: 1) identify agent behavior patterns and competency-controlling conditions by clustering agent behavior traces solely using interestingness data; and 2) identify the task elements mostly responsible for an agent's behavior, as measured through interestingness, by performing global and local analyses using SHAP values. Overall, our tools provide insights about RL agent competence, both their capabilities and limitations, enabling users to make more informed decisions about interventions, additional training, and other interactions in collaborative human-machine settings.


Complementary reinforcement learning toward explainable agents

Reinforcement learning (RL) algorithms allow agents to learn skills and ...

A Framework for Understanding and Visualizing Strategies of RL Agents

Recent years have seen significant advances in explainable AI as the nee...

Search-Based Testing of Reinforcement Learning

Evaluation of deep reinforcement learning (RL) is inherently challenging...

Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning

Deep reinforcement learning (RL) agents are becoming increasingly profic...

Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations

Enabling humans to identify potential flaws in an agent's decision makin...

SaLinA: Sequential Learning of Agents

SaLinA is a simple library that makes implementing complex sequential le...