DeepAI AI Chat
Log In Sign Up

Identifying Reasoning Flaws in Planning-Based RL Using Tree Explanations

by   Kin-Ho Lam, et al.

Enabling humans to identify potential flaws in an agent's decision making is an important Explainable AI application. We consider identifying such flaws in a planning-based deep reinforcement learning (RL) agent for a complex real-time strategy game. In particular, the agent makes decisions via tree search using a learned model and evaluation function over interpretable states and actions. This gives the potential for humans to identify flaws at the level of reasoning steps in the tree, even if the entire reasoning process is too complex to understand. However, it is unclear whether humans will be able to identify such flaws due to the size and complexity of trees. We describe a user interface and case study, where a small group of AI experts and developers attempt to identify reasoning flaws due to inaccurate agent learning. Overall, the interface allowed the group to identify a number of significant flaws of varying types, demonstrating the promise of this approach.


page 2

page 3

page 4


Beyond Value: CHECKLIST for Testing Inferences in Planning-Based RL

Reinforcement learning (RL) agents are commonly evaluated via their expe...

Decisions that Explain Themselves: A User-Centric Deep Reinforcement Learning Explanation System

With deep reinforcement learning (RL) systems like autonomous driving be...

Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning

Reinforcement Learning (RL) is a widely-used technique in many domains, ...

Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

In recent years, advances in deep learning have resulted in a plethora o...

Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop

Human-in-the-loop (HiL) reinforcement learning is gaining traction in do...

SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and Reasoning

Open-world survival games pose significant challenges for AI algorithms ...

Local Explanations for Reinforcement Learning

Many works in explainable AI have focused on explaining black-box classi...