Log In Sign Up

A Framework for Understanding and Visualizing Strategies of RL Agents

by   Pedro Sequeira, et al.

Recent years have seen significant advances in explainable AI as the need to understand deep learning models has gained importance with the increased emphasis on trust and ethics in AI. Comprehensible models for sequential decision tasks are a particular challenge as they require understanding not only individual predictions but a series of predictions that interact with environmental dynamics. We present a framework for learning comprehensible models of sequential decision tasks in which agent strategies are characterized using temporal logic formulas. Given a set of agent traces, we first cluster the traces using a novel embedding method that captures frequent action patterns. We then search for logical formulas that explain the agent strategies in the different clusters. We evaluate our framework on combat scenarios in StarCraft II (SC2), using traces from a handcrafted expert policy and a trained reinforcement learning agent. We implemented a feature extractor for SC2 environments that extracts traces as sequences of high-level features describing both the state of the environment and the agent's local behavior from agent replays. We further designed a visualization tool depicting the movement of units in the environment that helps understand how different task conditions lead to distinct agent behavior patterns in each trace cluster. Experimental results show that our framework is capable of separating agent traces into distinct groups of behaviors for which our approach to strategy inference produces consistent, meaningful, and easily understood strategy descriptions.


page 6

page 12

page 13

page 14


Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

In recent years, advances in deep learning have resulted in a plethora o...

Explaining Agent's Decision-making in a Hierarchical Reinforcement Learning Scenario

Reinforcement learning is a machine learning approach based on behaviora...

Search-Based Testing of Reinforcement Learning

Evaluation of deep reinforcement learning (RL) is inherently challenging...

ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents

As reinforcement learning methods increasingly amass accomplishments, th...

Induction of Subgoal Automata for Reinforcement Learning

In this work we present ISA, a novel approach for learning and exploitin...

Expert-driven Trace Clustering with Instance-level Constraints

Within the field of process mining, several different trace clustering a...