Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

07/02/2019
by   Aaron M. Roth, et al.
2

There is a growing desire in the field of reinforcement learning (and machine learning in general) to move from black-box models toward more "interpretable AI." We improve interpretability of reinforcement learning by increasing the utility of decision tree policies learned via reinforcement learning. These policies consist of a decision tree over the state space, which requires fewer parameters to express than traditional policy representations. Existing methods for creating decision tree policies via reinforcement learning focus on accurately representing an action-value function during training, but this leads to much larger trees than would otherwise be required. To address this shortcoming, we propose a novel algorithm which only increases tree size when the estimated discounted future reward of the overall policy would increase by a sufficient amount. Through evaluation in a simulated environment, we show that its performance is comparable or superior to traditional tree-based approaches and that it yields a more succinct policy. Additionally, we discuss tuning parameters to control the tradeoff between optimizing for smaller tree size or for overall reward.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/22/2018

Verifiable Reinforcement Learning via Policy Extraction

While deep reinforcement learning has successfully solved many challengi...
research
06/09/2022

There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazes

Interpretability is an essential building block for trustworthiness in r...
research
08/16/2021

Neural-to-Tree Policy Distillation with Policy Improvement Criterion

While deep reinforcement learning has achieved promising results in chal...
research
01/18/2020

Tree-Structured Policy based Progressive Reinforcement Learning for Temporally Language Grounding in Video

Temporally language grounding in untrimmed videos is a newly-raised task...
research
02/25/2021

Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

Current work in explainable reinforcement learning generally produces po...
research
01/18/2021

Interpretable Policy Specification and Synthesis through Natural Language and RL

Policy specification is a process by which a human can initialize a robo...
research
03/03/2022

MetaDT: Meta Decision Tree for Interpretable Few-Shot Learning

Few-Shot Learning (FSL) is a challenging task, which aims to recognize n...

Please sign up or login with your details

Forgot password? Click here to reset