Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods

02/25/2021
by   Nicholay Topin, et al.
7

Current work in explainable reinforcement learning generally produces policies in the form of a decision tree over the state space. Such policies can be used for formal safety verification, agent behavior prediction, and manual inspection of important features. However, existing approaches fit a decision tree after training or use a custom learning procedure which is not compatible with new learning techniques, such as those which use neural networks. To address this limitation, we propose a novel Markov Decision Process (MDP) type for learning decision tree policies: Iterative Bounding MDPs (IBMDPs). An IBMDP is constructed around a base MDP so each IBMDP policy is guaranteed to correspond to a decision tree policy for the base MDP when using a method-agnostic masking procedure. Because of this decision tree equivalence, any function approximator can be used during training, including a neural network, while yielding a decision tree policy for the base MDP. We present the required masking procedure as well as a modified value update step which allows IBMDPs to be solved using existing algorithms. We apply this procedure to produce IBMDP variants of recent reinforcement learning methods. We empirically show the benefits of our approach by solving IBMDPs to produce decision tree policies for the base MDPs.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/22/2018

Verifiable Reinforcement Learning via Policy Extraction

While deep reinforcement learning has successfully solved many challengi...
01/30/2023

Optimal Decision Tree Policies for Markov Decision Processes

Interpretability of reinforcement learning policies is essential for man...
07/02/2019

Conservative Q-Improvement: Reinforcement Learning for an Interpretable Decision-Tree Policy

There is a growing desire in the field of reinforcement learning (and ma...
06/16/2019

MoËT: Interpretable and Verifiable Reinforcement Learning via Mixture of Expert Trees

Deep Reinforcement Learning (DRL) has led to many recent breakthroughs o...
06/06/2021

Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning

In the predict-then-optimize framework, the objective is to train a pred...
12/21/2020

Universal Policies for Software-Defined MDPs

We introduce a new programming paradigm called oracle-guided decision pr...

Please sign up or login with your details

Forgot password? Click here to reset