Intervention Efficient Algorithm for Two-Stage Causal MDPs

11/01/2021
by   Rahul Madhavan, et al.
0

We study Markov Decision Processes (MDP) wherein states correspond to causal graphs that stochastically generate rewards. In this setup, the learner's goal is to identify atomic interventions that lead to high rewards by intervening on variables at each state. Generalizing the recent causal-bandit framework, the current work develops (simple) regret minimization guarantees for two-stage causal MDPs, with parallel causal graph at each state. We propose an algorithm that achieves an instance dependent regret bound. A key feature of our algorithm is that it utilizes convex optimization to address the exploration problem. We identify classes of instances wherein our regret guarantee is essentially tight, and experimentally validate our theoretical results.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/15/2021

Causal Markov Decision Processes: Learning Good Interventions Efficiently

We introduce causal Markov Decision Processes (C-MDPs), a new formalism ...
research
07/06/2021

Causal Bandits on General Graphs

We study the problem of determining the best intervention in a Causal Ba...
research
05/25/2019

Large Scale Markov Decision Processes with Changing Rewards

We consider Markov Decision Processes (MDPs) where the rewards are unkno...
research
05/08/2023

Learning Good Interventions in Causal Graphs via Covering

We study the causal bandit problem that entails identifying a near-optim...
research
12/13/2020

Budgeted and Non-budgeted Causal Bandits

Learning good interventions in a causal graph can be modelled as a stoch...
research
07/18/2021

Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonuses

Policy optimization is a widely-used method in reinforcement learning. D...
research
11/27/2022

Counterfactual Optimism: Rate Optimal Regret for Stochastic Contextual MDPs

We present the UC^3RL algorithm for regret minimization in Stochastic Co...

Please sign up or login with your details

Forgot password? Click here to reset