Optimizing Norm-Bounded Weighted Ambiguity Sets for Robust MDPs

12/04/2019
by   Reazul Hasan Russel, et al.
0

Optimal policies in Markov decision processes (MDPs) are very sensitive to model misspecification. This raises serious concerns about deploying them in high-stake domains. Robust MDPs (RMDP) provide a promising framework to mitigate vulnerabilities by computing policies with worst-case guarantees in reinforcement learning. The solution quality of an RMDP depends on the ambiguity set, which is a quantification of model uncertainties. In this paper, we propose a new approach for optimizing the shape of the ambiguity sets for RMDPs. Our method departs from the conventional idea of constructing a norm-bounded uniform and symmetric ambiguity set. We instead argue that the structure of a near-optimal ambiguity set is problem specific. Our proposed method computes a weight parameter from the value functions, and these weights then drive the shape of the ambiguity sets. Our theoretical analysis demonstrates the rationale of the proposed idea. We apply our method to several different problem domains, and the empirical results further furnish the practical promise of weighted near-optimal ambiguity sets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/23/2019

High-Confidence Policy Optimization: Reshaping Ambiguity Sets in Robust MDPs

Robust MDPs are a promising framework for computing robust policies in r...
research
11/15/2018

Tight Bayesian Ambiguity Sets for Robust MDPs

Robustness is important for sequential decision making in a stochastic d...
research
02/20/2019

Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs

Robust MDPs (RMDPs) can be used to compute policies with provable worst-...
research
05/27/2022

Robust Phi-Divergence MDPs

In recent years, robust Markov decision processes (MDPs) have emerged as...
research
09/14/2020

First-Order Methods for Wasserstein Distributionally Robust MDP

Markov Decision Processes (MDPs) are known to be sensitive to parameter ...
research
01/03/2023

Risk-Averse MDPs under Reward Ambiguity

We propose a distributionally robust return-risk model for Markov decisi...
research
09/16/2021

Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learning

In batch reinforcement learning, there can be poorly explored state-acti...

Please sign up or login with your details

Forgot password? Click here to reset