DeepAI AI Chat
Log In Sign Up

Robust and Adaptive Planning under Model Uncertainty

by   Apoorva Sharma, et al.
Stanford University

Planning under model uncertainty is a fundamental problem across many applications of decision making and learning. In this paper, we propose the Robust Adaptive Monte Carlo Planning (RAMCP) algorithm, which allows computation of risk-sensitive Bayes-adaptive policies that optimally trade off exploration, exploitation, and robustness. RAMCP formulates the risk-sensitive planning problem as a two-player zero-sum game, in which an adversary perturbs the agent's belief over the models. We introduce two versions of the RAMCP algorithm. The first, RAMCP-F, converges to an optimal risk-sensitive policy without having to rebuild the search tree as the underlying belief over models is perturbed. The second version, RAMCP-I, improves computational efficiency at the cost of losing theoretical guarantees, but is shown to yield empirical results comparable to RAMCP-F. RAMCP is demonstrated on an n-pull multi-armed bandit problem, as well as a patient treatment scenario.


page 1

page 2

page 3

page 4


Risk-Averse Bayes-Adaptive Reinforcement Learning

In this work, we address risk-averse Bayesadaptive reinforcement learnin...

Thompson Sampling for Gaussian Entropic Risk Bandits

The multi-armed bandit (MAB) problem is a ubiquitous decision-making pro...

POLY-HOOT: Monte-Carlo Planning in Continuous Space MDPs with Non-Asymptotic Analysis

Monte-Carlo planning, as exemplified by Monte-Carlo Tree Search (MCTS), ...

Risk-Constrained Thompson Sampling for CVaR Bandits

The multi-armed bandit (MAB) problem is a ubiquitous decision-making pro...

Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search

Bayes-optimal behavior, while well-defined, is often difficult to achiev...

A Minimax Tree Based Approach for Minimizing Detectability and Maximizing Visibility

We introduce and study the problem of planning a trajectory for an agent...

Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess

This paper introduces deep synoptic Monte Carlo planning (DSMCP) for lar...