Parameter-Independent Strategies for pMDPs via POMDPs

06/13/2018
by   Sebastian Arming, et al.
0

Markov Decision Processes (MDPs) are a popular class of models suitable for solving control decision problems in probabilistic reactive systems. We consider parametric MDPs (pMDPs) that include parameters in some of the transition probabilities to account for stochastic uncertainties of the environment such as noise or input disturbances. We study pMDPs with reachability objectives where the parameter values are unknown and impossible to measure directly during execution, but there is a probability distribution known over the parameter values. We study for the first time computing parameter-independent strategies that are expectation optimal, i.e., optimize the expected reachability probability under the probability distribution over the parameters. We present an encoding of our problem to partially observable MDPs (POMDPs), i.e., a reduction of our problem to computing optimal strategies in POMDPs. We evaluate our method experimentally on several benchmarks: a motivating (repeated) learner model; a series of benchmarks of varying configurations of a robot moving on a grid; and a consensus protocol.

READ FULL TEXT

page 1

page 2

page 3

page 4

09/28/2020

The Complexity of Reachability in Parametric Markov Decision Processes

This article presents the complexity of reachability decision problems f...
04/02/2019

On the Complexity of Reachability in Parametric Markov Decision Processes

This paper studies parametric Markov decision processes (pMDPs), an exte...
10/20/2017

Multi-Objective Approaches to Markov Decision Processes with Uncertain Transition Parameters

Markov decision processes (MDPs) are a popular model for performance ana...
08/24/2020

Qualitative Multi-Objective Reachability for Ordered Branching MDPs

We study qualitative multi-objective reachability problems for Ordered B...
05/17/2022

Sampling-Based Verification of CTMCs with Uncertain Rates

We employ uncertain parametric CTMCs with parametric transition rates an...
04/27/2018

Expectation Optimization with Probabilistic Guarantees in POMDPs with Discounted-sum Objectives

Partially-observable Markov decision processes (POMDPs) with discounted-...
03/16/2019

Parameter Synthesis for Markov Models

Markov chain analysis is a key technique in reliability engineering. A p...