Partial and Conditional Expectations in Markov Decision Processes with Integer Weights

02/12/2019
by   Jakob Piribauer, et al.
0

The paper addresses two variants of the stochastic shortest path problem ('optimize the accumulated weight until reaching a goal state') in Markov decision processes (MDPs) with integer weights. The first variant optimizes partial expected accumulated weights, where paths not leading to a goal state are assigned weight 0, while the second variant considers conditional expected accumulated weights, where the probability mass is redistributed to paths reaching the goal. Both variants constitute useful approaches to the analysis of systems without guarantees on the occurrence of an event of interest (reaching a goal state), but have only been studied in structures with non-negative weights. Our main results are as follows. There are polynomial-time algorithms to check the finiteness of the supremum of the partial or conditional expectations in MDPs with arbitrary integer weights. If finite, then optimal weight-based deterministic schedulers exist. In contrast to the setting of non-negative weights, optimal schedulers can need infinite memory and their value can be irrational. However, the optimal value can be approximated up to an absolute error of ϵ in time exponential in the size of the MDP and polynomial in (1/ϵ).

READ FULL TEXT
research
04/21/2022

The variance-penalized stochastic shortest path problem

The stochastic shortest path problem (SSPP) asks to resolve the non-dete...
research
04/30/2018

Stochastic Shortest Paths and Weight-Bounded Properties in Markov Decision Processes

The paper deals with finite-state Markov decision processes (MDPs) with ...
research
04/23/2020

On Skolem-hardness and saturation points in Markov decision processes

The Skolem problem and the related Positivity problem for linear recurre...
research
02/27/2023

Positivity-hardness results on Markov decision processes

This paper investigates a series of optimization problems for one-counte...
research
05/09/2023

Graph-Based Reductions for Parametric and Weighted MDPs

We study the complexity of reductions for weighted reachability in param...
research
06/07/2022

Concentration bounds for SSP Q-learning for average cost MDPs

We derive a concentration bound for a Q-learning algorithm for average c...
research
04/20/2018

Metrics that respect the support

In this work we explore the family of metrics determined by S-weights, i...

Please sign up or login with your details

Forgot password? Click here to reset