DeepAI
Log In Sign Up

Transience in Countable MDPs

12/26/2020
by   Stefan Kiefer, et al.
0

The Transience objective is not to visit any state infinitely often. While this is not possible in finite Markov Decision Process (MDP), it can be satisfied in countably infinite ones, e.g., if the transition graph is acyclic. We prove the following fundamental properties of Transience in countably infinite MDPs. 1. There exist uniformly ϵ-optimal MD strategies (memoryless deterministic) for Transience, even in infinitely branching MDPs. 2. Optimal strategies for Transience need not exist, even if the MDP is finitely branching. However, if an optimal strategy exists then there is also an optimal MD strategy. 3. If an MDP is universally transient (i.e., almost surely transient under all strategies) then many other objectives have a lower strategy complexity than in general MDPs. E.g., ϵ-optimal strategies for Safety and co-Büchi and optimal strategies for {0,1,2}-Parity (where they exist) can be chosen MD, even if the MDP is infinitely branching.

READ FULL TEXT

page 1

page 2

page 3

page 4

07/07/2020

Strategy Complexity of Parity Objectives in Countable MDPs

We study countably infinite MDPs with parity objectives. Unlike in finit...
04/24/2018

Learning-Based Mean-Payoff Optimization in an Unknown MDP under Omega-Regular Constraints

We formalize the problem of maximizing the mean-payoff value with high p...
04/25/2018

Distribution-based objectives for Markov Decision Processes

We consider distribution-based objectives for Markov Decision Processes ...
09/12/2022

Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach

In an Markov decision process (MDP), unobservable confounders may exist ...
04/25/2022

Strategy Synthesis for Global Window PCTL

Given a Markov decision process (MDP) M and a formula Φ, the strategy sy...
07/10/2020

Efficient MDP Analysis for Selfish-Mining in Blockchains

A proof of work (PoW) blockchain protocol distributes rewards to its par...
01/16/2020

Optimal by Design: Model-Driven Synthesis of Adaptation Strategies for Autonomous Systems

Many software systems have become too large and complex to be managed ef...