DeepAI AI Chat
Log In Sign Up

Solving Markov Decision Processes with Reachability Characterization from Mean First Passage Times

by   Shoubhik Debnath, et al.
University of Southern California
Indiana University

A new mechanism for efficiently solving the Markov decision processes (MDPs) is proposed in this paper. We introduce the notion of reachability landscape where we use the Mean First Passage Time (MFPT) as a means to characterize the reachability of every state in the state space. We show that such reachability characterization very well assesses the importance of states and thus provides a natural basis for effectively prioritizing states and approximating policies. Built on such a novel observation, we design two new algorithms -- Mean First Passage Time based Value Iteration (MFPT-VI) and Mean First Passage Time based Policy Iteration (MFPT-PI) -- that have been modified from the state-of-the-art solution methods. To validate our design, we have performed numerical evaluations in robotic decision-making scenarios, by comparing the proposed new methods with corresponding classic baseline mechanisms. The evaluation results showed that MFPT-VI and MFPT-PI have outperformed the state-of-the-art solutions in terms of both practical runtime and number of iterations. Aside from the advantage of fast convergence, this new solution method is intuitively easy to understand and practically simple to implement.


page 1

page 3

page 5

page 7


Reachability and Differential based Heuristics for Solving Markov Decision Processes

The solution convergence of Markov Decision Processes (MDPs) can be acce...

Reachable Space Characterization of Markov Decision Processes with Time Variability

We propose a solution to a time-varying variant of Markov Decision Proce...

Conditional Value-at-Risk for Reachability and Mean Payoff in Markov Decision Processes

We present the conditional value-at-risk (CVaR) in the context of Markov...

Structured Reachability Analysis for Markov Decision Processes

Recent research in decision theoretic planning has focussed on making th...

The Complexity of Graph-Based Reductions for Reachability in Markov Decision Processes

We study the never-worse relation (NWR) for Markov decision processes wi...

Faster saddle-point optimization for solving large-scale Markov decision processes

We consider the problem of computing optimal policies in average-reward ...

Topological Value Iteration Algorithms

Value iteration is a powerful yet inefficient algorithm for Markov decis...