DeepAI AI Chat
Log In Sign Up

Memoryless Exact Solutions for Deterministic MDPs with Sparse Rewards

by   Joshua R. Bertram, et al.
Iowa State University of Science and Technology

We propose an algorithm for deterministic continuous Markov Decision Processes with sparse rewards that computes the optimal policy exactly with no dependency on the size of the state space. The algorithm has time complexity of O( |R|^3 × |A|^2 ) and memory complexity of O( |R| × |A| ), where |R| is the number of reward sources and |A| is the number of actions. Furthermore, we describe a companion algorithm that can follow the optimal policy from any initial state without computing the entire value function, instead computing on-demand the value of states as they are needed. The algorithm to solve the MDP does not depend on the size of the state space for either time or memory complexity, and the ability to follow the optimal policy is linear in time and space with the path length of following the optimal policy from the initial state. We demonstrate the algorithm operation side by side with value iteration on tractable MDPs.


page 1

page 2

page 3

page 4


Explainable Deterministic MDPs

We present a method for a certain class of Markov Decision Processes (MD...

Fast Online Exact Solutions for Deterministic MDPs with Sparse Rewards

Markov Decision Processes (MDPs) are a mathematical framework for modeli...

Intermittently Observable Markov Decision Processes

This paper investigates MDPs with intermittent state information. We con...

Scalable Bilinear π Learning Using State and Action Features

Approximate linear programming (ALP) represents one of the major algorit...

First Order Decision Diagrams for Relational MDPs

Markov decision processes capture sequential decision making under uncer...

An exact solution in Markov decision process with multiplicative rewards as a general framework

We develop an exactly solvable framework of Markov decision process with...