DeepAI AI Chat
Log In Sign Up

An Adaptive State Aggregation Algorithm for Markov Decision Processes

07/23/2021
by   Guanting Chen, et al.
berkeley college
6

Value iteration is a well-known method of solving Markov Decision Processes (MDPs) that is simple to implement and boasts strong theoretical convergence guarantees. However, the computational cost of value iteration quickly becomes infeasible as the size of the state space increases. Various methods have been proposed to overcome this issue for value iteration in large state and action space MDPs, often at the price, however, of generalizability and algorithmic simplicity. In this paper, we propose an intuitive algorithm for solving MDPs that reduces the cost of value iteration updates by dynamically grouping together states with similar cost-to-go values. We also prove that our algorithm converges almost surely to within 2ε / (1 - γ) of the true optimal value in the ℓ^∞ norm, where γ is the discount factor and aggregated states differ by at most ε. Numerical experiments on a variety of simulated environments confirm the robustness of our algorithm and its ability to solve MDPs with much cheaper updates especially as the scale of the MDP problem increases.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/12/2022

Geometric Policy Iteration for Markov Decision Processes

Recently discovered polyhedral structures of the value function for fini...
06/20/2019

Max-Plus Matching Pursuit for Deterministic Markov Decision Processes

We consider deterministic Markov decision processes (MDPs) and apply max...
01/16/2014

Topological Value Iteration Algorithms

Value iteration is a powerful yet inefficient algorithm for Markov decis...
01/16/2015

Value Iteration with Options and State Aggregation

This paper presents a way of solving Markov Decision Processes that comb...
09/14/2020

First-Order Methods for Wasserstein Distributionally Robust MDP

Markov Decision Processes (MDPs) are known to be sensitive to parameter ...
06/09/2019

Toward Solving 2-TBSG Efficiently

2-TBSG is a two-player game model which aims to find Nash equilibriums a...
01/04/2019

Solving Markov Decision Processes with Reachability Characterization from Mean First Passage Times

A new mechanism for efficiently solving the Markov decision processes (M...