On the Complexity of Value Iteration

07/13/2018
by   Nikhil Balaji, et al.
0

Value iteration is a fundamental algorithm for solving Markov Decision Processes (MDPs). It computes the maximal n-step payoff by iterating n times a recurrence equation which is naturally associated to the MDP. At the same time, value iteration provides a policy for the MDP that is optimal on a given finite horizon n. In this paper, we settle the computational complexity of value iteration. We show that, given a horizon n in binary and an MDP, computing an optimal policy is EXP-complete, thus resolving an open problem that goes back to the seminal 1987 paper on the complexity of MDPs by Papadimitriou and Tsitsiklis. As a stepping stone, we show that it is EXP-complete to compute the n-fold iteration (with n in binary) of a function given by a straight-line program over the integers with and + as operators.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/13/2018

On the Complexity of Iterative Tropical Computation with Applications to Markov Decision Processes

We study the complexity of evaluating powered functions implemented by s...
research
02/23/2023

Intermittently Observable Markov Decision Processes

This paper investigates MDPs with intermittent state information. We con...
research
01/28/2022

Planning and Learning with Adaptive Lookahead

The classical Policy Iteration (PI) algorithm alternates between greedy ...
research
07/11/2022

Cluster-Based Control of Transition-Independent MDPs

This work studies the ability of a third-party influencer to control the...
research
12/08/2016

Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning

We study the online estimation of the optimal policy of a Markov decisio...
research
01/24/2023

A Practitioner's Guide to MDP Model Checking Algorithms

Model checking undiscounted reachability and expected-reward properties ...
research
01/16/2014

Topological Value Iteration Algorithms

Value iteration is a powerful yet inefficient algorithm for Markov decis...

Please sign up or login with your details

Forgot password? Click here to reset