Lower Bounds for Policy Iteration on Multi-action MDPs

by   Kumar Ashutosh, et al.

Policy Iteration (PI) is a classical family of algorithms to compute an optimal policy for any given Markov Decision Problem (MDP). The basic idea in PI is to begin with some initial policy and to repeatedly update the policy to one from an improving set, until an optimal policy is reached. Different variants of PI result from the (switching) rule used for improvement. An important theoretical question is how many iterations a specified PI variant will take to terminate as a function of the number of states n and the number of actions k in the input MDP. While there has been considerable progress towards upper-bounding this number, there are fewer results on lower bounds. In particular, existing lower bounds primarily focus on the special case of k = 2 actions. We devise lower bounds for k ≥ 3. Our main result is that a particular variant of PI can take Ω(k^n/2) iterations to terminate. We also generalise existing constructions on 2-action MDPs to scale lower bounds by a factor of k for some common deterministic variants of PI, and by log(k) for corresponding randomised variants.



There are no comments yet.


page 1

page 2

page 3

page 4


Analysis of Lower Bounds for Simple Policy Iteration

Policy iteration is a family of algorithms that are used to find an opti...

Improved and Generalized Upper Bounds on the Complexity of Policy Iteration

Given a Markov Decision Process (MDP) with n states and a totalnumber m ...

mGPT: A Probabilistic Planner Based on Heuristic Search

We describe the version of the GPT planner used in the probabilistic tra...

On the Complexity of Policy Iteration

Decision-making problems in uncertain or stochastic domains are often fo...

Memoryless Exact Solutions for Deterministic MDPs with Sparse Rewards

We propose an algorithm for deterministic continuous Markov Decision Pro...

Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions

We consider the problem of local planning in fixed-horizon Markov Decisi...

Learning Optimal Antenna Tilt Control Policies: A Contextual Linear Bandit Approach

Controlling antenna tilts in cellular networks is imperative to reach an...

Code Repositories


Repository for the IEEE CDC 2020 Paper "Lower Bounds for Policy Iteration on Multi-action MDPs"(https://arxiv.org/abs/2009.07842)

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.