Lower Bounds for Policy Iteration on Multi-action MDPs

09/16/2020
by   Kumar Ashutosh, et al.
6

Policy Iteration (PI) is a classical family of algorithms to compute an optimal policy for any given Markov Decision Problem (MDP). The basic idea in PI is to begin with some initial policy and to repeatedly update the policy to one from an improving set, until an optimal policy is reached. Different variants of PI result from the (switching) rule used for improvement. An important theoretical question is how many iterations a specified PI variant will take to terminate as a function of the number of states n and the number of actions k in the input MDP. While there has been considerable progress towards upper-bounding this number, there are fewer results on lower bounds. In particular, existing lower bounds primarily focus on the special case of k = 2 actions. We devise lower bounds for k ≥ 3. Our main result is that a particular variant of PI can take Ω(k^n/2) iterations to terminate. We also generalise existing constructions on 2-action MDPs to scale lower bounds by a factor of k for some common deterministic variants of PI, and by log(k) for corresponding randomised variants.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

11/28/2019

Analysis of Lower Bounds for Simple Policy Iteration

Policy iteration is a family of algorithms that are used to find an opti...
06/03/2013

Improved and Generalized Upper Bounds on the Complexity of Policy Iteration

Given a Markov Decision Process (MDP) with n states and a totalnumber m ...
09/09/2011

mGPT: A Probabilistic Planner Based on Heuristic Search

We describe the version of the GPT planner used in the probabilistic tra...
01/23/2013

On the Complexity of Policy Iteration

Decision-making problems in uncertain or stochastic domains are often fo...
05/17/2018

Memoryless Exact Solutions for Deterministic MDPs with Sparse Rewards

We propose an algorithm for deterministic continuous Markov Decision Pro...
10/03/2020

Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions

We consider the problem of local planning in fixed-horizon Markov Decisi...
01/06/2022

Learning Optimal Antenna Tilt Control Policies: A Contextual Linear Bandit Approach

Controlling antenna tilts in cellular networks is imperative to reach an...

Code Repositories

PI_Lower_Bound

Repository for the IEEE CDC 2020 Paper "Lower Bounds for Policy Iteration on Multi-action MDPs"(https://arxiv.org/abs/2009.07842)


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.