DeepAI AI Chat
Log In Sign Up

Optimistic Initialization and Greediness Lead to Polynomial Time Learning in Factored MDPs - Extended Version

by   Istvan Szita, et al.

In this paper we propose an algorithm for polynomial-time reinforcement learning in factored Markov decision processes (FMDPs). The factored optimistic initial model (FOIM) algorithm, maintains an empirical model of the FMDP in a conventional way, and always follows a greedy policy with respect to its model. The only trick of the algorithm is that the model is initialized optimistically. We prove that with suitable initialization (i) FOIM converges to the fixed point of approximate value iteration (AVI); (ii) the number of steps when the agent makes non-near-optimal decisions (with respect to the solution of AVI) is polynomial in all relevant quantities; (iii) the per-step costs of the algorithm are also polynomial. To our best knowledge, FOIM is the first algorithm with these properties. This extended version contains the rigorous proofs of the main theorem. A version of this paper appeared in ICML'09.


page 1

page 2

page 3

page 4


Polynomial Time Reinforcement Learning in Correlated FMDPs with Linear Value Functions

Many reinforcement learning (RL) environments in practice feature enormo...

MDPs with Unawareness

Markov decision processes (MDPs) are widely used for modeling decision-m...

Efficient Strategy Iteration for Mean Payoff in Markov Decision Processes

Markov decision processes (MDPs) are standard models for probabilistic s...

Polynomial Value Iteration Algorithms for Detrerminstic MDPs

Value iteration is a commonly used and empirically competitive method in...

Optimal probabilistic polynomial time compression and the Slepian-Wolf theorem: tighter version and simple proofs

We give simplify the proofs of the 2 results in Marius Zimand's paper "K...

Warehouse Problem with Bounds, Fixed Costs and Complementarity Constraints

This paper studies an open question in the warehouse problem where a mer...

Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learning

This study addresses the problem of selecting dynamically, at each time ...