Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes

by   A. Fern, et al.

We study an approach to policy selection for large relational Markov Decision Processes (MDPs). We consider a variant of approximate policy iteration (API) that replaces the usual value-function learning step with a learning step in policy space. This is advantageous in domains where good policies are easier to represent and learn than the corresponding value functions, which is often the case for the relational MDPs we are interested in. In order to apply API to such problems, we introduce a relational policy language and corresponding learner. In addition, we introduce a new bootstrapping routine for goal-based planning domains, based on random walks. Such bootstrapping is necessary for many large relational MDPs, where reward is extremely sparse, as API is ineffective in such domains when initialized with an uninformed policy. Our experiments show that the resulting system is able to find good policies for a number of classical planning domains and their stochastic variants by solving them as extremely large relational MDPs. The experiments also point to some limitations of our approach, suggesting future work.



page 1

page 2

page 3

page 4


Inductive Policy Selection for First-Order MDPs

We select policies for large Markov Decision Processes (MDPs) with compa...

Bootstrapping Skills

The monolithic approach to policy representation in Markov Decision Proc...

Automatic Induction of Bellman-Error Features for Probabilistic Planning

Domain-specific features are important in representing problem structure...

First Order Decision Diagrams for Relational MDPs

Markov decision processes capture sequential decision making under uncer...

Representation Policy Iteration

This paper addresses a fundamental issue central to approximation method...

Generalized Neural Policies for Relational MDPs

A Relational Markov Decision Process (RMDP) is a first-order representat...

Solving Relational MDPs with Exogenous Events and Additive Rewards

We formalize a simple but natural subclass of service domains for relati...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.