Automatic Induction of Bellman-Error Features for Probabilistic Planning

by   Jia-Hong Wu, et al.

Domain-specific features are important in representing problem structure throughout machine learning and decision-theoretic planning. In planning, once state features are provided, domain-independent algorithms such as approximate value iteration can learn weighted combinations of those features that often perform well as heuristic estimates of state value (e.g., distance to the goal). Successful applications in real-world domains often require features crafted by human experts. Here, we propose automatic processes for learning useful domain-specific feature sets with little or no human intervention. Our methods select and add features that describe state-space regions of high inconsistency in the Bellman equation (statewise Bellman error) during approximate value iteration. Our method can be applied using any real-valued-feature hypothesis space and corresponding learning method for selecting features from training sets of state-value pairs. We evaluate the method with hypothesis spaces defined by both relational and propositional feature languages, using nine probabilistic planning domains. We show that approximate value iteration using a relational feature space performs at the state-of-the-art in domain-independent stochastic relational planning. Our method provides the first domain-independent approach that plays Tetris successfully (without human-engineered features).



There are no comments yet.


page 1

page 2

page 3

page 4


Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes

We study an approach to policy selection for large relational Markov Dec...

Fitted Q-Learning for Relational Domains

We consider the problem of Approximate Dynamic Programming in relational...

Probabilistic Relational Planning with First Order Decision Diagrams

Dynamic programming algorithms have been successfully applied to proposi...

Lifted Model Checking for Relational MDPs

Model checking has been developed for verifying the behaviour of systems...

Structure Selection from Streaming Relational Data

Statistical relational learning techniques have been successfully applie...

Solving Relational MDPs with Exogenous Events and Additive Rewards

We formalize a simple but natural subclass of service domains for relati...

Value Iteration Networks on Multiple Levels of Abstraction

Learning-based methods are promising to plan robot motion without perfor...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.