Learning Generalized Policies Without Supervision Using GNNs

05/12/2022
by   Simon Ståhlberg, et al.
0

We consider the problem of learning generalized policies for classical planning domains using graph neural networks from small instances represented in lifted STRIPS. The problem has been considered before but the proposed neural architectures are complex and the results are often mixed. In this work, we use a simple and general GNN architecture and aim at obtaining crisp experimental results and a deeper understanding: either the policy greedy in the learned value function achieves close to 100 larger than those used in training, or the failure must be understood, and possibly fixed, logically. For this, we exploit the relation established between the expressive power of GNNs and the C_2 fragment of first-order logic (namely, FOL with 2 variables and counting quantifiers). We find for example that domains with general policies that require more expressive features can be solved with GNNs once the states are extended with suitable "derived atoms" encoding role compositions and transitive closures that do not fit into C_2. The work follows the GNN approach for learning optimal general policies in a supervised fashion (Stahlberg, Bonet, Geffner, 2022); but the learned policies are no longer required to be optimal (which expands the scope, as many planning domains do not have general optimal policies) and are learned without supervision. Interestingly, value-based reinforcement learning methods that aim to produce optimal policies, do not always yield policies that generalize, as the goals of optimality and generality are in conflict in domains where optimal planning is NP-hard.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
09/21/2021

Learning General Optimal Policies with Graph Neural Networks: Expressive Power, Transparency, and Limits

It has been recently shown that general policies for many classical plan...
research
12/15/2020

General Policies, Serializations, and Planning Width

It has been observed that in many of the benchmark planning domains, ato...
research
01/03/2021

Learning General Policies from Small Examples Without Supervision

Generalized planning is concerned with the computation of general polici...
research
09/26/2020

Graph neural induction of value iteration

Many reinforcement learning tasks can benefit from explicit planning bas...
research
05/05/2020

Generalized Planning With Deep Reinforcement Learning

A hallmark of intelligence is the ability to deduce general principles f...
research
07/24/2023

Learning Resource Allocation Policy: Vertex-GNN or Edge-GNN?

Graph neural networks (GNNs) update the hidden representations of vertic...
research
05/10/2021

Expressing and Exploiting the Common Subgoal Structure of Classical Planning Domains Using Sketches: Extended Version

Width-based planning methods exploit the use of conjunctive goals for de...

Please sign up or login with your details

Forgot password? Click here to reset