DeepAI AI Chat
Log In Sign Up

GrASP: Gradient-Based Affordance Selection for Planning

by   Vivek Veeriah, et al.

Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for selecting affordances useful for planning – for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are goal-and-state-conditional mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive-action and option affordances, and that simultaneously learning to select affordances and planning with a learned value-equivalent model can outperform model-free RL.


Model-Based Planning in Discrete Action Spaces

Planning actions using learned and differentiable forward models of the ...

Discovery of Options via Meta-Learned Subgoals

Temporal abstractions in the form of options have been shown to help rei...

Options Discovery with Budgeted Reinforcement Learning

We consider the problem of learning hierarchical policies for Reinforcem...

Gradient-Based Mixed Planning with Discrete and Continuous Actions

Dealing with planning problems with both discrete logical relations and ...

Disentangling Options with Hellinger Distance Regularizer

In reinforcement learning (RL), temporal abstraction still remains as an...

TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning

Combining deep model-free reinforcement learning with on-line planning i...