DeepAI AI Chat
Log In Sign Up

GrASP: Gradient-Based Affordance Selection for Planning

02/08/2022
by   Vivek Veeriah, et al.
0

Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for selecting affordances useful for planning – for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are goal-and-state-conditional mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive-action and option affordances, and that simultaneously learning to select affordances and planning with a learned value-equivalent model can outperform model-free RL.

READ FULL TEXT
05/19/2017

Model-Based Planning in Discrete Action Spaces

Planning actions using learned and differentiable forward models of the ...
02/12/2021

Discovery of Options via Meta-Learned Subgoals

Temporal abstractions in the form of options have been shown to help rei...
11/21/2016

Options Discovery with Budgeted Reinforcement Learning

We consider the problem of learning hierarchical policies for Reinforcem...
10/19/2021

Gradient-Based Mixed Planning with Discrete and Continuous Actions

Dealing with planning problems with both discrete logical relations and ...
04/15/2019

Disentangling Options with Hellinger Distance Regularizer

In reinforcement learning (RL), temporal abstraction still remains as an...
10/31/2017

TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning

Combining deep model-free reinforcement learning with on-line planning i...