Toward Discovering Options that Achieve Faster Planning

05/25/2022
by   Yi Wan, et al.
0

We propose a new objective for option discovery that emphasizes the computational advantage of using options in planning. For a given set of episodic tasks and a given number of options, the objective prefers options that can be used to achieve a high return by composing few options. By composing few options, fast planning can be achieved. When faced with new tasks similar to the given ones, the discovered options are also expected to accelerate planning. Our objective extends the objective proposed by Harb et al. (2018) for the single-task setting to the multi-task setting. A closer look at Harb et al.'s objective shows that the best options discovered given one task are not likely to be useful for future unseen tasks and that the multi-task setting is indeed necessary for this purpose. In the same paper, Harb et al. also proposed an algorithm to optimize their objective, and the algorithm can be naturally extended to the multi-task setting. We empirically show that in the four-room domain the extension does not achieve a high objective value and propose a new algorithm that better optimizes the proposed objective. In the same four-room domain, we show that 1) a higher objective value is typically associated with options with which fewer planning iterations are needed to achieve near-optimal performance, 2) our new algorithm achieves a high objective value, which is close to the value achieved by a set of human-designed options, 3) the best number of planning iterations given the discovered options is much smaller and matches it obtained given human-designed options, and 4) the options produced by our algorithm also make intuitive sense because they move to and terminate at cells near hallways connecting two neighbor rooms.

READ FULL TEXT
research
02/12/2021

Discovery of Options via Meta-Learned Subgoals

Temporal abstractions in the form of options have been shown to help rei...
research
09/18/2016

Principled Option Learning in Markov Decision Processes

It is well known that options can make planning more efficient, among th...
research
10/16/2018

Finding Options that Minimize Planning Time

While adding temporally abstract actions, or options, to an agent's acti...
research
10/27/2018

Learning Abstract Options

Building systems that autonomously create temporal abstractions from dat...
research
01/10/2013

Decision-Theoretic Planning with Concurrent Temporally Extended Actions

We investigate a model for planning under uncertainty with temporallyext...
research
09/30/2022

Multi-Task Option Learning and Discovery for Stochastic Path Planning

This paper addresses the problem of reliably and efficiently solving bro...
research
12/03/2016

A Matrix Splitting Perspective on Planning with Options

We show that the Bellman operator underlying the options framework leads...

Please sign up or login with your details

Forgot password? Click here to reset