DeepAI AI Chat
Log In Sign Up

Skill-based Meta-Reinforcement Learning

by   Taewook Nam, et al.
KAIST 수리과학과
University of Southern California

While deep reinforcement learning methods have shown impressive results in robot learning, their sample inefficiency makes the learning of complex, long-horizon behaviors with real robot systems infeasible. To mitigate this issue, meta-reinforcement learning methods aim to enable fast learning on novel tasks by learning how to learn. Yet, the application has been limited to short-horizon tasks with dense rewards. To enable learning long-horizon behaviors, recent works have explored leveraging prior experience in the form of offline datasets without reward or task annotations. While these approaches yield improved sample efficiency, millions of interactions with environments are still required to solve complex tasks. In this work, we devise a method that enables meta-learning on long-horizon, sparse-reward tasks, allowing us to solve unseen target tasks with orders of magnitude fewer environment interactions. Our core idea is to leverage prior experience extracted from offline datasets during meta-learning. Specifically, we propose to (1) extract reusable skills and a skill prior from offline datasets, (2) meta-train a high-level policy that learns to efficiently compose learned skills into long-horizon behaviors, and (3) rapidly adapt the meta-trained policy to solve an unseen target task. Experimental results on continuous control tasks in navigation and manipulation demonstrate that the proposed method can efficiently solve long-horizon novel target tasks by combining the strengths of meta-learning and the usage of offline datasets, while prior approaches in RL, meta-RL, and multi-task RL require substantially more environment interactions to solve the tasks.


Skill-based Model-based Reinforcement Learning

Model-based reinforcement learning (RL) is a sample-efficient way of lea...

Demonstration-Guided Reinforcement Learning with Learned Skills

Demonstration-guided reinforcement learning (RL) is a promising approach...

Mixline: A Hybrid Reinforcement Learning Framework for Long-horizon Bimanual Coffee Stirring Task

Bimanual activities like coffee stirring, which require coordination of ...

Meta-Learning Transferable Parameterized Skills

We propose a novel parameterized skill-learning algorithm that aims to l...

Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks

In recent years, the robotics community has made substantial progress in...

Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks

In this paper, we study the problem of learning a repertoire of low-leve...

Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback

A promising approach to solving challenging long-horizon tasks has been ...