DeepAI AI Chat
Log In Sign Up

LISA: Learning Interpretable Skill Abstractions from Language

by   Divyansh Garg, et al.

Learning policies that effectually utilize language instructions in complex, multi-task environments is an important problem in imitation learning. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. To encode complex instructions into skills that can generalize to unseen instructions, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable skills from language-conditioned demonstrations. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA is able to outperform a strong non-hierarchical baseline in the low data regime and compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.


page 7

page 12

page 13

page 14

page 15

page 16

page 20

page 22


Divide Conquer Imitation Learning

When cast into the Deep Reinforcement Learning framework, many robotics ...

SKILL-IL: Disentangling Skill and Knowledge in Multitask Imitation Learning

In this work, we introduce a new perspective for learning transferable c...

Guiding Policies with Language via Meta-Learning

Behavioral skills or policies for autonomous agents are conventionally l...

Text2Motion: From Natural Language Instructions to Feasible Plans

We propose Text2Motion, a language-based planning framework enabling rob...

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

In recent years, much progress has been made in learning robotic manipul...

Hierarchical Few-Shot Imitation with Skill Transition Models

A desirable property of autonomous agents is the ability to both solve l...

Towards A Measure Of General Machine Intelligence

To build increasingly general-purpose artificial intelligence systems th...