On Efficient Online Imitation Learning via Classification

09/26/2022
by   Yichen Li, et al.
0

Imitation learning (IL) is a general learning paradigm for tackling sequential decision-making problems. Interactive imitation learning, where learners can interactively query for expert demonstrations, has been shown to achieve provably superior sample efficiency guarantees compared with its offline counterpart or reinforcement learning. In this work, we study classification-based online imitation learning (abbrev. COIL) and the fundamental feasibility to design oracle-efficient regret-minimization algorithms in this setting, with a focus on the general nonrealizable case. We make the following contributions: (1) we show that in the COIL problem, any proper online learning algorithm cannot guarantee a sublinear regret in general; (2) we propose Logger, an improper online learning algorithmic framework, that reduces COIL to online linear optimization, by utilizing a new definition of mixed policy class; (3) we design two oracle-efficient algorithms within the Logger framework that enjoy different sample and interaction round complexity tradeoffs, and conduct finite-sample analyses to show their improvements over naive behavior cloning; (4) we show that under the standard complexity-theoretic assumptions, efficient dynamic regret minimization is infeasible in the Logger framework. Our work puts classification-based online imitation learning, an important IL setup, into a firmer foundation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/23/2014

Reinforcement and Imitation Learning via Interactive No-Regret Learning

Recent work has demonstrated that problems-- particularly imitation lear...
research
12/03/2019

Continuous Online Learning and New Insights to Online Imitation Learning

Online learning is a powerful tool for analyzing iterative algorithms. H...
research
11/02/2010

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Sequential prediction problems such as imitation learning, where future ...
research
07/11/2023

Selective Sampling and Imitation Learning via Online Regression

We consider the problem of Imitation Learning (IL) by actively querying ...
research
06/12/2018

Model-Based Imitation Learning with Accelerated Convergence

Sample efficiency is critical in solving real-world reinforcement learni...
research
11/29/2020

Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning

Thompson sampling (TS) has emerged as a robust technique for contextual ...
research
05/07/2021

CoDE: Collocation for Demonstration Encoding

Roboticists frequently turn to Imitation learning (IL) for data efficien...

Please sign up or login with your details

Forgot password? Click here to reset