Near Optimal Provable Uniform Convergence in Off-Policy Evaluation for Reinforcement Learning

07/07/2020
by   Ming Yin, et al.
4

The Off-Policy Evaluation aims at estimating the performance of target policy π using offline data rolled in by a logging policy μ. Intensive studies have been conducted and the recent marginalized importance sampling (MIS) achieves the sample efficiency for OPE. However, it is rarely known if uniform convergence guarantees in OPE can be obtained efficiently. In this paper, we consider this new question and reveal the comprehensive relationship between OPE and offline learning for the first time. For the global policy class, by using the fully model-based OPE estimator, our best result is able to achieve ϵ-uniform convergence with complexity O(H^3·min(S,H)/d_mϵ^2), where d_m is an instance-dependent quantity decided by μ. This result is only one factor away from our uniform convergence lower bound up to a logarithmic factor. For the local policy class, ϵ-uniform convergence is achieved with the optimal complexity O(H^3/d_mϵ^2) in the off-policy setting. This result complements the work of sparse model-based planning (Agarwal et al. 2019) with generative model. Lastly, one interesting corollary of our intermediate result implies a refined analysis over simulation lemma.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

05/13/2021

Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings

This work studies the statistical limits of uniform convergence for offl...
01/29/2020

Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learning

We consider the problem of off-policy evaluation for reinforcement learn...
10/17/2021

Towards Instance-Optimal Offline Reinforcement Learning with Pessimism

We study the offline reinforcement learning (offline RL) problem, where ...
02/06/2022

Stochastic Gradient Descent with Dependent Data for Offline Reinforcement Learning

In reinforcement learning (RL), offline learning decoupled learning from...
05/26/2020

Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model

We investigate the sample efficiency of reinforcement learning in a γ-di...
02/21/2017

Sample Efficient Policy Search for Optimal Stopping Domains

Optimal stopping problems consider the question of deciding when to stop...
12/10/2020

Efficient Learning for Clustering and Optimizing Context-Dependent Designs

We consider a simulation optimization problem for a context-dependent de...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.