Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

02/01/2021

∙

Finding the minimal structural assumptions that empower sample-efficient learning is one of the most important research directions in Reinforcement Learning (RL). This paper advances our understanding of this fundamental question by introducing a new complexity measure – Bellman Eluder (BE) dimension. We show that the family of RL problems of low BE dimension is remarkably rich, which subsumes a vast majority of existing tractable RL problems including but not limited to tabular MDPs, linear MDPs, reactive POMDPs, low Bellman rank problems as well as low Eluder dimension problems. This paper further designs a new optimization-based algorithm – GOLF, and reanalyzes a hypothesis elimination-based algorithm – OLIVE (proposed in Jiang et al. (2017)). We prove that both algorithms learn the near-optimal policies of low BE dimension problems in a number of samples that is polynomial in all relevant parameters, but independent of the size of state-action space. Our regret and sample complexity results match or improve the best existing results for several well-known subclasses of low BE dimension problems.

READ FULL TEXT

Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

Optimistic MLE – A Generic Model-based Algorithm for Partially Observable Sequential Decision Making

Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL

A General Framework for Sample-Efficient Function Approximation in Reinforcement Learning

Bilinear Classes: A Structural Framework for Provable Generalization in RL

Explicit Explore-Exploit Algorithms in Continuous State Spaces

Unified Algorithms for RL with Decision-Estimation Coefficients: No-Regret, PAC, and Reward-Free Learning

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithms

Related Research

Optimistic MLE – A Generic Model-based Algorithm for Partially Observable Sequential Decision Making

Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL

A General Framework for Sample-Efficient Function Approximation in Reinforcement Learning

Bilinear Classes: A Structural Framework for Provable Generalization in RL

Explicit Explore-Exploit Algorithms in Continuous State Spaces

Unified Algorithms for RL with Decision-Estimation Coefficients: No-Regret, PAC, and Reward-Free Learning

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting