Sample-Efficient Reinforcement Learning for Linearly-Parameterized MDPs with a Generative Model
share
The curse of dimensionality is a widely known issue in reinforcement learning (RL). In the tabular setting where the state space 𝒮 and the action space 𝒜 are both finite, to obtain a nearly optimal policy with sampling access to a generative model, the minimax optimal sample complexity scales linearly with |𝒮|×|𝒜|, which can be prohibitively large when 𝒮 or 𝒜 is large. This paper considers a Markov decision process (MDP) that admits a set of state-action features, which can linearly express (or approximate) its probability transition kernel. We show that a model-based approach (resp.Q-learning) provably learns an ε-optimal policy (resp.Q-function) with high probability as soon as the sample size exceeds the order of K/(1-γ)^3ε^2 (resp.K/(1-γ)^4ε^2), up to some logarithmic factor. Here K is the feature dimension and γ∈(0,1) is the discount factor of the MDP. Both sample complexity bounds are provably tight, and our result for the model-based approach matches the minimax lower bound. Our results show that for arbitrarily large-scale MDP, both the model-based approach and Q-learning are sample-efficient when K is relatively small, and hence the title of this paper.
READ FULL TEXT