Model selection for contextual bandits

by   Dylan J. Foster, et al.

We introduce the problem of model selection for contextual bandits, wherein a learner must adapt to the complexity of the optimal policy while balancing exploration and exploitation. Our main result is a new model selection guarantee for linear contextual bandits. We work in the stochastic realizable setting with a sequence of nested linear policy classes of dimension d_1 < d_2 < ..., where the m^-th class contains the optimal policy, and we design an algorithm that achieves Õ(T^2/3d^1/3_m^) regret with no prior knowledge of the optimal dimension d_m^. The algorithm also achieves regret Õ(T^3/4 + √(Td_m^)), which is optimal for d_m^≥√(T). This is the first contextual bandit model selection result with non-vacuous regret for all values of d_m^ and, to the best of our knowledge, is the first guarantee of its type in any contextual bandit setting. The core of the algorithm is a new estimator for the gap in best loss achievable by two linear policy classes, which we show admits a convergence rate faster than what is required to learn either class.


Optimal Model Selection in Contextual Bandits with Many Classes via Offline Oracles

We study the problem of model selection for contextual bandits, in which...

Open Problem: Model Selection for Contextual Bandits

In statistical learning, algorithms for model selection allow the learne...

Instance-optimal PAC Algorithms for Contextual Bandits

In the stochastic contextual bandit setting, regret-minimizing algorithm...

Sublinear Optimal Policy Value Estimation in Contextual Bandits

We study the problem of estimating the expected reward of the optimal po...

Leveraging Good Representations in Linear Contextual Bandits

The linear contextual bandit literature is mostly focused on the design ...

Model Selection in Batch Policy Optimization

We study the problem of model selection in batch policy optimization: gi...

Design of Experiments for Stochastic Contextual Linear Bandits

In the stochastic linear contextual bandit setting there exist several m...