Model selection for contextual bandits

06/03/2019
by   Dylan J. Foster, et al.
0

We introduce the problem of model selection for contextual bandits, wherein a learner must adapt to the complexity of the optimal policy while balancing exploration and exploitation. Our main result is a new model selection guarantee for linear contextual bandits. We work in the stochastic realizable setting with a sequence of nested linear policy classes of dimension d_1 < d_2 < ..., where the m^-th class contains the optimal policy, and we design an algorithm that achieves Õ(T^2/3d^1/3_m^) regret with no prior knowledge of the optimal dimension d_m^. The algorithm also achieves regret Õ(T^3/4 + √(Td_m^)), which is optimal for d_m^≥√(T). This is the first contextual bandit model selection result with non-vacuous regret for all values of d_m^ and, to the best of our knowledge, is the first guarantee of its type in any contextual bandit setting. The core of the algorithm is a new estimator for the gap in best loss achievable by two linear policy classes, which we show admits a convergence rate faster than what is required to learn either class.

READ FULL TEXT
06/11/2021

Optimal Model Selection in Contextual Bandits with Many Classes via Offline Oracles

We study the problem of model selection for contextual bandits, in which...
06/19/2020

Open Problem: Model Selection for Contextual Bandits

In statistical learning, algorithms for model selection allow the learne...
07/05/2022

Instance-optimal PAC Algorithms for Contextual Bandits

In the stochastic contextual bandit setting, regret-minimizing algorithm...
12/12/2019

Sublinear Optimal Policy Value Estimation in Contextual Bandits

We study the problem of estimating the expected reward of the optimal po...
04/08/2021

Leveraging Good Representations in Linear Contextual Bandits

The linear contextual bandit literature is mostly focused on the design ...
12/23/2021

Model Selection in Batch Policy Optimization

We study the problem of model selection in batch policy optimization: gi...
07/21/2021

Design of Experiments for Stochastic Contextual Linear Bandits

In the stochastic linear contextual bandit setting there exist several m...