Model selection for contextual bandits

06/03/2019
by   Dylan J. Foster, et al.
0

We introduce the problem of model selection for contextual bandits, wherein a learner must adapt to the complexity of the optimal policy while balancing exploration and exploitation. Our main result is a new model selection guarantee for linear contextual bandits. We work in the stochastic realizable setting with a sequence of nested linear policy classes of dimension d_1 < d_2 < ..., where the m^-th class contains the optimal policy, and we design an algorithm that achieves Õ(T^2/3d^1/3_m^) regret with no prior knowledge of the optimal dimension d_m^. The algorithm also achieves regret Õ(T^3/4 + √(Td_m^)), which is optimal for d_m^≥√(T). This is the first contextual bandit model selection result with non-vacuous regret for all values of d_m^ and, to the best of our knowledge, is the first guarantee of its type in any contextual bandit setting. The core of the algorithm is a new estimator for the gap in best loss achievable by two linear policy classes, which we show admits a convergence rate faster than what is required to learn either class.

READ FULL TEXT
research
06/11/2021

Optimal Model Selection in Contextual Bandits with Many Classes via Offline Oracles

We study the problem of model selection for contextual bandits, in which...
research
06/19/2020

Open Problem: Model Selection for Contextual Bandits

In statistical learning, algorithms for model selection allow the learne...
research
12/23/2021

Model Selection in Batch Policy Optimization

We study the problem of model selection in batch policy optimization: gi...
research
02/12/2021

Pareto Optimal Model Selection in Linear Bandits

We study a model selection problem in the linear bandit setting, where t...
research
02/19/2023

Estimating Optimal Policy Value in General Linear Contextual Bandits

In many bandit problems, the maximal reward achievable by a policy is of...
research
12/12/2019

Sublinear Optimal Policy Value Estimation in Contextual Bandits

We study the problem of estimating the expected reward of the optimal po...
research
04/08/2021

Leveraging Good Representations in Linear Contextual Bandits

The linear contextual bandit literature is mostly focused on the design ...

Please sign up or login with your details

Forgot password? Click here to reset