Adaptive Exploration in Linear Contextual Bandit

1 Introduction

The contextual linear bandit is a practical setting for many sequential decision-making problems, especially in on-line applications (Agarwal et al., 2009; Li et al., 2010). Our main contribution is a new algorithm that is asymptotically optimal, computationally efficient and empirically well-behaved in finite-time regimes. As a consequence of asymptotic optimality, the algorithm adapts to certain easy cases where it achieves sub-logarithmic regret.

Popular approaches for regret minimising in contextual bandits include $ϵ$ -greedy (Langford and Zhang, 2007), explicit optimism-based algorithms (Dani et al., 2008; Rusmevichientong and Tsitsiklis, 2010; Chu et al., 2011; Abbasi-Yadkori et al., 2011)

, and implicit ones, such as Thompson sampling

(Agrawal and Goyal, 2013). Although these algorithms enjoy near-optimal worst-case guarantees and can be quite practical, they are known to be arbitrarily suboptimal in the asymptotic regime, even in the non-contextual linear bandit (Lattimore and Szepesvari, 2017).

We propose an optimization-based algorithm that estimates and tracks the optimal allocation for each context/action pair. This technique is most well known for its effectiveness in pure exploration

(Chan and Lai, 2006; Garivier and Kaufmann, 2016; Degenne et al., 2019, and others). The approach has been used in regret minimisation in linear bandits with fixed action sets (Lattimore and Szepesvari, 2017) and structured bandits (Combes et al., 2017). The last two articles provide algorithms for the non-contextual case and hence cannot be applied directly to our setting. More importantly, however, the algorithms are not practical. The first algorithm uses a complicated three-phase construction that barely updates its estimates. The second algorithm is not designed to handle large action spaces and has a ‘lower-order’ term in the regret that depends linearly on the number of actions and dominates the regret in all practical regimes. This lower-order term is not merely a product of the analysis, but also reflected in the experiments (see Section 5.4 for details).

The most related work is by Ok et al. (2018)

who study a reinforcement learning setting. A stochastic contextual bandit can be viewed as a Markov decision process where the state represents the context and the transition is independent of the action. The structured nature of the mentioned paper means our setting is covered by their algorithm. Again, however, the algorithm is too general to exploit the specific structure of the contextual bandit problem. Their algorithm is asymptotically optimal, but suffers from lower-order terms that are linear in the number of actions and dominate the regret in all practically interesting regimes. In contrast, our algorithm is asymptotically optimal, but also practical in finite-horizon regimes, as will be demonstrated by our experiments.

The contextual linear bandit also serves as an interesting example where the asymptotics of the problem are not indicative of what should be expected in finite-time (see the second scenario in Section 5.2). This is in contrast to many other bandit models where the asymptotic regret is also roughly optimal in finite time (Lattimore and Szepesvári, 2019). There is an important lesson here. Designing algorithms that optimize for the asymptotic regret may make huge sacrifices in finite-time.

Another interesting phenomenon is related to the idea of ‘natural exploration’ that occurs in contextual bandits (Bastani et al., 2017; Kannan et al., 2018). A number of authors have started to investigate the striking performance of greedy algorithms in contextual bandits. In most bandit settings the greedy policy does not explore sufficiently and suffers linear regret. In some contextual bandit problems, however, the changing features ensure the algorithm cannot help but explore. Our algorithm and analysis highlights this effect (see Section 3.1 for details). If the context distribution is sufficiently rich, then the algorithm is eventually almost completely greedy and enjoys sub-logarithmic regret. An advantage of our approach is that we do not need strong assumptions on the context distribution: the algorithm adapts to the problem in a data-dependent fashion in the sense that even when the context distribution is not sufficiently rich, we preserve the usual optimality guarantee. As another contribution we also prove that algorithms based on optimism enjoy sub-logarithmic regret in this setting (Section 3.1).

The rest of the paper is organized as follows. Section 2 introduces the basic problem setup. Section 3 studies the asymptotic regret lower bound for linear contextual bandit. Section 4 introduces our optimal allocation matching algorithm and presents the asymptotic regret upper bound. Section 5 conducts several experiments. Section 6 discusses interesting directions for future works.

Notation

Let $[n] = {1, 2, \dots, n}$ . For a matrix $A$

and vector

x

, we denote

∥ x ∥_{A} = \sqrt{x^{⊤} A x}

. The cardinality of a set

\cA

is denoted by

|\cA|

2 Problem Setting

We consider the stochastic $K$ -armed contextual linear bandit with a horizon of $n$ rounds and $M$ possible contexts. The assumption that the contexts are discrete is often reasonable in practice. For instance, in a recommender system users are often clustered into finitely many user groups. For each context $m \in [M]$ there is a known feature/action set $\cAm⊂Rd$ with $|\cAm|=K$ . The interaction protocol is as follows. First the environment samples a sequence of independent contexts $(c_{t})_{t = 1}^{n}$ from an unknown distribution $p$ over $[M]$

and each context is assumed to appear with positive probability. At the start of round

t

the context

c_{t}

is revealed to the learner, who may use their observations to choose an action

Xt∈\cAt=\cAct

. The reward is

Y_{t} = ⟨ X_{t}, θ ⟩ + η_{t},

where $(η_{t})_{t = 1}^{n}$

is a sequence of independent standard Gaussian random variables and

θ \in R^{d}

is an unknown parameter. The Gaussian assumption can be relaxed to conditional sub-Gaussian assumption for the regret upper bound, but is necessary for the regret lower bound. Throughout, we consider a frequentist setting in the sense that

θ

is fixed. For simplicity, we assume each

\cAm

spans

R^{d}

and

∥ x ∥_{2} \leq 1

for all

x∈∪m\cAm

The performance metric is the cumulative expected regret, which measures the difference between the expected cumulative reward collected by the omniscient policy that knows $θ$ and the learner’s expected cumulative reward. The optimal arm associated with context $m$ is $x∗m=\argmaxx∈\cAm⟨x,θ⟩$ . Then the expected cumulative regret of a policy $π$ when facing the bandit determined by $θ$ is

R_{θ}^{π} (n) = E [n \sum t = 1 ⟨ x_{c_{t}}^{*}, θ ⟩ - n \sum t = 1 Y_{t}] .

Note that this cumulative regret also depends on the context distribution $p$ and action sets. They are omitted from the notation to reduce clutter and because there will never be ambiguity.

3 Asymptotic Lower Bound

We investigate the fundamental limit of linear contextual bandit by deriving its instance-dependent asymptotic lower bound. First, we define the class of policies that are taken into consideration.

[Consistent Policy] A policy $π$ is called consistent if the regret is subpolynomial for any bandit in that class and all context distributions:

R_{θ}^{π} (n) = o (n^{ϵ}), for all ϵ > 0 and % all θ \in R^{d} .

(1)

The next lemma is the key ingredient in proving the asymptotic lower bound. Given a context $m$ and $x∈\cAm$ let $Δ_{x}^{m} = ⟨ x_{m}^{*} - x, θ ⟩$ be the suboptimality gap. Furthermore, let $Δmin=minm∈[M]minx∈\cAm,Δmx>0Δmx$ .

Assume that $p (m) > 0$ for all $m \in [M]$ and that $x_{m}^{*}$ is uniquely defined for each context $m$ and let $π$ be consistent. Then for sufficiently large $n$ the expected covariance matrix

{¯ G}_{n} = E [n \sum t = 1 X_{t} X_{t}^{⊤}],

(2)

is invertible. Furthermore, for any context $m$ and any arm $x∈\cAm$ ,

limsup n \to \infty log (n) ∥ ∥ x - x_{m}^{*} {∥ ∥}_{{¯ G}_{n}^{- 1}}^{2} \leq \frac{(Δ_{x}^{m})^{2}}{2} .

(3)

The proof is deferred to Appendix A.1 in the supplementary material. Intuitively, the lemma shows that any consistent policy must collect sufficient statistical evidence at confidence level $1 - 1 / n$

that suboptimal arms really are suboptimal. This corresponds to ensuring that the width of an appropriate confidence interval

\sqrt{2 log (n)} ∥ x - x_{m}^{*} ∥_{{¯ G}_{n}^{- 1}}

is approximately smaller than the sub-optimality gap

Δ_{x}^{m}

[Asymptotic Lower Bound] Under the same conditions as Section 3,

liminfn→∞Rπθ(n)log(n)≥\cC(θ,\cA1,…,\cAM),

(4)

where $\cC(θ,\cA1,…,\cAM)$ is defined as the optimal value of the following optimization problem:

infαx,m∈[0,∞]M∑m=1∑x∈\cAmαx,mΔmx

(5)

subject to the constraint that for any context $m$ and suboptimal arm $x∈\cAm$ ,

x⊤⎛⎝M∑m=1∑x∈\cAmαx,mxx⊤⎞⎠−1x≤(Δmx)22.

(6)

Given the result in Lemma 3, the proof of Theorem 3 follows exactly the same idea of the proof of Corollary 2 in Lattimore and Szepesvari (2017) and thus is omitted here. Later on we will prove a matching upper bound in Theorem 4.2 and argue that our asymtotical lower bound is sharp.

In the above we adopt the convention that $\infty \times 0 = 0$ so that $α_{x, m} Δ_{x}^{m} = 0$ whenever $Δ_{x}^{m} = 0$ . The inverse of a matrix with infinite entries is defined by passing to the limit in the obvious way, and is not technically an inverse.

Let us denote ${α∗x,m}x∈\cAm,m∈[M]$ as an optimal solution to the above optimization problem. It serves as the optimal allocation rule for each arm such that the cumulative regret is minimized subject to the width of the confidence interval of each sub-optimal arm is small. Specifically, $α_{x, m}^{*} log (n)$ can be interpreted as the approximate optimal number of times arm $x$ should be played having observed context $m$ .

Our lower bound may also be derived from the general bound in Ok et al. (2018), since a stochastic contextual bandit can be viewed as a kind of Markov decision process. We use an alternative proof technique and the two lower bound statements have different forms. The proof is included for completeness.

When $M = 1$ and $\cA1={e1,…,ed}$ is the standard basis vectors, the problem reduces to classical multi-armed bandit and $\cC(θ,\cA1)=∑x∈\cA1,Δx>02/Δx$ , which matches the well-known asymptotic lower bound by Lai and Robbins (1985).

The constant $\cC(θ,\cA1,…,\cAM)$ depends on both the unknown parameter $θ$ and the action sets $\cA1,…,\cAM$ , but not the context distribution $p$ . In this sense there is a certain discontinuity in the hardness measure $\cC$ as a function of the context distribution. More precisely, problems where $p (m)$ is arbitrarily close to zero may have different regret asymptotically than the problem obtained by removing context $m$ entirely. Clearly as $p (m)$ tends to zero the $m$ th context is observed with vanishingly small probability in finite time and hence the asymptotically optimal regret may not be representative of the finite-time hardness.

3.1 Sub-logarithmic regret

Our matching upper and lower bounds reveal the interesting phenomenon that if the action sets satisfy certain conditions, then sub-logarithmic regret is possible. Consider the scenario that the set of optimal arms ${x_{1}^{*}, \dots, x_{M}^{*}}$ spans $R^{d}$ . Let $Λ \in R$ be a large constant to be defined subsequently and for each context $m$ and arm $x∈\cAm$ let

α_{x, m} = {\begin{matrix} 0 & if x \neq x_{m}^{*} Λ & else . \end{matrix}

Then,

M∑m=1∑x∈\cAmαx,mxx⊤=ΛM∑m=1x∗mx∗⊤m.

(7)

Since the set of optimal arms spans $R^{d}$ it holds that for any context $m$ and arm $x∈\cAm$ ,

x^{⊤} {(M \sum m = 1 x_{m}^{*} x_{m}^{* ⊤})}^{- 1} x < \infty .

(8)

Combining Eq. 7 and Eq. 8,

x⊤⎛⎝M∑m=1∑x∈\cAmαx,mxx⊤⎞⎠−1x=Λ−1x⊤(M∑m=1x∗mx∗⊤m)−1x.

Hence, the constraint in Eq. 6 is satisfied for sufficiently large $Λ$ . Since with this choice of $(α_{x, m})$ we have $∑Mm=1∑x∈\cAmαx,mΔmx=0$ , it follows that $\cC(θ,\cA1,…,\cAM)=0$ . Therefore our upper bound will show that when the set of optimal actions ${x_{1}^{*}, \dots, x_{M}^{*}}$ spans $R^{d}$ our new algorithm satisfies

liminf n \to \infty \frac{R_{θ}^{π} (n)}{log (n)} = 0 .

The choice of $α_{x, m}$ above shows that when ${x_{1}^{*}, \dots, x_{M}^{*}}$ span $R^{d}$ , then an asymptotically optimal algorithm only needs to play suboptimal arms sub-logarithmically often, which means the algorithm is eventually very close to the greedy algorithm. Bastani et al. (2017); Kannan et al. (2018) also investigate the striking performance of greedy algorithms in contextual bandits. However, Bastani et al. (2017) assume the covariate diversity on the context distribution while Kannan et al. (2018) assume the context is artificially perturbed with a noise. Both conditions are hard to check in practice. In addition, Bastani et al. (2017) only provide a rate-optimal algorithm while our algorithm is optimal in constants (see Theorem 4.2 for details).

As claimed in the introduction, we also prove that algorithms based on optimism can enjoy bounded regret when the set of optimal actions spans the space of all actions. The proof of the following theorem is given in LABEL:sec:thm:ucb.

Consider the policy $π$ that plays optimistically by

Xt=\argmaxx∈\cAct⟨^θt−1,x⟩+∥x∥G−1tβ1/2t.

Suppose that $θ$ is such that ${x_{1}^{*}, \dots, x_{M}^{*}}$ spans $R^{d}$ . Then, for suitable $(β_{t})_{t = 1}^{n}$ with $β_{t} = O (d log (t))$ , the expected regret ${limsup}_{n \to \infty} R_{θ}^{π} (n) < \infty$ .

Note, the choice of $(β_{t})$ for which the above theorem holds also guarantees the standard $~ O (d \sqrt{n})$ minimax bound for this algorithm, showing that LinUCB can adapt online to this nice case.

4 Optimal Allocation Matching

The instance-dependent asymptotic lower bound provides an optimal allocation rule. However, the optimal allocation ${α_{x, m}^{*}}_{x, m}$ depends on the unknown sub-optimality gap. In this section, we present a novel matching algorithm that simultaneously estimates the unknown parameter $θ$ using least squares and updates the allocation rule.

4.1 Algorithm

Let $N_{x} (t) = \sum_{s = 1}^{t} I (X_{s} = x)$ be the number of pulls of arm $x$ after round $t$ and $G_{t} = \sum_{s = 1}^{t} X_{s} X_{s}^{⊤}$ . The least squares estimator is ${^θ}_{t} = G_{t}^{- 1} \sum_{x = 1}^{t} X_{s} Y_{s}$ . For each context $m$ the estimated sub-optimality gap of arm $x∈\cAm$ is $^Δmx(t)=maxy∈\cAm⟨y−x,^θt⟩$ and the estimated optimal arm is $^x∗m(t)=\argmaxx∈\cAm⟨x,^θt⟩$ . The minimum nonzero estimated gap is

^Δmin(t)=minm∈[M]minx∈\cAm,^Δmx(t)>0^Δmx(t).

Next we define a similar optimization problem as in (5) but with a different normalisation.

Let $f_{n, δ}$ be the constant given by

f_{n, δ} = 2 (1 + 1 / log (n)) log (1 / δ) + c d log (d log (n)),

(9)

where $c$ is an absolute constant. We write $f_{n} = f_{n, 1 / n}$ . For any $~Δ∈[0,∞)|∪m\cAm|$ define $T (~ Δ)$ as a solution of the following optimization problem:

min(Tmx)x,m∈[0,∞]M∑m=1∑x∈\cAmTmx~Δmx,

(10)

subject to

∥x∥2H−1T≤Δ2xfn,∀x∈\cAm,m∈[M].

and that $HT=∑Mm=1∑x∈\cAmTmxxx⊤$ is invertible.

If $~ Δ$ is an estimate of $Δ$ , we call the solution $T (~ Δ)$ an approximated allocation rule in contrast to the optimal allocation rule defined in Remark 3. Our algorithm alternates between exploration and exploitation, depending on whether or not all the arms have satisfied the approximated allocation rule. We are now ready to describe the algorithm, which starts with a brief initialisation phase.

Initialisation

In the first $d$ rounds the algorithm chooses any action $X_{t}$ in the action set such that $X_{t}$ is not in the span of ${X_{1}, \dots, X_{t - 1}}$ . This is always possible by the assumption that $\cAm$ spans $R^{d}$ for all contexts $m$ . At the end of the initialisation phase $G_{t}$ is guaranteed to be invertible.

Main phase

In each round after the initialisation phase the algorithm checks if the following criterion holds for any $x∈\cAct$ :

∥x∥2G−1t−1≤max{(^Δmin(t−1))2fn,(^Δctx(t−1))2fn}.

(11)

The algorithm exploits if Eq. 11 holds and explores otherwise, as explained below.

Exploitation.

The algorithm exploits by taking the greedy action:

Xt=\argmaxx∈\cActx⊤^θt−1.

(12)

Exploration.

The algorithm explores when Eq. 11 does not hold. This means that some actions have not been explored sufficiently. There are two cases to consider. First, when there exists an $x′∈\cAct$ such that

N_{x^{'}} (t - 1) < min (T_{x^{'}}^{c_{t}} (^Δ (t - 1)), f_{n} / {^Δ}_{min}^{2} (t - 1)),

the algorithm then computes two actions

	$b_{1}$	$=\argminx∈\cActNx(t−1)min(Tctx(^Δ(t−1)),fn/^Δ2min(t−1)),$
	$b_{2}$	$=\argminx∈\cActNx(t−1).$		(13)

Let $s (t)$ be the number of exploration rounds defined in Algorithm 1. If $N_{b_{2}} (t - 1) \leq ε_{t} s (t)$ the algorithm plays arm $X_{t} = b_{2}$ that executes the forced exploration. Otherwise it plays arm $X_{t} = b_{1}$ . Finally, rounds where there does not exist an $x′∈\cAct$ are called wasted. In these rounds the algorithm acts optimistically as LinUCB (Abbasi-Yadkori et al., 2011):

Xt=\argmaxx∈\cActx⊤^θt−1+√fn,1/(s(t))2∥x∥G−1t−1,

(14)

where $f_{n, 1 / (s (t))^{2}}$ is defined in Eq. 9. The complete algorithm is presented in Algorithm 1.

The naive forced exploration can be improved by calculating a barycentric spanner (Awerbuch and Kleinberg, 2008) for each action set and then playing the least played action in the spanner. In normal practical setups this makes very little difference, where the forced exploration plays a limited role. For finite-time worst-case analysis, however, it may be crucial, since otherwise the regret may depend linearly on the number of actions, while using the spanner guarantees the forced exploration is sample efficient.

4.2 Asymptotic Upper Bound

Our main theorem is that Algorithm 1 is asymptotically optimal under mild assumptions. Suppose that $T_{x}^{m} (Δ)$ is uniquely defined and $T_{x}^{m} (\cdot)$ is continuous at $Δ$ for all contexts $m$ and actions $x∈\cAm$ . Then the policy $π_{oam}$ proposed in Algorithm 1 with $ϵ_{t} = 1 / log (log (t))$ satisfies

limsupn→∞Rπoamθ(n)log(n)≤\cC(θ,\cA1,…,\cAM).

(15)

Together with the asymptotic lower bound in Theorem 3, we can argue that optimal allocation matching algorithm is asymptotical optimal and the lower bound (4) is sharp. The assumption that $T_{x}^{m} (\cdot)$ is continuous at $Δ$ is used to ensure the stability of our algorithm. We prove that the uniqueness assumption actually implies the continuity (LABEL:lem:cont in the supplementary material). There are, however, certain corner cases where uniqueness does not hold. For example when $θ=(1,0)⊤,\cA={(1,0),(0,1),(0,−1)}$ .

Input: exploration parameter

ε_{t}

, exploration counter

s (d) = 0

# initialisation for $t = 1$ to $d$ do

Observe an action set

\cAct

, pull arm

X_{t}

such that

X_{t}

is not in the span of

{X_{1}, \dots, X_{t - 1}}

end for

for $t = d + 1$ to $n$ do

Observe an action set

\cAct

and compute the optimization problem (10) based on the estimated gap

^Δ (t - 1)

. if $∥x∥2G−1t−1≤max{^Δ2min(t−1)fn,(^Δctx(t−1))2fn},∀x∈\cAct$ , then

# exploitation Pull arm

Xt=\argmaxx∈\cActx⊤^θt−1

else

# exploration

s (t) = s (t - 1) + 1

if $Nx(t−1)≥min(Tx(^Δ(t−1)),fn/(^Δmin(t−1)))2,∀x∈\cAct$ , then

Pull arm according to LinUCB in (14).

else

Calculate

b_{1}, b_{2}

in (13). if $N_{b_{2}} (t - 1) \leq ε_{t} s (t - 1)$ then

Pull arm

X_{t} = b_{2}

else

Pull arm

X_{t} = b_{1}

end if

Update

{^θ}_{t}, {^Δ}_{x}^{c_{t}} (t), {^Δ}_{min} (t)

end for

Algorithm 1 Optimal Allocation Matching

4.3 Proof Sketch

The complete proof is deferred to Appendix A.2 in the supplementary material. At a high level the analysis of the optimization-based approach consists of three parts. (1) Showing that the algorithm’s estimate of the true parameter is close to the truth in finite time. (2) Showing that the algorithm subsequently samples arms approximately according to the unknown optimal allocation and (3) Showing that the greedy action when arms have been sampled sufficiently according to the optimal allocation is optimal with high probability. Existing optimization-based algorithms suffer from dominant ‘lower-order’ terms because they use simple empirical means for Part (1), while here we use the data-efficient least-squares estimator.

We denote $Explore = F-Explore \cup UW-Explore \cup W-Explore$ as the set of exploration rounds, decomposed into disjoint sets of forced exploration $(X_{t} = b_{1})$ , unwasted exploration $(X_{t} = b_{2})$ and wasted exploration (LinUCB), and let Exploit be the set of exploitation rounds.

Regret while exploiting

The criterion in Eq. 11 guarantees that the greedy action is optimal with high probability in exploitation rounds. To see this, note that if $t$ is an exploitation round, then the sub-optimality gap of greedy action $X_{t}$ satisfies the following with high probability:

Δ_{X_{t}}^{c_{t}} ≲ \sqrt{\frac{log (log (n))}{1 \lor N_{X_{t}} (t - 1)}} < Δ_{min} .

Since the instantaneous regret either vanishes or is larger than $Δ_{min}$ , we have

E ⎡ ⎣ \sum t \in Exploit Δ_{t}^{c_{t}} ⎤ ⎦ = o (log (n)) .

Regret while exploring

Based on the design of our algorithm, the regret while exploring is decomposed into three kinds of explorations,

E ⎡ ⎣ \sum t \in Explore Δ_{X_{t}}^{c_{t}} ⎤ ⎦ = E ⎡ ⎣ \sum t \in F-Explore Δ_{X_{t}}^{c_{t}} ⎤ ⎦ + E ⎡ ⎣ \sum t \in W-Explore Δ_{X_{t}}^{c_{t}} ⎤ ⎦ + E ⎡ ⎣ \sum t \in UW-Explore Δ_{X_{t}}^{c_{t}} ⎤ ⎦ .

Shortly we argue that the regret incurred in $W-Explore \cup UW-Explore$ is at most logarithmic and hence the regret in rounds associated with forced exploration is sub-logarithmic:

E ⎡ ⎣ \sum t \in F-Explore Δ_{X_{t}}^{c_{t}} ⎤ ⎦ = O (ϵ_{n} | Explore |) = o (log (n)) .

E ⎡ ⎣ \sum t \in W-Explore Δ_{t}^{c_{t}} ⎤ ⎦ = o (log (n)) .

The regret in UW-Explore is logarithmic in $n$ with the asymptotically optimal constant using the definition of the optimal allocation:

limsupn→∞E[∑t∈UW-% ExploreΔctt]log(n)=\cC(θ,\cA1,…,\cAM).

Of course many details have been hidden here, which are covered in detail in the supplementary material.

5 Experiments

In this section, we first empirically compare our proposed algorithm and LinUCB (Abbasi-Yadkori et al., 2011) on some specific problem instances to showcase their strengths and weaknesses. We examine OSSB (Combes et al., 2017) on instances with large action spaces to illustrate its weakness due to ignoring the linear structure. Since Combes et al. (2017) demonstrated that OSSB dominates the algorithm of Lattimore and Szepesvari (2017), we omit this algorithm from our experiments. In the end, we include the comparison with LinTS (Agrawal and Goyal, 2013).

To save computation, we follow the lazy-update approach, similar to that proposed in Section 5.1 of (Abbasi-Yadkori et al., 2011): The idea is to recompute the optimization problem (10) whenever $det (G_{t})$ increases by a constant factor $(1 + ζ)$ and in all scenarios we choose (the arbitrary value) $ζ = 0.1$ . All codes were written in Python. To solve the convex optimization problem (10), we use the CVXPY library (Diamond and Boyd, 2016).

5.1 Fixed Action Set

Finite-armed linear bandits with fixed action set are a special case of linear contextual bandits. Let $d = 2$ and let the true parameter be $θ = (1, 0)^{⊤}$ . The action set $\cA={x1,x2,x3}$ is fixed and $x_{1} = (1, 0)^{⊤}$ , $x_{2} = (0, 1)^{⊤}$ , $x_{3} = (1 - u, 5 u)^{⊤}$ . We consider $u = {0.1, 0.2}$ . By construction, $x_{1}$ is the optimal arm. From Figure 1, we observe that LinUCB suffers significantly more regret than our algorithm. The reason is that if $u$ is very small, then $x_{1}$ and $x_{3}$ point in almost the same direction and so choosing only these arms does not provide sufficient information to quickly learn which of $x_{1}$ or $x_{3}$ is optimal. On the other hand, $x_{2}$ and $x_{1}$ point in very different directions and so choosing $x_{2}$ allows a learning agent to quickly identify that $x_{1}$ is in fact optimal. LinUCB stops pulling $x_{2}$ once it is optimistic and thus does not balance this trade-off between information and regret. Our algorithm, however, takes this into consideration by tracking the optimal allocation ratios.

Figure 1:
Fixed action set. The results are averaged over 100 realizations. Here and also later, the shaded areas show the standard errors.

5.2 Changing Action Set

We consider a simple but representative case when there are only two action sets $\cA1$ and $\cA2$ available.

Scenario One. In each round, $\cA1$ is drawn with probability 0.3 while $\cA2$ is drawn with probability 0.7. Set $\cA1$ contains $x_{1}^{1} = (1, 0, 0)^{⊤}$ , $x_{2}^{1} = (0, 1, 0)^{⊤}$ , and $x_{3}^{1} = (0.9, 0.5, 0)^{⊤}$ , while set $\cA2$ contains $x_{1}^{2} = (0, 1, 0)^{⊤}$ , $x_{2}^{2} = (0, 0, 1)^{⊤}$ , and $x_{3}^{2} = (0, 0.5, 0.9)^{⊤}$ . The true parameter $θ$ is $(1, 0, 1)^{⊤}$ . From the left panel of Figure 2, we observe that LinUCB, while starts better, eventually again suffers more regret than our algorithm.

Figure 2: Changing action sets. The left panel is for scenario one and the right panel is for scenario two. The results are averaged over 100 realizations.

Scenario Two. In each round, $\cA1$ is drawn with probability $0.99$ , while $\cA2$ is drawn with probability $0.01$ . Set $\cA1$ contains three actions: $x_{1}^{1} = (1, 0)^{⊤}$ , $x_{2}^{1} = (0, 1)^{⊤}$ , $x_{3}^{1} = (0.9, 0.5)^{⊤}$ , while set $\cA2$ contains three actions: $x_{1}^{2} = (0, 1)^{⊤}$ , $x_{2}^{2} = (- 1, 0)^{⊤}$ , $x_{3}^{2} = (- 1, 0)$ . Apparently, $x_{1}^{1}$ and $x_{1}^{2}$ are the optimal arms for each action set and they span $R^{2}$ . Based on the allocation rule in Section 3.1, the algorithm is advised to pull actions $x_{1}^{1}$ and $x_{1}^{2}$ very often based on asymptotics. However, since the probability that $\cA2$ is drawn is extremely small, we are very likely to fall back to wasted exploration and use LinUCB to explore. Thus, in the short term, our algorithm will suffer from the drawback that optimistic algorithms also suffer from and what is described in Section 5.1. Although, the asymptotics will eventually “kick in”, it may take extremely long time to see the benefits of this and the algorithm’s finite-time performance will be poor. Indeed, this is seen on the right panel of Figure 2, which shows that the performance of our algorithm and that of LinUCB nearly coincide in this case.

5.3 Bounded Regret

In Section 3, we showed that when the optimal arms of all action sets span $R^{d}$ , our algorithm achieves sub-logarithmic regret. Through experiments, we justify that the algorthm could even suffer bounded regret. We consider $M = 2$ . At each round, $\cA1$ is drawn with probability 0.8 while $\cA2$ is drawn with probability 0.2 and the true parameter $θ$ is $(1, 0)^{⊤}$ .Set $\cA1$ contains three actions: $x_{1}^{1} = (1, 0)^{⊤}$ , $x_{2}^{1} = (0, 1)^{⊤}$ , $x_{3}^{1} = (0.9, 0.5)^{⊤}$ , while set $\cA2$ contains three actions: $x_{1}^{2} = (0, 1)^{⊤}$ , $x_{2}^{2} = (- 1, 0)^{⊤}$ , $x_{3}^{2} = (- 1, 0)$ . As discussed before, $x_{1}^{1}$ and $x_{1}^{2}$ are the optimal arms for each action set and they span $R^{2}$ . The results are shown in the left subpanel of Figure 3. Our algorithm achieved bounded regret in a relatively short time period. Interestingly, we found that LinUCB can also achieve bounded regret when the optimal arms of changing action sets span $R^{d}$ .

Figure 3: The left panel is for bounded regret and right panel is for large action space. The results are averaged over 100 realizations.

5.4 Large Action Space

We consider finite-armed linear bandit with fixed action set. Let $d = 2$ and $θ = (1, 0)^{⊤}$ . We generate $100$ uniformly distributed on the $d$ -dimensional unit sphere. The results are shown in the right subfigure of Figure 3

. When the action space is large, OSSB suffers significantly large regret and performs unstable due to the ignoring of linear structure. The regret of (the theoretically justified version of) LinTS is also very large due to the unnecessary variance factor required by its theory.

6 Discussion

We presented a new optimization-based algorithm for linear contextual bandits that is asymptotically optimal and adapts to both the action sets and unknown parameter. The new algorithm enjoys sub-logarithmic regret when the collection of optimal actions spans $R^{d}$ , a property that we also prove for optimism-based approaches. There are many open questions. A natural starting point is to prove near-minimax optimality of the new algorithm, possibly with minor modifications. Our work also highlights the dangers of focusing too intensely on asymptotics, which for contextual bandits hide completely the dependence on the context distribution. This motivates the intriguing challenge to understand the finite-time instance-dependent regret. Another open direction is to consider the asymptotics when the context space is continuous, which has not seen any attention.

References

Abbasi-Yadkori et al. (2011) Abbasi-Yadkori, Y., Pál, D. and Szepesvári, C. (2011). Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems.
Agarwal et al. (2009) Agarwal, D., Chen, B.-C., Elango, P., Motgi, N., Park, S.-T., Ramakrishnan, R., Roy, S. and Zachariah, J. (2009). Online models for content optimization. In Advances in Neural Information Processing Systems.
Agrawal and Goyal (2013) Agrawal, S. and Goyal, N. (2013). Thompson sampling for contextual bandits with linear payoffs. In
International Conference on Machine Learning
.
Awerbuch and Kleinberg (2008) Awerbuch, B. and Kleinberg, R. (2008). Online linear optimization and adaptive routing. Journal of Computer and System Sciences 74 97–114.
Bastani et al. (2017) Bastani, H., Bayati, M. and Khosravi, K. (2017). Mostly exploration-free algorithms for contextual bandits. arXiv preprint arXiv:1704.09011 .
Chan and Lai (2006) Chan, H. P. and Lai, T. L. (2006). Sequential generalized likelihood ratios and adaptive treatment allocation for optimal sequential selection. Sequential Analysis 25 179–201.
Chu et al. (2011) Chu, W., Li, L., Reyzin, L. and Schapire, R. (2011). Contextual bandits with linear payoff functions. In
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics
.
Combes et al. (2017) Combes, R., Magureanu, S. and Proutiere, A. (2017). Minimal exploration in structured stochastic bandits. In Advances in Neural Information Processing Systems.
Dani et al. (2008) Dani, V., Hayes, T. P. and Kakade, S. M. (2008). Stochastic linear optimization under bandit feedback .
Degenne et al. (2019) Degenne, R., Koolen, W. M. and Ménard, P. (2019). Non-asymptotic pure exploration by solving games. arXiv preprint arXiv:1906.10431 .
Diamond and Boyd (2016) Diamond, S. and Boyd, S. (2016). CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research 17 1–5.
Garivier and Kaufmann (2016) Garivier, A. and Kaufmann, E. (2016). Optimal best arm identification with fixed confidence. In 29th Annual Conference on Learning Theory (V. Feldman, A. Rakhlin and O. Shamir, eds.), vol. 49 of Proceedings of Machine Learning Research. PMLR, Columbia University, New York, New York, USA.
Kannan et al. (2018) Kannan, S., Morgenstern, J. H., Roth, A., Waggoner, B. and Wu, Z. S. (2018). A smoothed analysis of the greedy algorithm for the linear contextual bandit problem. In Advances in Neural Information Processing Systems.
Lai and Robbins (1985) Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6 4–22.
Langford and Zhang (2007) Langford, J. and Zhang, T. (2007).
The epoch-greedy algorithm for contextual multi-armed bandits.
In Proceedings of the 20th International Conference on Neural Information Processing Systems. Citeseer.
Lattimore and Szepesvari (2017) Lattimore, T. and Szepesvari, C. (2017).
The end of optimism? an asymptotic analysis of finite-armed linear bandits.
In Artificial Intelligence and Statistics.
Lattimore and Szepesvári (2019) Lattimore, T. and Szepesvári, C. (2019). Bandit algorithms. preprint .
Li et al. (2010) Li, L., Chu, W., Langford, J. and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web. WWW ’10, ACM, New York, NY, USA.
URL http://doi.acm.org/10.1145/1772690.1772758
Ok et al. (2018) Ok, J., Proutiere, A. and Tranos, D. (2018). Exploration in structured reinforcement learning. In Advances in Neural Information Processing Systems.
Rusmevichientong and Tsitsiklis (2010) Rusmevichientong, P. and Tsitsiklis, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research 35 395–411.
Tsybakov (2008) Tsybakov, A. B. (2008). Introduction to Nonparametric Estimation. 1st ed. Springer Publishing Company, Incorporated.
Vershynin (2010) Vershynin, R. (2010). Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 .

Appendix A Proofs of Asymptotic Lower and Upper Bounds

First of all, we define the sub-optimal action set as $\cAm−=\cAm∖{x:Δmx=0}$ and denote $\cA=∪Mm=1\cAm$ and $\cA−=∪Mm=1\cAm−$ .

a.1 Proof of Lemma 3

The proof idea follows if ${¯ G}_{n}$ is not sufficiently large in every direction, then some alternative parameters are not sufficiently identifiable.

Step One.

We fix a consistent policy $π$ and fix a context $m \in [M]$ as well as a sub-optimal arm $x∈\cAm−$ . Consider another parameter $~ θ \in R^{d}$ such that it is close to $θ$ but $x_{m}^{*}$ is not the optimal arm in bandit $~ θ$ for action set $\cAm$ . Specifically, we construct

~ θ = θ + \frac{H (x - x_{m}^{*})}{∥ x - x_{m}^{*} ∥_{H}^{2}} (Δ_{x}^{m} + ε),

where $H \in R^{d \times d}$ is some positive semi-definite matrix and $ε > 0$ is some absolute constant that will be specified later. Since the sub-optimality gap ${~ Δ}_{x_{m}^{*}}^{m}$ satisfies

⟨ x - x_{m}^{*}, ~ θ ⟩ = ⟨ x - x_{m}^{*}, θ ⟩ + Δ_{x}^{m} + ε = ε > 0,

(16)

it ensures that $x_{m}^{*}$ is $ε$ -suboptimal in bandit $~ θ$ .

We define $T_{x} (n) = \sum_{t = 1}^{n} I (X_{t} = x)$ and let $P$ and $~ P$ be the measures on the sequence of outcomes $(X_{1}, Y_{1}, \dots, X_{n}, Y_{n})$ induced by the interaction between the policy and the bandit $θ$ and $~ θ$ respectively. By the definition of ${¯ G}_{n}$ in (2), we have

$\frac{1}{2} ∥ θ - ~ θ ∥_{{¯ G}_{n}}^{2}$	$=$	$\frac{1}{2} (θ - ~ θ)^{⊤} {¯ G}_{n} (θ - ~ θ)$
	$=$	$12(θ−~θ)⊤E[∑x∈\cATx(n)xx⊤](θ−~θ)$
	$=$	$12∑x∈\cAE[Tx(n)]⟨x,θ−~θ⟩2.$

Applying the Bretagnolle-Huber inequality inequality in Lemma LABEL:lem:kl and divergence decomposition lemma in Lemma LABEL:lem:inf-processing, it holds that for any event $\cD$ ,

12∥θ−~θ∥2¯Gn=KL(P,~P)≥log(12(P(\cD)+~P(\cDc))).

(17)

Step Two.

In the following, we start to derive a lower bound of $R_{θ}^{π} (n)$ ,

$R_{θ}^{π} (n)$	$=$	$E [n \sum t = 1 ⟨ x_{c_{t}}^{} - X_{t}, θ ⟩] = E [M \sum m = 1 \sum t : c_{t} = m ⟨ x_{m}^{} - X_{t}, θ ⟩]$
	$\geq$	$E [\sum t : c_{t} = m ⟨ x_{m}^{*} - X_{t}, θ ⟩] = E [\sum t : c_{t} = m Δ_{X_{t}}^{m}]$
	$\geq$	$Δ_{min} E [\sum t : c_{t} = m I (X_{t} \neq x_{m}^{})] = Δ_{min} E [n \sum t = 1 I (c_{t} = m) - n \sum t = 1 I (c_{t} = m) I (X_{t} = x_{m}^{})],$

where the first inequality comes from the fact that $⟨ x_{m}^{*} - X_{t}, θ ⟩ \geq 0$ for all $m \in [M]$ . Define the event $\cD$ as follows,

\cD={n∑t=1I(ct=m)I(Xt=x∗m)≤12n∑t=1I(ct=m)}.

(18)

When event $\cD$ holds, we will only pull at most half of total rounds for the optimal action of action set $m$ . Then it holds that

	$R_{θ}^{π} (n)$	$\geq$	$ΔminE[(n∑t=1I(ct=m)−n∑t=1I(ct=m)I(Xt=x∗m))I(\cD)]$
		$\geq$	$ΔminE[12n∑t=1I(ct=m)I(\cD)].$

Define another event $\cB$ as follows,

\cB={12n∑t=1I(ct=m)≥npm2−δ/2},

(19)

where $δ > 0$ will be chosen later and $p_{m}$ is the probability that the environment picks context $m$ . From the definition of $c_{t}$ , we have $E [\sum_{t = 1}^{n} I (c_{t} = m)] = n p_{m} .$ By the standard Hoeffding’s inequality (Vershynin, 2010), it holds that

P (\frac{1}{2} n \sum t = 1 I (c_{t} = m) - \frac{n p_{m}}{2} \geq \frac{δ}{2}) \geq 1 - exp (- \frac{2 δ^{2}}{n}),

which implies

P(\cBc)≤exp(−2δ2/n).

By the definition of events $\cD,\cB$ in (18),(19), we have

$R_{θ}^{π} (n)$	$\geq$	$ΔminE[12n∑t=1I(ct=m)I(\cD)I(\cB)]$
	$\geq$	$ΔminE[(12npm−δ2)I(\cD)I(\cB)]$
	$=$	$Δmin(12npm−δ2)P(\cD∩\cB)$
	$\geq$	$Δmin(12npm−δ2)(P(\cD)−P(\cBc)).$

Letting $δ = n p_{m} / 2$ , we have

Rπθ(n)≥Δminnpm4(P(\cD)−exp(−np2m2)).

(20)

On the other hand, we let $~ E$ is taken with respect to probability measures $~ P$ . Then $R_{~ θ}^{π} (n)$ can be lower bounded as follows,

	$R_{~ θ}^{π} (n)$	$=$	$~ E [M \sum m = 1 n \sum t = 1 I (c_{t} = m) {~ Δ}_{X_{t}}^{m}]$
		$\geq$

where we throw out all the sub-optimality gap terms except ${~ Δ}_{x_{m}^{*}}^{m}$ . Using the fact that ${~ Δ}_{x_{m}^{*}}^{m}$ is $ε$ -suboptimal, it holds that

$R_{~ θ}^{π} (n)$	$\geq$	$ε~E[(n∑t=1I(ct=m)I(Xt=x∗m))I(\cDc)]$	(21)
	$>$	$ε~E[12n∑t=1I(ct=m)I(\cDc)]$
	$\geq$	$ε~E[12n∑t=1I(ct=m)I(\cDc)I(\cB)]$
	$\geq$	$ε(npm2−δ2)P′(\cDc∩\cB)$
	$\geq$	$ε(npm2−δ2)(~P(\cDc)−~P(\cBc))$
	$=$	$ε(npm2−δ2)(~P(\cDc)−exp(−2δ2n))$
	$=$	$εnpm4~P(\cDc)−εnpm4exp(−np2m2).$

Now we have derived the lower bounds (20)(21) for $R_{θ}^{π} (n), R_{~ θ}^{π} (n)$ respectively.

Step Three.

Combining the lower bounds of $R_{θ}^{π} (n)$ and $R_{~ θ}^{π} (n)$ together, it holds that

R

Adaptive Exploration in Linear Contextual Bandit

An Asymptotically Optimal Primal-Dual Incremental Algorithm for Contextual Linear Bandits

Parallelizing Contextual Linear Bandits

Breaking the √(T) Barrier: Instance-Independent Logarithmic Regret in Stochastic Contextual Linear Bandits

Exploiting the Natural Exploration In Contextual Bandits

Information Directed Sampling for Linear Partial Monitoring

Leveraging Good Representations in Linear Contextual Bandits

Power-Constrained Bandits

1 Introduction

Notation

2 Problem Setting

3 Asymptotic Lower Bound

3.1 Sub-logarithmic regret

4 Optimal Allocation Matching

4.1 Algorithm

Initialisation

Main phase

Exploitation.

Exploration.

4.2 Asymptotic Upper Bound

4.3 Proof Sketch

Regret while exploiting

Regret while exploring

5 Experiments

5.1 Fixed Action Set

5.2 Changing Action Set

5.3 Bounded Regret

5.4 Large Action Space

6 Discussion

References

Appendix A Proofs of Asymptotic Lower and Upper Bounds

a.1 Proof of Lemma 3

Step One.

Step Two.

Step Three.

Adaptive Exploration in Linear Contextual Bandit

Related Research

An Asymptotically Optimal Primal-Dual Incremental Algorithm for Contextual Linear Bandits

Parallelizing Contextual Linear Bandits

Breaking the √(T) Barrier: Instance-Independent Logarithmic Regret in Stochastic Contextual Linear Bandits

Exploiting the Natural Exploration In Contextual Bandits

Information Directed Sampling for Linear Partial Monitoring

Leveraging Good Representations in Linear Contextual Bandits

Power-Constrained Bandits

1 Introduction

Notation

2 Problem Setting

3 Asymptotic Lower Bound

3.1 Sub-logarithmic regret

4 Optimal Allocation Matching

4.1 Algorithm

Initialisation

Main phase

Exploitation.

Exploration.

4.2 Asymptotic Upper Bound

4.3 Proof Sketch

Regret while exploiting

Regret while exploring

5 Experiments

5.1 Fixed Action Set

5.2 Changing Action Set

5.3 Bounded Regret

5.4 Large Action Space

6 Discussion

References

Appendix A Proofs of Asymptotic Lower and Upper Bounds

a.1 Proof of Lemma 3

Step One.

Step Two.

Step Three.