On Worst-case Regret of Linear Thompson Sampling

1 Introduction

Recently, there has been a rise in the use of experiments by many organizations to optimize decisions (e.g., product recommendation in e-Commerce, ad selection in digital advertising, or testing medical interventions in healthcare). However, running an experiment involves an opportunity cost or regret (e.g., exposing some users or patients to a potentially inferior experience or treatment). To reduce this opportunity cost, a growing number of enterprises leverage multi-armed bandit (MAB) experiments (Scott, 2010, 2015; Johari et al., 2017). MAB approach works by adaptively updating decisions based on partially available results of the experiment to minimize the regret. These practical motivations that date back to Thompson (1933); Lai and Robbins (1985), combined with its mathematical richness, have made the MAB problem subject of intense study in statistics, operations research, electrical engineering, computer science, and economics, over the last few decades (Russo et al., 2018; Lattimore and Szepesvari, 2019).

This paper considers a general version of the MAB problem, the stochastic linear bandit problem, in which an decision-maker is sequentially choosing actions among given action sets and observes rewards

corresponding to the selected actions. The rewards are stochastic and their means depend on the actions through a fixed linear function. While initially unknown to the decision-maker, the reward function can be estimated as more decisions are made and their rewards are observed. The main goal of the decision-maker is to maximize its cumulative expected reward, over a sequence of decision epochs (or time periods). Equivalently, one can measure difference (referred by expected regret or regret for short) between the best achievable cumulative expected reward, obtained by an

oracle that has access to the true mean of the reward function, and the cumulative expected reward obtained by the decision-maker.

The regret can be measured in a Bayesian or in a frequentist fashion. The Bayesian regret is used when the mean reward functions depend on random parameters and the expectations are taken with respect to the randomness in the reward functions, the uknown parameters, and new potential randomness introduced by the decision-maker. But the frequentist regret (also referred to by worst-case regret) is used when the mean reward functions are deterministic, so the expectation is only with respect to the other two sources of randomness.

The main challenge of the decision-maker is to design algorithms that efficiently balance between the exploration (experimenting untested actions) and the exploitation (choosing high-reward actions). Two approaches to this problem have attracted a great deal of attention. Dani et al. (2008); Abbasi-Yadkori et al. (2011) utilize optimism in face of uncertainty and obtain policies with $\Orderlog[]d√T$ worst-case regret bounds which is, as shown by Dani et al. (2008), minimax optimal up to logarithmic factors. The other approach, introduced by Thompson (1933)

, arises from a heuristic idea in the Bayesian setting which suggests sampling from the posterior distribution of the reward function, given past observatrions, and choosing the best action as if this sample were the true reward function. This approach is known as Thompson Sampling (TS) or posterior sampling and although it is Bayesian in nature, it can be applied in the frequentist setting as well. This idea has become increasingly popular in practice due to its simplicity and empirical performance

(Scott, 2010, 2015; Russo et al., 2018).

TS has been extensively studied from both theoretical and empirical points of view. Most notably, Agrawal and Goyal (2012, 2013a) prove minimax near-optimal worst-case guarantees for TS in the multi-armed bandit (MAB) setting. Russo and Van Roy (2014) use the connection between TS and optimistic policies to provide the first theoretical guarantee for TS that covers a wide range of problems including the stochastic linear bandit problem in which TS heuristic is refered by LinTS. Their analysis yields a $\Orderlog[]d√T$ Bayesian regret bound for this problem which cannot be improved in general.

In the frequentist setting, however, Agrawal and Goyal (2013b); Abeille et al. (2017) have obtained $\Orderlog[]d√dT$ regret bounds for a variant of LinTS which samples from a posterior distribution whose variance is inflated by a factor $\Orderlogd$ . This bound is far from the optimal rate by a factor of $\sqrt{d}$ . It has been an open question as to whether this extra factor can be eliminated in the linear bandit problem, e.g., stated in (Russo et al., 2018, page 78). We answer this question negatively. In particular, we construct examples to show that LinTS without inflation can incur linear regret up to time $exp\Orderd$ when the noise distribution and/or the prior distribution does not match the ones that LinTS assumes. The striking fact about these examples is that they can successfully deceive LinTS even if one reduces the variance of the noise. In fact, we will show that noiseless observations can cause LinTS to fail for an exponentially long time. This issue with LinTS is important to be understood because of the following reasons:

In many applications, the exact prior and noise distributions are either unknown or not easy to sample from. In these cases, one needs to estimate or approximate the posterior distribution. However, as our examples demonstrate, LinTS is not robust to these mismatches.
This issue opens the door for adversarial attacks. Notice that in the posterior computation, it is often assumed that, conditional on the history, the set of actions is independent of the true reward function. This assumption may not hold true when an adversary who has some knowledge about the true parameter can change action sets. This scenario is in particular applicable in the presence of a competing firm that has acquired more data about the same problem.

We here emphasize that these concerns are not applicable to optimism in the face of uncertainty linear bandit (OFUL) algorithm of Abbasi-Yadkori et al. (2011). These two issues thus call for the necessity of a better understanding of LinTS in the frequentist setting. In fact, on the positive side, we use the framework introduced in Hamidi and Bayati (2020) to prove that under additional assumptions the inflation parameter can be significantly reduced while still holding the theoretical guarantees. We validate our assumptions through simulations in a synthetic setting.

2 Setting and notation

For any positive integer $n$ , we denote ${1, 2, \dots, n}$ by $[n]$ . Letting $Σ$ be a positive semi-definite matrix, by $\normAΣ$ we mean $\sqrt{A^{⊤} Σ A}$

for any vector

A

of suitable size. For a matrix

M

with the singular values

σ_{1} \geq \dots \geq σ_{n}

, we define its operator norm and nuclear norm as

\normOpM:=σ1

and trace norm as

\normNucM:=∑i∈[n]σi

respectively.

Let $(A_{t})_{t = 1}^{T}$ be a sequence of $T$ random compact subsets of $\IRd$ where $T∈\IN$ is the time horizon. We further assume that $\normA2≤a$ for all $A \in A_{t}$ almost surely. A policy $π$ sequentially interacts with this environment in $T$ rounds. At time $t \in [T]$ , it receives $A_{t}$ and chooses an action ${˜ A}_{t} \in A_{t}$ and receives a stochastic reward $Yt=\inner[]Θ⋆,˜At+εt$ where $Θ^{⋆}$ is the unknown (and potentially random) vector of parameters. By $A_{t}^{⋆} \in A_{t}$ we denote the arm with maximum expected reward. We denote the history of observations up to time $t$ by $F_{t}$ . More precisely, we define $F_{t} := (A_{1}, {˜ A}_{1}, Y_{1}, \dots, A_{t - 1}, {˜ A}_{t - 1}, Y_{t - 1}, A_{t})$ . In this model, a policy $π$ is formally defined as a (stochastic) function that maps $F_{t}$ to an element of $A_{t}$ .

We compare policies through their cumulative Bayesian regret defined as

Regret(T,π):=T∑t=1\IEsupA∈At\inner[]Θ⋆,A−\inner[]Θ⋆,˜At.

Notice that the expectation is taken with respect to the entire randomness in our model, including the prior distribution. The frequentist regret bounds also follow by taking the prior the distribution to be the measure that puts all the mass on a single vector.

3 Bayesian analyses are brittle

In this section, we demonstrate that LinTS with proper posterior update rule may incur linear regret when the assumptions are slightly violated. These examples, in particular, solve an open question mentioned in (Russo et al., 2018, §8.1.2). More precisely, we show that LinTS’s Bayesian regret (thereby, its worst-case regret) can grow linearly up to time $exp(\Orderd)$ whenever the prior distribution or the noise distribution mismatches with the one that LinTS works with. It, furthermore, follows from our strategy that one needs the inflation rate of at least $Ω (d / log d)$ to avoid these problems.

3.1 Noise reduction and LinTS’s failure

Here we show that reducing noise or the variance of the prior distribution can cause LinTS to fail. Our strategy for proving these results involves the following two steps:

We first construct small problem instances for which ${˜ Θ}_{t}$ is marginally biased.
We then show that by combining independent copies of these biased instances Thompson sampling can get linear Bayes regret.

0: Inflation parameter

ι

1: Initialize

Σ_{1} \leftarrow λ I

and

{ˆ Θ}_{1} \leftarrow 0

2: for

t = 1, 2, \dots

3: Observe

A_{t}

4: Sample

˜Θt∼\NormalˆΘtι2Σt

˜At←\argmaxA∈At\inner[]A,˜Θt

6: Observe reward

R_{t}

Σ_{t + 1}^{- 1} \leftarrow Σ_{t}^{- 1} + {˜ A}_{t} {˜ A}_{t}^{⊤}

{ˆ Θ}_{t + 1} \leftarrow Σ_{t + 1} (Σ_{t}^{- 1} {ˆ Θ}_{t} + {˜ A}_{t} R_{t})

9: end for

Algorithm 1 Linear Thompson sampling

Bias-introducing action sets.

In this section, we construct an example in which ${˜ Θ}_{t}$ is marginally biased provided that either the prior distribution or the noise distribution mismatches the one that LinTS uses. Fix $σ^{2}, τ^{2} \geq 0$ and let $Θ⋆∼\Normal0σ2I2$ be the vector of unobserved parameters. At time $t \in {1, 2, 3}$ , we reveal the following action sets to the policy:

A_{t} := ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} {e_{1}} & if t = 1, {e_{2}} & if t = 2, {e_{1}, e_{2}} & if t = 3 . \end{matrix}

For $t \leq 2$ , LinTS has only one choice $e_{t}$ and thus ${˜ A}_{t} = e_{t}$ . Assume that $Y_{t} = Θ_{t}^{⋆} + ε_{t}$ is revealed to the algorithm where $εt∼\Normal0τ2$ . At time $t = 3$ for the first time, LinTS has two choices. Let $i$ be such that ${˜ A}_{3} = e_{i}$ . Then, $Y_{3} = Θ_{i}^{⋆} + ε_{3}$ is given to the algorithm where $ε3∼\Normal01$ . The following lemma asserts that ${ˆ Θ}_{4}$ is marginally biased. Let $V = e_{1} + e_{2}$ . For any $σ, τ \geq 0$ , we have

\inner[]V,\IE[]ˆΘ4

= \frac{(σ^{2} - τ^{2}) β}{6 \sqrt{σ^{2} + τ^{2} + 2}},

(1)

where $β:=\IEmax{A,B}>$ where $A$ and $B$

are two independent standard normal random variables. Furthermore,

{ˆ Θ}_{4}

satisfies

\IEexp(s\inner[]V,ˆΘ4−\IE[]ˆΘ4)≤exp(s2(4σ+4τ+2)22),    for all s∈\IR.

(2)

Stacking biased settings.

We prove that, by combining independent copies of the above example, LinTS can choose an incorrect action for at least $exp(\Orderd)$ rounds. Let $d$ be a positive integer and define $Θ⋆∼PΘ⋆=\Normal0σ2I2d$ . In the first $3 d$ rounds, follow the action sets in the previous section for each pairs $(Θ_{2 i - 1}^{⋆}, Θ_{2 i}^{⋆})$ for $i \in [d]$ . Namely, define

\begin{matrix} A_{t} := ⎧ ⎪ ⎨ ⎪ ⎩ \begin{matrix} {e_{t}} & if t \leq 2 d, {e_{2 (t - 2 d) - 1}, e_{2 (t - 2 d)}} & if 2 d + 1 \leq t \leq 3 d, {0, A} & otherwise, \end{matrix} \end{matrix}

(3)

where

A:=\signτ2−σ2√d⋅2d∑i=1ei.

The following key lemma states that with constant probability

A

is the optimal action while LinTS perceives it as suboptimal with enormous gap. Letting

p_{0} := \frac{1}{2} (1 - Φ (1)) > 0

, we have

\IP\inner[]Θ⋆,A≥√2σ  %and  \inner[]ˆΘ3d+1,A≤−C1√d2≥p0.

We denote the above event by $B$ . Conditional on this event, for all $t > 3 d$ , the optimal arm is $A$ and the regret incurred by choosing $0$ is at least $\sqrt{2} σ$ . Moreover, let $q$ be the probability of choosing $A$ at $t = 3 d + 1$ . As we will see this probability is exponentially small as a function of $d$ and whenever $A$ is not chosen, the probability of selecting $A$ in the next round remains unchanged. This observation holds true up to the first time that $A$ is picked which can, in turn, take an exponentially long time. By making this argument rigorous we can state the following proposition: For fixed $σ \neq τ \geq 0$ , we have

Regret(T,πLinTS)≥\OrderT.

3.2 Mean shift and fixed action sets

In this subsection, we construct an example in which LinTS incurs linear Bayes regret while the action set is fixed over time. This example, nonetheless, might be less appealing than the one in the previous subsection as we shift the mean of the prior distribution. Let $μ, σ, τ > 0$ be fixed and for $d∈\IN$ , set the prior distribution to be $PΘ⋆:=\Normalμ13dσ2I3d$ . We now reveal the action set $A_{t} := {0, A^{'}, A}$ to LinTS for all $t \in [T]$ where

A^{'} := - \frac{1}{\sqrt{d}} d \sum i = 1 e_{i} and A := \frac{1}{\sqrt{d}} 3 d \sum i = d + 1 e_{i} - \frac{1}{} \sqrt{d} d \sum i = 1 e_{i} .

(4)

The next proposition highlights the key observations about why LinTS fails in this simple setting: For fixed $μ, σ > 0$ and for sufficiently large $d$ , we have

$\innerΘ⋆,A′≤−12μ√d≤12μ√d≤\innerΘ⋆,A$ with probability at least $\frac{7}{8}$ ,
${˜ A}_{1} = A^{'}$ with probability $\frac{1}{4}$ ,
Conditional on ${˜ A}_{1} = A^{'}$ , $\inner[]ˆΘ2,A∨\inner[]ˆΘ2,A′≤−18μ√d$ , with probability at least $\frac{15}{16}$ ,
Conditional on ${˜ A}_{1} = A^{'}$ , ${˜ A}_{2} \neq 0$ with probability at most $exp(\Orderd)$ ,
For $T≤exp(\Orderd)$ , $Regret(T,πLinTS)≥\OrderT√d$ .

One can slightly modify the proof to obtain similar result for

PΘ⋆:=\Normal0σ2I3d+ρ13d1⊤3d.

It is easy to see that for any arbitrary constant $ρ$ , the same rate as in Equation 13 is achievable. Also, for $ρ = d^{- α}$ where $α < 1$ , one can still get non-trivial results.

4 Improving LinTS

The aim of this section is to introduce a novel approach to improve the inflation parameter in LinTS under additional assumptions. Before stating our results, we discuss the insights that leads to these assumptions.

4.1 Insights into LinTS’s optimism mechanism

This subsection is dedicated to the intuitions about the optimism mechanism of LinTS. We assume that $ˆ Θ$ is the ridge estimator for the parameter $Θ^{⋆}$ at some time and $C$ is the confidence set that contains $˜ Θ$ and $Θ^{⋆}$ with high probability. We reveal the action set ${A^{⋆}, 0}$ to the policy where $A^{⋆}$ is the optimal arm, i.e., $\innerΘ⋆,A⋆>0$ . LinTS chooses $A^{⋆}$ only if

\inner[]˜Θ,A⋆>0.

The left-hand side of this inequality can be decomposed as

\inner[]˜Θ,A⋆=\inner[]˜Θ−ˆΘ,A⋆+\inner[]ˆΘ−Θ⋆,A⋆+\inner[]Θ⋆,A⋆.

This implies that a sufficient condition for $\inner[]˜Θ,A⋆>0$ to hold is

\inner[]˜Θ−ˆΘ,A⋆≥\inner[]Θ⋆−ˆΘ,A⋆.

(5)

This inequality requires $C := ˜ Θ - ˆ Θ$ to compensate for the underestimation of the reward caused by the estimation error vector $E := Θ^{⋆} - ˆ Θ$ . These vectors are illustrated in Figure 1.

OFUL explicitly seeks $˜ Θ \in C$ that maximizes the left-hand size of Equation 5, and as $Θ^{⋆} \in C$ with high probability, the desired “compensation inequality” holds and $A^{⋆}$ is selected. Thompson sampling, on the other hand, follows a stochastic approach and resorts to a randomly sampled point in $C$ to solve Equation 5. Recall that $ˆ Θ$ is the ridge estimator for the collected data thus far. In a fixed design setting (which is not true in our bandit problem), the error vector $E$ will be pointing to a random direction. Therefore, provided that $A^{⋆}$ is independent of $E$ , we have

\abs[]\inner[]E,A⋆≈\Order1√d\normE2⋅\normA⋆2.

(6)

The same expression also holds for $\abs[]\inner[]C,A⋆$ ; therefore, the compensation inequality holds with constant probability. To summarize our observation, the inequality Equation 6 holds true if the following two conditions are met

The error vector $E$ is distributed in a random direction.
The optimal action $A^{⋆}$ is independent of $E$ .

The crucial point in the analysis of LinTS in the Bayesian setting is that whenever LinTS has access to the true prior and noise distribution, the first condition above holds. In Section 3, nonetheless, we have shown that this condition is violated if LinTS uses an incorrect prior or noise distribution in computing the posterior. Agrawal and Goyal (2013b); Abeille et al. (2017) take a conservative approach and propose to inflate the variance of the posterior distribution by a factor of $\Orderlog[]d$ to ensure $\innerC,A⋆≥\innerE,A⋆$ with constant probability. We now present an alternative approach that leverages the randomness of the optimal action to reduce the need for exploration. The following assumption requires the optimal arm (rather then the error vector) to be distributed in a random direction. Assume that for any $V∈\IRd$ with $\normV2=1$ , we have

\IP\innerA⋆t,V>ν√d\normA⋆t2≤1t3,

for some fixed $ν \in [1, \sqrt{d}]$

. Unfortunately, this condition alone does not suffice to reduce the inflation rate of the posterior distribution. To see this, consider a case in which the largest eigenvalue of

Σ

is much larger than the other ones; thereby,

\normOpΣ≈\normNucΣ

. Figure 2 illustrates this situation. In this case, we have

Figure 2: An illustration of a thin confidence set.

However, it follows from the definition of LinTS that . Assuming that $\IEA⋆A⋆⊤≈Id$ , we realize that $\IE[]\normA⋆2Σ≈\normNucΣ$ . This suggests $\normA⋆Σ$ is proportional to $√\normNucΣ/d ⋅\normA⋆2$ . Now, we can see that Section 4.1 is not sufficient for ensuring Equation 5 as we have

\abs[]\inner[]E,A⋆≈√\normOpΣ⋅\normA⋆2≫√\normNucΣd⋅\normA⋆2≈\abs[]\inner[]C,A⋆.

This observation implies the necessity of the inflation rate of order $\sqrt{d}$ when the eigenvalues of $Σ$ differ in magnitude significantly. To make this notion precise, we define the thinness coefficient corresponding to $Σ$ to be

ψ(Σ):=√d⋅\normOpΣ\normNucΣ.

We also make the following assumption. For $Ψ, ω > 0$ , we have

\IP\normA⋆Σ<ω√\normNucΣd⋅\normA⋆2≤1t3

for any positive definite $Σ$ with $ψ (Σ) \leq Ψ$ . With this assumption, we are now ready to state our formal results.

4.2 Formal results

At time $t \in [T]$ , we say that problem is well-posed if $ψ (Σ_{t}) \leq Ψ$ , and the following inequalities are satisfied:

\normA⋆Σt≥ω√\normNucΣd⋅\normA⋆2,     \abs[]\inner[]A⋆t,ˆΘt−Θ⋆≤ν√d\normA⋆t2⋅\norm[]ˆΘt−Θ⋆2.

(7)

We denote the indicator function for this event by $W_{t}$ . The next lemma, loosely speaking, asserts that LinTS is optimistic with constant probability. [Optimism of LinTS] Set $ρ := σ \sqrt{d log (1 + \frac{T a^{2}}{λ})} + \frac{1}{\sqrt{λ}} θ$ and $ι := \frac{ν Ψ}{ω} \cdot \frac{ρ}{\sqrt{d}}$ . Whenever $W_{t} = 1$ , we have

\IP\inner[]˜Θt,A⋆t≥\inner[]Θ⋆,A⋆t\givenFt≥Φ(−1).

(8)

Using Theorem 2 in Hamidi and Bayati (2020), we can prove the following result: Under Sections 4.1 and 4.1, provided that $ψ (Σ_{t}) > Ψ$ with probability at most $\frac{1}{t^{3}}$ , we have

Regret(T,πLinTS)≤\Order√dρ2ι2Tlog(T).

It is worth mentioning that one can also reduce the radius of the confidence ball in OFUL under these assumptions. More precisely, one can replace the radius of the confidence ball $ρ$ with $ι$ while maintaining the same regret bound. Although this does not improve the regret bound, it may improve the empirical performance as it avoids unnecessary exploration. The main caveat of this result is the assumption that $ψ (Σ_{t}) \leq Ψ$ holds with high probability since it is not a mere property of the action sets; indeed, it also depends on the policy through the actions that it chooses. We fix this problem by setting the inflation rate $ι := ρ$ whenever $ψ (Σ_{t}) > Ψ$ . This way, we have the following result. If $∑Tt=1\IPψ(Σt)>Ψ≤C$ , we have

Regret(T,πLinTS)≤\Order√dρ2ι2Tlog(T)+CT.

We will see in Section 5 that, in our simulations, $ψ (Σ_{t})$ is indeed large for a short period of time.

5 Simulations

5.1 Average failure time of LinTS

We validate the examples in Section 3 through the following simulations:

Noise reduction example.

In this simulation, for each $d \in {2, 2^{2}, 2^{3}, \dots, 2^{18}}$ , we generate $Θ⋆∼\Normal0I2d$ and run LinTS for $3 d$ rounds using the action sets Equation 3. The reward for choosing an action ${˜ A}_{t} \in A_{t}$ is simply given by $Yt=\inner[]Θ⋆,˜At$ . Therefore, no noise is added to the reward. We then compute the probability $p$ that $\inner[]ˆΘ3d+1,12d>0$ . We repeat this procedure for 50 times to get $(p_{i})_{1}^{50}$ and take the maximum $p_{max} := {max}_{i \in [50]} p_{i}$ . Figure 2(a) displays $log (1 / p_{max})$ against $d$ .

Fixed action set example.

For given $d$ and $μ$ , we draw $Θ⋆∼\Normalμ13dI3d$ . Then, we reveal the action set $A_{t} = {0, A, A^{'}}$ as defined in Equation 4. Then, conditional on ${˜ A}_{t} = A^{'}$ , we compute the probability $p$ that the next arm is either $A$ or $A^{'}$ . We repeat this process 50 times to get $(p_{i})_{1}^{50}$ and as before we define $p_{max}$ to be their maximum. Figure 2(b) shows $log (1 / p_{max})$ for $μ = 0.1$ when $d$ varies between 1 and 120000. Figure 2(c), on the other hand, illustrates $log (1 / p_{max})$ for $d = 2000$ and $μ$ varying between 0 and 1.

5.2 Thinness over time

Here we investigate how thinness varies over time. We take a similar setting as described in the simulations section in Russo and Van Roy (2014). For $d > 0$ , we generate $Θ⋆∼\Normal010Id$ . At each time $t$ , we generate $k$ i.i.d. random vectors from $Unif ([- \sqrt{0.1}, \sqrt{0.1}]^{d})$ . Each of the following policies then chooses one of these actions:

TS-1: LinTS with no inflation ( $ι = 1$ ).
TS-2: LinTS with $ι = 5$ .
TS-3: LinTS with $ι = 5$ whenever $ψ \leq 2.0$ and $ι = ρ_{t}$ otherwise where

$ρ_{t} := \sqrt{2 log ⎛ ⎜ ⎝ \frac{det (Σ_{t})^{- \frac{1}{2}} det (0.1 I_{d})^{- \frac{1}{2}}}{0.0001} ⎞ ⎟ ⎠} + \sqrt{d} .$
TS-4: LinTS with $ι = ρ_{t}$ .

Each policy chooses ${˜ A}_{t}^{i}$ for $i = 1, 2, 3, 4$ and receives feedback $Yit=\inner[]Θ⋆,˜Ait+εit$ where $ε_{t}^{i}$ are i.i.d. standard Gaussian random variables. Next, we compute the thinness parameter for $I / 10 + \sum_{j = 1}^{t} {˜ A}_{j}^{i} {˜ A}_{j}^{i ⊤}$ . We repeat this procedure 50 times. Figure 5 displays the thinness of these policies in our experiments. This in particular shows that the thinness stays close to 1 for larger values of $t$ . Figure 5 also shows the cumulative regret of these policies. Notice that, while TS-3 may inflate the posterior variance by $ρ_{t}$ , its performance is in fact much closer to TS-1 and TS-2.

Appendix A Proofs of Section 3

Proof of Section 3.1.

It follows from the definition of ${ˆ Θ}_{3}$ that

{ˆ Θ}_{3} = \frac{1}{2} [\begin{matrix} r_{1} r_{2} \end{matrix}] = \frac{1}{2} [\begin{matrix} Θ_{1}^{⋆} + ε_{1} Θ_{2}^{⋆} + ε_{2} \end{matrix}] .

Next, at $t = 3$ , the $i$ -th entry is updated according to

	${ˆ Θ}_{4, i}$	$= \frac{r_{i} + r_{3}}{3}$
		$= \frac{2 Θ_{i}^{⋆} + ε_{i} + ε_{3}}{3}$
		$= \frac{Θ_{i}^{⋆} + ε_{i}}{2} + \frac{Θ_{i}^{⋆} - ε_{i}}{6} + \frac{ε_{3}}{3}$
		$= {ˆ Θ}_{3, i} + \frac{Θ_{i}^{⋆} - ε_{i}}{} 6 + \frac{ε_{3}}{3} .$

Moreover, the other entry remains unchanged, in other words

{ˆ Θ}_{4, 3 - i} = {ˆ Θ}_{3, 3 - i} .

Therefore, setting $V = e_{1} + e_{2}$ , we have

	$\inner[]ˆΘ4,V$	$=\inner[]ˆΘ4,V$
		$= {ˆ Θ}_{4, 1} + {ˆ Θ}_{4, 2}$
		$=ˆΘ3,1+ˆΘ3,2+Θ⋆i−εi6+ε33.\yesnumber$

and in particular

\inner[]\IE[]ˆΘ4,V

=\IEΘ⋆i−εi6.

(9)

We can now compute this expression in terms of the selection bias coefficient given by

β:=\IEmax{A,B}>0,

where $A$ and $B$ are two independent standard normal random variables. Our main tool in this calculation is Appendix C in Appendix C. Recall that

i=\argmaxj∈[1,2]˜Θ3,j.

By definition, $˜Θ3∼\Normal0(σ2+τ2+24)I2$ . Therefore, we have

\IE˜Θ3,i

= \sqrt{\frac{σ^{2} + τ^{2} + 2}{4}} \cdot β .

On the other hand, it follows from the symmetry that

	$\IE˜Θ3,i$	$=2\IE˜Θ3,1⋅I(i=1)$
		$=2\IE˜Θ3,1⋅I(˜Θ3,1≥˜Θ3,2).$

Using Appendix C for the sequence

A1:=Θ⋆12,        A2:=ε12,        and        A3:=˜Θ3−ˆΘ3∼\Normal012,

we infer that

	$\IEΘ⋆12⋅I(˜Θ3,1≥˜Θ3,2)$	$= \frac{σ^{2}}{σ^{2} + τ^{2} + 2} \cdot \frac{\sqrt{σ^{2} + τ^{2} + 2}}{4} \cdot β$
		$= \frac{σ^{2} β}{4 \sqrt{σ^{2} + τ^{2} + 2}} .$

Consequently, we can write

	$\IEΘ⋆i$	$=2\IEΘ⋆1⋅I(˜Θ3,1≥˜Θ3,2)$
		$=4\IEΘ⋆12⋅I(˜Θ3,1≥˜Θ3,2)$
		$=σ2β√σ2+τ2+2.\yesnumber$

Similarly, we can conclude that

\IEεi

= \frac{τ^{2} β}{\sqrt{σ^{2} + τ^{2} + 2}} .

(10)

Combining Equation 9 with Appendix A and Equation 10, we obtain

\inner[]\IE[]ˆΘ4,V

= \frac{(σ^{2} - τ^{2}) β}{6 \sqrt{σ^{2} + τ^{2} + 2}} .

This equality implies that ${ˆ Θ}_{4}$ is marginally biased whenever $σ^{2} \neq τ^{2}$ . Finally, Appendix A gives

	$\normSG[]\inner[]ˆΘ4,V$	$=\normSG[]ˆΘ3,1+ˆΘ3,2+Θ⋆i−εi6+ε33$
		$≤\normSG[]ˆΘ3,1+\normSG[]ˆΘ3,2+16\normSG[]Θ⋆i+16\normSG[]εi+13\normSG[]ε3$
		$≤√σ2+τ2+16\normSG[]Θ⋆i+16\normSG[]εi+13.$

Noting that

\normSG[]Θ⋆i=\normSG[]\absΘ⋆i≤\normSG[]\absΘ⋆1+\absΘ⋆2≤2\normSG[]\absΘ⋆1=2σ

and similarly for $ε_{i}$ , we get that

	$\normSG[]\inner[]ˆΘ4,V$	$\leq \sqrt{σ^{2} + τ^{2}} + \frac{1}{3} (σ + τ) + \frac{1}{3}$
		$\leq 2 (σ + τ) + 1.$

Therefore, we have

		$≤\normSG[]\inner[]ˆΘ4,V+\normSG[]\IE[]\inner[]ˆΘ4,V$
		$\leq 4 (σ + τ) + 2.$

This implies that the m.g.f. of $ˆΘ4−\IE[]ˆΘ4$ satisfies

\IEexp(s\inner[]V,ˆΘ4−\IE[]ˆΘ4)≤exp(s2(4σ+4τ+2)22),    for all s∈\IR.\forceqed

∎

Proof of Section 3.1.

It follows from Equation 1 that

\IE\inner[]ˆΘ3d+1,A=−C1√d

(11)

where $C1:=\absσ2−τ2⋅β12√σ2+τ2+2$ . Assuming $σ^{2} \neq τ^{2}$ , we observe $C_{1} > 0$ . Moreover, Equation 2 implies that

\IEexp(s(\inner[]ˆΘ3d+1,A+C1√d))≤exp(s2(4σ+4τ+2)22),    for all s∈\IR.

which means

\normSG[]\inner[]ˆΘ3d+1,A+C1√d

\leq (4 σ + 4 τ + 2) .

Using this inequality in combination with Equation 11, we assert the following concentration inequality

	$\IP\inner[]ˆΘ3d+1,A≤−C1√d2$	$=\IP\inner[]ˆΘ3d+1,A+C1√d≤C1√d2$
		$\geq 1 - exp (- C_{2} d),$

where $C_{2} := \frac{C_{1}^{2}}{8}$ .

Next, note that $\innerΘ⋆,A∼\Normal02σ2$ , and thus, we have

\IP\inner[]Θ⋆,A≥√2σ=1−Φ(1).

For sufficiently large values of $d$ , we have $exp (- C_{2} d) \leq \frac{1}{2} (1 - Φ (1))$ , and hence

\IP\inner[]Θ⋆,A≥√2σ  %and  \inner[]ˆΘ3d+1,A≤−C1√d2≥p0.\forceqed

∎

Proof of Section 3.1.

For $t > 3 d$ , let $Z_{t}$ be given by

Z_{t} := {\begin{matrix} 1 & if action A is never selected up % to time t, 0 & otherwise. \end{matrix}

We now have the following lower bound for the regret of Algorithm 1:

	$Regret (T, π^{LinTS}, P_{Θ^{⋆}})$	$≥\IPB⋅\IE√2σ⋅T∑t=3d+1Zt\givenB$
		$≥√2σp0⋅T∑t=3d+1\IEZt\givenB$
		$=√2σp0⋅T∑t=3d+1\IPZt\givenB.$

Define $q:=\IPZ3d+1\givenB$ . We get that

	$\IPZt\givenB$	$=\IPZt\givenB,Zt−1⋅\IPZt−1\givenB$
		$=q⋅\IPZt−1\givenB$
		$= q^{t - 3 d} .$

Furthermore, it follows from the definition of $q$ that

	$q$	$≥\IP\inner[]˜Θ3d+1−ˆΘ3d+1,A≤C1√d2$
		$≥\IP\Normal01≤C1√d2$
		$\geq 1 - exp (- C_{2} d) .$

By combining the above, we have that

	$Regret (T, π^{LinTS}, P_{Θ^{⋆}})$	$\geq \sqrt{2} σ p_{0} T \sum t = 3 d + 1 {(1 - exp (- C_{2} d))}_{2}^{t - 3 d}$
		$= \sqrt{2} σ p_{0} T - 3 d \sum t = 1 max {1 - t exp (- C_{2} d), 0} .$

This immediately follows that

Regret (T, π^{LinTS}, P_{Θ^{⋆}}) \geq \frac{\sqrt{2} σ p_{0}}{e} (min {T, exp (C_{2} d - 1)} - 3 d),

which demonstrates that the regret of Thompson sampling grows linearly up to time $exp (C_{2} d - 1)$ . ∎

Proof of Section 3.2.

Notice that

\innerΘ⋆,A′∼\Normal−μ√dσ2     and     \innerΘ⋆,A∼\Normalμ√d3σ2.

Therefore, $\innerΘ⋆,A≥12μ√d$ and $\innerΘ⋆,A′≤−12μ√d$ simultaneously with probability at least $1 - 2 exp (- \frac{μ^{2} d}{8 σ^{2}})$ . This thus implies that $A$ is the optimal arm with high probability. For sufficiently large $d$ , this probability exceeds $\frac{7}{8}$ .

On the other hand, at $t = 1$ , LinTS (Algorithm 1) will choose $A^{'}$ with probability $\frac{1}{4}$ . This holds true as $A^{'}$ is chosen if and only if

\inner[]˜Θ1,A′>0     and     \inner[]˜Θ1,A−A′<0.

The claim follows from the fact these two random variables are two centered and independent normal random variables. In this case, we have

Σ_{1} := I_{3 d} - \frac{1}{2} A^{'} A^{' ⊤} and

On Worst-case Regret of Linear Thompson Sampling

Authors

A General Framework to Analyze Stochastic Linear Bandit

Instance-Dependent Regret Analysis of Kernelized Bandits

Generalised Entropy MDPs and Minimax Regret

The price of unfairness in linear bandits with biased feedback

God Save the Queen

Variance-reduced Q-learning is minimax optimal

Minimax Policies for Combinatorial Prediction Games

1 Introduction

2 Setting and notation

3 Bayesian analyses are brittle

3.1 Noise reduction and LinTS’s failure

Bias-introducing action sets.

Stacking biased settings.

3.2 Mean shift and fixed action sets

4 Improving LinTS

4.1 Insights into LinTS’s optimism mechanism

4.2 Formal results

5 Simulations

5.1 Average failure time of LinTS

Noise reduction example.

Fixed action set example.

5.2 Thinness over time

Appendix A Proofs of Section 3

Proof of Section 3.1.

Proof of Section 3.1.

Proof of Section 3.1.

Proof of Section 3.2.

On Worst-case Regret of Linear Thompson Sampling

Authors

Related Research

A General Framework to Analyze Stochastic Linear Bandit

Instance-Dependent Regret Analysis of Kernelized Bandits

Generalised Entropy MDPs and Minimax Regret

The price of unfairness in linear bandits with biased feedback

God Save the Queen

Variance-reduced Q-learning is minimax optimal

Minimax Policies for Combinatorial Prediction Games

This week in AI

1 Introduction

2 Setting and notation

3 Bayesian analyses are brittle

3.1 Noise reduction and LinTS’s failure

Bias-introducing action sets.

Stacking biased settings.

3.2 Mean shift and fixed action sets

4 Improving LinTS

4.1 Insights into LinTS’s optimism mechanism

4.2 Formal results

5 Simulations

5.1 Average failure time of LinTS

Noise reduction example.

Fixed action set example.

5.2 Thinness over time

Appendix A Proofs of Section 3

Proof of Section 3.1.

Proof of Section 3.1.

Proof of Section 3.1.

Proof of Section 3.2.