Near-optimal Oracle-efficient Algorithms for Stationary and Non-Stationary Stochastic Linear Bandits

1 Introduction

A multi-armed bandit is the simplest model of decision making that involves the exploration versus exploitation trade-off [Lai and Robbins, 1985]. Linear bandits are an extension of multi-armed bandits where reward has linear structure with a finite-dimensional feature associated with each arm [Abe et al., 2003, Dani et al., 2008]. Two standard exploration strategies in stochastic linear bandits are Upper Confidence Bound algorithm (LinUCB) [Abbasi-Yadkori et al., 2011] and linear Thomson Sampling [Agrawal and Goyal, 2013]

. The former relies on optimism in face of uncertainty and is a deterministic algorithm built upon the construction of a high-probability confidence ellipsoid for the unknown parameter vector. The latter is a Bayesian solution that maximizes the expected rewards according to a parameter sampled from posterior distribution.

Chapelle and Li [2011] showed that linear Thompson Sampling empirically performs better is more robust to corrupted or delayed feedback than LinUCB. From a theoretical perspective, it enjoys a regret bound that is a factor of

\sqrt{d}

worse than minimax-optimal regret bound

~ Θ (d \sqrt{T})

that LinUCB enjoys. However, the minimax optimality of optimism comes at a cost: implementing UCB type algorithms can lead to NP-hard optimization problems even for convex action sets [Agrawal, 2019].

Random perturbation methods were originally proposed in the 1950s by Hannan [1957] in the full information setting where losses of all actions are observed. Kalai and Vempala [2005] showed that Hannan’s perturbation approach leads to efficient algorithms by making repeated calls to an offline optimization oracle. They also gave a new name to this family of randomized algorithms: Follow the Perturbed Leader (FTPL). Recent work [Abernethy et al., 2014, 2015, Kim and Tewari, 2019] has studied the relationship between FTPL algorithms and Follow the Regularized Leader (FTRL) algorithms and also investigated whether FTPL algorithms achieve minimax optimal regret in both full and partial information settings.

Abeille et al. [2017] viewed linear Thompson Sampling as a perturbation based algorithm, characterized a family of perturbations whose regrets can be analyzed, and raised an open problem to find a minimax-optimal perturbation. In addition to its significant role in smartly balancing exploration with exploitation, a perturbation based approach to linear bandits also reduces the problem to one call to the offline optimization oracle at each round. Recent works [Kveton et al., 2019a, b] have proposed randomized algorithms that use perturbation as a means to achieve oracle-efficient computation as well as better theoretical guarantee than linear Thompson Sampling, but there is still gap between their regret bounds and the lower bound of $Ω (d \sqrt{T})$ . This gap is logarithmic in the number of actions which can introduce extra dependence on dimensions for large or finite action spaces.

Linear bandit problems were originally motivated by applications such as online ad placement with features extracted from the ads and website users. However, users’ preferences often evolve with time which leads to interest in the non-stationary variant of linear bandits. Accordingly, adaptive algorithms that accommodate time-variation of environments have been studied in a rich line of works in both multi-armed bandit

[Besbes et al., 2014] and linear bandit. SW-LinUCB [Cheung et al., 2019] and D-LinUCB [Russac et al., 2019] were constructed on the basis of optimism principle under the uncertainty via sliding window and exponential discounting weights, respectively. Luo et al. [2017] and Chen et al. [2019] studied fully adaptive and oracle-efficient algorithms assuming access to an optimization oracle. It is still open problem to design a practically simple, oracle-efficient and statistically optimal algorithm for non-stationary linear bandits.

Contributions

We design and analyze two algorithms that enjoy not only computational efficiency (assuming access to an offline optimization oracle), but also statistical optimality (in terms of regret) in linear bandit problems.

In section 2, we consider a stationary environment and present an algorithm called Follow-The-Gaussian-Perturbed Leader (FTGPL) that (1) achieves the minimax-optimal $~ O (d \sqrt{T})$ regret, (2) matches the empirical performance of Linear Thompson Sampling, and (3) can be efficiently implemented given oracle access to the offline optimization problem. Furthermore, it solves an open problem raised in Abeille et al. [2017], namely finding a randomized algorithm that achieves minimax-optimal frequentist regret in stochastic linear bandits.

In section 3, we study the non-stationary setting and propose the weighted variant of FTGPL with exponential discounting, called Discounted Follow-The-Gaussian-Perturbed Leader (D-FTGPL). It gracefully adjusts to the total time-variation in the true parameter so that it enjoys not only a regret bound comparable to those of SW-LinUCB and D-LinUCB, but also computational efficiency due to sole reliance on an offline optimization oracle for the action set.

2 Stationary Stochastic Linear Bandit

2.1 Preliminary

In stationary stochastic linear bandit, a learner chooses an action $X_{t}$ from a given action set $X_{t} \subset R^{d}$ in every round $t$ , and he subsequently observes reward $Y_{t} = ⟨ X_{t}, θ^{⋆} ⟩ + η_{t}$ where $θ^{⋆} \in R^{d}$ is an unknown parameter and $η_{t}$

is conditionally 1-subGaussian random variable. For simplicity, assume that for all

x \in X_{t}

∥ x ∥_{2} \leq 1

∥ θ^{⋆} ∥_{2} \leq 1

, and thus

| ⟨ x, θ^{⋆} ⟩ |_{2} \leq 1

As a measure of evaluating a learner, the regret is defined as the difference between rewards the learner would have received had he played the best in hindsight, and the rewards he actually received. Therefore, minimizing the regret is equivalent to maximizing the expected cumulative reward. Denote the best action in a round $t$ as $x_{t}^{⋆} = arg {max}_{x \in X_{t}} ⟨ x, θ^{⋆} ⟩$ and the expected regret as $E [R (T)] = E [\sum_{t = 1}^{T} ⟨ x_{t}^{⋆}, θ^{⋆} ⟩ - ⟨ X_{t}, θ^{⋆} ⟩]$ .

To learn about unknown parameter $θ^{⋆}$ from history up to time $t - 1$ , $H_{t - 1} = {(X_{l}, Y_{l})_{1 \leq l \leq t - 1}}$ , algorithms heavily rely on $l^{2}$

-regularized least-squares estimate of

θ^{⋆}

{^θ}_{t}

, and confidence ellipsoid centered from

{^θ}_{t}

. Here, we define

{^θ}_{t} = V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Y_{l}

, where

V_{t, λ} = λ I_{d} + \sum_{l = 1}^{t - 1} X_{l} X_{l}^{T}

2.2 Open Problem : The Best of Three Worlds

The standard solutions in stationary stochastic linear bandit are optimism based algorithm (LinUCB, Abbasi-Yadkori et al. [2011]) and linear Thompson Sampling (LinTS, Abeille et al. [2017]). While the former obtains the theoretically optimal regret bound $~ O (d \sqrt{T})$ matched to lower bound $Ω (d \sqrt{T})$ , the latter empirically performs better in spite of regret bound $\sqrt{d}$ worse than LinUCB [Chapelle and Li, 2011]. Also, some recent works proposed a series of randomized algorithms for (Generalized) linear bandit; PHE [Kveton et al., 2019a], FPL-GLM [Kveton et al., 2019b], and RandUCB [Vaswani et al., 2019]. They are categorized in terms of regret bounds, randomness, and oracle access in Table 1.

Algorithm	Regret	Random	Oracle
LinUCB	$~ O (d \sqrt{T})$	No	No
LinTS	$~ O (d^{3 / 2} \sqrt{T})$	Yes	Yes
PHE	$~ O (d \sqrt{T log \| X_{t} \|})$	Yes	Yes
FPL-GLM	$~ O (d \sqrt{T log \| X_{t} \|})$	Yes	Yes
randUCB	$~ O (d \sqrt{T})$	Yes	No
FTGPL	$~ O (d \sqrt{T})$	Yes	Yes

Table 1: Stationary Stochastic Linear Bandit

PHE and FPL-GLM algorithms allow efficient implementation in that they are designed to choose an action by maximizing the expected reward where historical rewards are perturbed via Binomial and Gaussian distributions,

	$X_{t}$	$= arg max x \in X_{t} {~ f}_{t} (x) = arg max x \in X_{t} ⟨ x, {~ θ}_{t} ⟩$
	$where {~ θ}_{t}$	$= arg min θ \in R^{d} L_{θ} ({(X_{l}, Y_{l} + Z_{l})}_{l = 1}^{t - 1}), Z_{l} \sim D .$

But they are limited in that regret bounds, $~ O (d \sqrt{T log | X_{t} |})$ , depend on the number of arms, $| X_{t} |$ , and theoretically unavailable when action set is not finite.

RandUCB [Vaswani et al., 2019] is a randomized version of LinUCB by randomly sampling confidence level from a certain distribution $D$ by setting ${~ f}_{t} (x) = ⟨ x, {^θ}_{t} ⟩ + Z \cdot ∥ x ∥_{V_{t, λ}^{- 1}}$ where $Z \sim D$ . It requires computation of $∥ x ∥_{V_{t, λ}^{- 1}}$ for all actions $x \in X_{t}$ like LinUCB so that it cannot be efficiently implemented in an infinite-arms setting, while it achieves theoretically optimal regret bounds of LinUCB as well as matches empirical performance of LinTS.

It is an open problem to construct algorithms that are random, theoretically optimal in regret bound, and efficient in implementation, thus achieving the best of three worlds.

2.3 Follow The Gaussian Perturbed Leader (FTGPL)

Input:

λ > 0, a > 0

Initialize

V_{1} = λ I_{d}

and

b_{1} = 0

for

t = 1

T

t \leq d

then

Randomly play

X_{t} \in X_{t}

and receive reward

Y_{t}

else

Obtain

{~ θ}_{t} = V_{t, λ}^{- 1} b_{t}

Oracle :

X_{t} = arg {max}_{x \in X_{t}} ⟨ x, {~ θ}_{t} ⟩

Play action

X_{t}

and receive reward

Y_{t}

end if

Update

V_{t + 1, λ} = V_{t, λ} + X_{t} X_{t}^{T}

Update

b_{t + 1} = b_{t} + X_{t} (Y_{t} + Z_{t})

where

Z_{t} \sim N (0, a^{2})

end for

Algorithm 1 Follow The Gaussian Perturbed Leader (FTGPL)

While the regret analysis for Linear Thompson sampling [Abeille et al., 2017] is based on anti-concentration event that perturbed estimate ${~ θ}_{t}$ be spread widely enough from least-squares estimate ${^θ}_{t}^{l s}$ in $d$ -dimensional space, the general framework for regret analysis that PHE, FTP-GLM, and RandUCB algorithms share depends on anti-concentration event that perturbed expected reward ${~ f}_{t} (x_{t}^{⋆})$ is apart enough from expected reward $⟨ x_{t}^{⋆}, θ_{t}^{l s} ⟩$ for the best action $x^{⋆}$ as,

Eantit={|~f(x⋆t)−⟨x⋆t,^θlst⟩|>c1∥x⋆t∥V−1t,λ},

for a certain $c_{1} \geq 1$ . This dimension reduction in parameter space from $d$ to $1$ is underlying intuition how extra $\sqrt{d}$ gap in regret is eliminated.

Define concentration event that perturbed expected reward ${~ f}_{t} (x)$ is close enough to expected reward $⟨ x, θ_{t}^{l s} ⟩$ for all $x \in X_{t}$ as,

Econct={∀x∈X;|~ft(x)−⟨x,^θlst⟩|≤c2∥x∥V−1t,λ}.

for a certain $c_{2} \geq c_{1}$ . In PHE and FPL-GLM, ${~ f}_{t} (x)$ is set as $⟨ x, {^θ}_{t}^{l s} ⟩ + V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ , and $| X_{t} |$ different inequalities should hold simultaneously to make $E_{t}^{c o n c}$ holds, which adds extra $log | X_{t} |$ term in regret bound. While, RandUCB does not produce extra $log | X_{t} |$ term since $E_{t}^{c o n c}$ is reduced to single inequality independent of $∥ x ∥_{V_{t, λ}^{- 1}}$ term by setting ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{l s} ⟩ + Z \cdot ∥ x ∥_{V_{t, λ}^{- 1}}$ .

Motivation

The natural question arises, which form of ${~ f}_{t} (x)$ enables us to have both efficient implementation and optimal regret bound? I firmly have an answer, ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{l s} ⟩ + x_{i}^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ where $Z_{l} \sim N (0, a^{2})$ . The motivation is that ${~ f}_{t}$ between RandUCB and FTGPL algorithm are approximately equivalent once corresponding perturbations are Gaussian due to its linear invariant property,

x^{T} V_{t, λ}^{- 1} t - 1 \sum l = 1 X_{l} Z_{l} ≃ Z \cdot ∥ x ∥_{V_{t, λ}^{- 1}}

where $Z_{l}, Z \sim N (0, a^{2})$ .

Oracle point of view

We assume that our learner has access to an algorithm that returns a near-optimal solution to the offline problem, called an offline optimization oracle. It returns the optimal action that maximizes the expected reward from a given action space $X \subset R^{d}$ when a parameter $θ \in R^{d}$ is given as input.

Definition 1 (Offline Optimization Oracle).

There exists an algorithm, $A . M . O .$ , which when given a pair of action space $X \subset R^{d}$ , and a parameter $θ \in R^{d}$ , computes

A . M . O . (X, θ) = arg max x \in X ⟨ x, θ ⟩ .

In comparison with LinUCB and RandUCB algorithms required to compute spectral norms of all actions in every round, the main advantage of using FTPGL algorithm is that it relies on an offline optimization oracle in every round $t$ so that the optimal action can be efficiently obtained within polynomial times even from infinite action set $| X_{t} | = \infty$ , as well as its regret bound is also independent of size of action set, $| X_{t} |$ .

Perspective of Linear Thompson Sampling

Another interpretation is from a perspective of Linear Thompson Sampling [Abeille et al., 2017] where perturbed estimate is defined as,

{~ θ}_{t} = {^θ}_{t}^{l s} + a \cdot V_{t, λ}^{- 1 / 2} η_{t}, where η_{t} \sim D .

It is equivalent to Linear Thompson Sampling where randomly sampled vector $η_{t}$ is $V_{t, λ}^{- 1 / 2} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ where $Z_{l} \sim N (0, a^{2})$ . Generally, linear Thompson sampling empirically performs well but the existence of minimax-optimal perturbation in linear Thompson sampling has been open [Abeille et al., 2017], and we provide the FTPGL algorithm as an answer for this open problem.

2.4 Analysis

2.4.1 General Framework for Regret Analysis

We construct a general regret bound for linear bandit algorithm on the top of prior work of Kveton et al. [2019a]. The difference from their work is that action sets $X_{t}$ vary from time $t$ and can have infinite arms. Firstly, we define three three events as follow,

	$Els={∀x∈Xt,∀t∈[T];\|⟨x,^θlst−θ⋆⟩\|≤c1∥x∥V−1t,λ},$
	$Econct={∀x∈Xt;\|~ft(x)−⟨x,^θlst⟩\|≤c2∥x∥V−1t,λ},$
	$Eantit={\|~ft(x⋆t)−⟨x⋆t,^θlst⟩\|>c1∥x⋆t∥V−1t,λ}.$

The choice of ${~ f}_{t} (x)$ is made by algorithmic design, which decides choices on both $c_{1}$ and $c_{2}$ simultaneously. At time $t$ , the general algorithm which maximizes perturbed expected reward ${~ f}_{t} (x)$ over action space $X_{t}$ is considered. The following theorem and lemma are simple extension of Theorem 1 and Lemma 2 from the work of Kveton et al. [2019a].

Theorem 2 (Expected Regret).

Assume we have $c_{1}, c_{2} \geq 1$ satisfying $P (E^{l s}) \geq 1 - p_{1}$ , $P (E_{t}^{c o n c}) \geq 1 - p_{2}$ , and $P (E_{t}^{a n t i}) \geq p_{3}$ , and $c_{3} = 2 d log (1 + T / (d λ))$ . Let $A$ be an algorithm that chooses arm $X_{t} = arg {max}_{X_{t}} {~ f}_{t} (x)$ at time $t$ . Then the expected regret of $A$ is bounded as

E [R (T)] \leq (c_{1} + c_{2}) (1 + \frac{2}{p_{3} - p_{2}}) \sqrt{c_{3} T} + T (p_{1} + p_{2}) + d .

We defer the proof of Theorem 2 to Appendix B.1, since a similar regret bound in non-stationary setting will be carefully studied in Theorem 12. It is more general regret analysis in that non-stationary setting contains stationary setting as a special case where total variation of rewards, $B_{T}$ is zero.

2.4.2 Expected Regret of FTGPL

We set ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{l s} ⟩ + x^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ , where $Z_{l} \sim N (0, a^{2})$ in FTGPL algorithm. At each time $t$ , it chooses an action $X_{t} = arg {max}_{x \in X_{t}} {~ f}_{t} (x)$ by oracle property.

Corollary 3 (Expected Regret of FTGPL).

Assume we have

	$c_{1}$	$= \sqrt{d log T + d log (1 + T / (d λ))} + λ^{1 / 2}$
	$c_{2}$	$= a \sqrt{2 log T}, a^{2} = 14 c_{1}^{2}, λ = λ_{min} (V_{d + 1, λ}) / 2$

Let A be a Follow-The-Gaussian-Perturbed-Leader algorithm with $Z_{l} \sim N (0, a^{2})$ . Then the expected regret of $A$ is bounded by $~ O (d \sqrt{T})$ where logarithmic terms in $T$ and constants are ignored in $~ O$ .

The optimal choices of $c_{1}, c_{2}, a$ and $λ$ heavily depends on the following three lemmas that control the probabilities of three events $E^{l s}, E_{t}^{c o n c}$ , and $E_{t}^{a n t i}$ . The main contribution over previous works [Kveton et al., 2019a, b] is that the parameter $c_{2}$ is chosen independent of the number of action sets, $X_{t}$ thanks to an invariant property of Gaussian distributions over linear combination. Thus, our analysis is novel in that it can be extended to infinite arm setting as well as contextual bandit setting.

Compared to standard solutions in stochastic linear bandit, the FTTGPL algorithm (Algorithm 1) is not only theoretically guaranteed to have the regret bound comparable to that of LinUCB [Abbasi-Yadkori et al., 2011], but also shows empirical performance and computational efficiency on par with Linear Thompson sampling algorithm [Agrawal and Goyal, 2013, Abeille et al., 2017], thus achieving the best of both worlds.

Lemma 4 (Least-Squares Confidence Ellipsoid, Theorem 2 [Abbasi-Yadkori et al., 2011]).

For any $λ > 0$ , and $c_{1} = \sqrt{d log T + d log (1 + T / (d λ))} + λ^{1 / 2}$ , then the event $E^{l s}$ holds with probability at least $1 - 1 / T$ .

Lemma 5 (Concentration).

Given history $H_{t - 1}$ , suppose ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{l s} ⟩ + x^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ , where $Z_{l} \sim N (0, a^{2})$ , and $c_{2} = a \sqrt{2 log T}$ . Then, $P ({¯ E}_{t}^{c o n c}) \leq 1 / T$ .

Proof.

Given history $H_{t - 1}$ , $x^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ is equivalent to $\sqrt{\sum_{l = 1}^{t - 1} x^{T} V_{t, λ}^{- 1} X_{l}} \cdot Z$ where $Z \sim N (0, a^{2})$ by linearity of Gaussian distributions. Recall $V_{t, 0} = \sum_{l = 1}^{t - 1} X_{l} X_{l}^{T}$ by definition.

		$P (E_{t}^{c o n c})$
	$=$	$P (\forall x \in X_{t}; \| x^{T} V_{t, λ}^{- 1} t - 1 \sum l = 1 X_{l} Z_{l} \| \leq c_{2} ∥ x ∥_{V_{t, λ}^{- 1}})$
	$=$	$P (\forall x \in X_{t}; \| Z \| \leq \frac{c_{2} ∥ x ∥_{V_{t, λ}^{- 1}}}{\sqrt{x^{T} V_{t, λ}^{- 1} V_{t, 0} V_{t, λ}^{- 1} x}})$
	$\geq$	$P (\forall x \in X_{t}; \| Z \| \leq \frac{c_{2} ∥ x ∥_{V_{t, λ}^{- 1}}}{\sqrt{x^{T} V_{t, λ}^{- 1} V_{t, 0} + λ I) V_{t, λ}^{- 1} x}})$
	$=$	$P (\| Z \| \leq c_{2}) \geq 1 - 1 / T where c_{2} = a \sqrt{2 log T} .$

The last equality holds since the event becomes independent on action $x \in X_{t}$ after cancelling out $∥ x ∥_{V_{t, λ}^{- 1}}$ on the right hand side of event condition. ∎

Lemma 6 (Anti-concentration).

Given $H_{t - 1}$ , suppose ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{l s} ⟩ + x^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ , $Z_{l} \sim N (0, a^{2})$ . Then, $P (E_{t}^{a n t i}) \geq \frac{1}{4 \sqrt{π}} exp (- \frac{7 c_{1}^{2}}{a^{2} (1 - λ / λ_{min} (V_{d + 1, λ}))})$ .
Furthermore, once we set $λ = \frac{λ_{min} (V_{d + 1, λ})}{2}$ and $a^{2} = 14 c_{1}^{2}$ , $P (E_{t}^{a n t i}) \geq e^{- 1} / (4 π)$ .

Proof.

By Gaussian linearity, $x^{T} V_{t, λ}^{- 1} \sum_{l = 1}^{t - 1} X_{l} Z_{l}$ is equivalent to $\sqrt{\sum_{l = 1}^{t - 1} x^{T} V_{t, λ}^{- 1} X_{l}} \cdot Z$ where $Z \sim N (0, a^{2})$ given history $H_{t - 1}$ .

P (E_{t}^{a n t i}) = P (| x_{t}^{⋆ T} V_{t, λ}^{- 1} t - 1 \sum l = 1 X_{l} Z_{l} | \geq c_{1} ∥ x_{t}^{⋆} ∥_{V_{t, λ}^{- 1}})

	$= P (\| Z \| \geq \frac{c_{1} ∥ x_{t}^{⋆} ∥_{V_{t, λ}^{- 1}}}{\sqrt{x_{t}^{⋆ T} V_{t, λ}^{- 1} V_{t, 0} V_{t, λ}^{- 1} x_{t}^{⋆}}})$
	$\geq \frac{1}{4 \sqrt{π}} exp (- \frac{7 c_{1}^{2} ∥ x_{t}^{⋆} ∥_{V_{t, λ}^{- 1}}^{2}}{a^{2} \cdot x_{t}^{⋆ T} V_{t, λ}^{- 1} V_{t, 0} V_{t, λ}^{- 1} x_{t}^{⋆}})$
	$\geq \frac{1}{4 \sqrt{π}} exp (- \frac{7 c_{1}^{2} ∥ x_{t}^{⋆} ∥_{V_{t, λ}^{- 1}}^{2}}{a^{2} \cdot ∥ x_{t}^{⋆} ∥_{V_{t, λ}^{- 1}}^{2} - a^{2} \cdot λ x_{t}^{⋆ T} V_{t, λ}^{- 2} x_{t}^{⋆}})$
	$\geq \frac{1}{4 \sqrt{π}} exp (- \frac{7 c_{1}^{2}}{a^{2} (1 - λ / λ_{min} (V_{d + 1, λ}))})$
	$= \frac{1}{4 \sqrt{π}} exp (- 1) where a^{2} = 14 c_{1}^{2}, λ = \frac{λ_{min} (V_{d + 1, λ})}{2} .$

The first inequality holds because of anti-concentration of Gaussian distribution (Lemma A). ∎

3 Non-Stationary Stochastic Linear Bandit

3.1 Preliminary

In each round $t \in [T]$ , an action set $X_{t} \in R^{d}$ is given to the learner and he has to choose an action $X_{t} \in X_{t}$ . Then, the reward $Y_{t} = ⟨ X_{t}, θ_{t}^{⋆} ⟩ + η_{t}$ is observed to the learner. $θ_{t}^{⋆} \in R^{d}$ is an unknown time-varying parameter and $η_{t}$ is a conditionally 1-subGaussin random variable. Unlike stationary setting, the non-stationary variant allows unknown parameter $θ_{t}^{⋆}$ to be time-variant within total variation $B_{T} = \sum_{t = 1}^{T - 1} ∥ θ_{t}^{⋆} - θ_{t + 1}^{⋆} ∥_{2}$ . It is a great way of quantifying time-variations of $θ_{t}^{⋆}$ in that it covers both slowly-changing and abruptly-changing environments. Simply, assume that for all $x \in X_{t}$ , $∥ x ∥_{2} \leq 1$ , $∥ θ_{t}^{⋆} ∥_{2} \leq 1$ , and thus $| ⟨ x, θ_{t}^{⋆} ⟩ |_{2} \leq 1$ .

In a similar way to stationary setting, denote the best action in a round $t$ as $x_{t}^{⋆} = arg {max}_{x \in X_{t}} ⟨ x, θ_{t}^{⋆} ⟩$ and the expected dynamic regret as $E [R (T)] = E [\sum_{t = 1}^{T} ⟨ x_{t}^{⋆}, θ_{t}^{⋆} ⟩ - ⟨ X_{t}, θ_{t}^{⋆} ⟩]$ where $X_{t}$ is chosen action at time $t$ . The goal of the learner is to minimize the expected dynamic regret.

3.2 Open Problem

In a stationary stochastic environment where reward has a linear structure, Linear Upper Confidence Bound algorithm (LinUCB) follows a principle of optimism in the face of uncertainty. Under this principle, two recent works of Wu et al. [2018], Russac et al. [2019] proposed SW-LinUCB and D-LinUCB, which are non-stationary variants of LinUCB to adapt to time-variation of $θ_{t}^{⋆}$ . They rely on weighted least-squares estimators with equal weights only given to recent $w$ observations where $w$ is length of a sliding-window, and exponentially discounted weights, respectively.

As described in Table 2, SW-LinUCB and D-LinUCB are deterministic as well as asymptotically achieve the minimax optimal dynamic regret $Θ (d^{2 / 3} B_{T}^{1 / 3} T^{2 / 3})$ , but share inefficiency of implementation with LinUCB and RandUCB algorithms in that the computation of spectral norms of all actions are required.

It is still open to find algorithms that are theoretically optimal in regret bound and efficient in implementation even in infinite action space.

Algorithm	Regret	Random	Oracle
D-LinUCB	$O (d^{\frac{2}{3}} B_{T}^{\frac{1}{3}} T^{\frac{2}{3}})$	No	No
SW-LinUCB	$O (d^{\frac{2}{3}} B_{T}^{\frac{1}{3}} T^{\frac{2}{3}})$	No	No
D-FTGPL	$O (d^{\frac{2}{3}} B_{T}^{\frac{1}{3}} T^{\frac{2}{3}})$	Yes	Yes

Table 2: Non-Stationary Stochastic Linear Bandit

3.3 Weighted Least-Squares Estimator

We firstly study the weighted least-squares estimator with discounting factor $0 < γ < 1$ . In round $t$ , the weighted least-squares estimator is obtained in both implicit and closed forms,

	${^θ}_{t}^{w l s}$	$= arg max θ t - 1 \sum l = 1 γ^{t - l} (Y_{l} - ⟨ X_{l}, θ ⟩)^{2} + \frac{λ γ^{- t}}{2} ∥ θ ∥_{2}^{2}$
		$= W_{t, λ}^{- 1} t - 1 \sum s = 1 γ^{- l} X_{l} Y_{l},$

where $W_{t, λ} = \sum_{l = 1}^{t - 1} γ^{- l} X_{l} X_{l}^{T} + λ γ^{- t} I_{d}$ . Additionally, we define ${~ W}_{t, λ} = \sum_{l = 1}^{t - 1} γ^{- 2 l} X_{l} X_{l}^{T} + λ γ^{- 2 t} I_{d}$ . This form is closely connected with the covariance matrix of ${^θ}_{t}^{w l s}$ . For simplicity in notations, we define $V_{t} = W_{t, λ} {~ W}_{t, λ}^{- 1} W_{t, λ}$ .

Lemma 7 (Weighted Least-Sqaures Confidence Ellipsoid, Theorem 1 [Russac et al., 2019]).

Assume the stationary setting where $θ_{t}^{⋆} = θ^{⋆}$ and thus $B_{T} = 0$ . For any $δ > 0$ , $P (\forall t \geq 1, ∥ {^θ}_{t}^{w l s} - θ^{⋆} ∥_{W_{t, λ} {~ W}_{t, λ}^{- 1} W_{t, λ}} \leq β_{t}) \geq 1 - δ$ where $β_{t} = \sqrt{λ} + \sqrt{2 log (1 / δ) + d log (1 + \frac{(1 - γ^{2 t})}{λ d (1 - γ^{2})})}$ .

While Lemma 7 states that the confidence ellipsoid $C_{t} = {θ \in R^{d} : ∥ θ - θ_{t}^{w l s} ∥_{W_{t, λ} {~ W}_{t, λ}^{- 1} W_{t, λ}} \leq β_{t}}$ contains true parameter $θ_{t}^{⋆}$ with high probability in stationary setting, the true parameter $θ_{t}^{⋆}$ is not necessarily inside the confidence ellipsoid $C_{t}$ in the non-stationary setting because of variation on the parameters. We alternatively define a surrogate parameter ${¯ θ}_{t} = W_{t, λ}^{- 1} (\sum_{l = 1}^{t - 1} γ^{- l} X_{l} X_{l}^{T} θ_{l}^{⋆} + λ γ^{- (t - 1)} θ_{t}^{⋆})$ , which belongs to $C_{t}$ with probability at least $1 - δ$ , which is formally stated in Lemma 9.

3.4 Discounted Follow The Gaussian Perturbed Leader (D-FTGPL)

The idea of perturbing history using Gaussian distribution in FTGPL algorithm (Algorithm 1) can be directly applied to non-stationary setting by replacing ${^θ}_{t}^{l s}$ with ${^θ}_{t}^{w l s}$ . Accordingly, we would set ${~ f}_{t} (x) = ⟨ x, {^θ}_{t}^{w l s} ⟩ + x V_{t}^{- 1} \sum_{l = 1}^{t - 1} γ^{- l} X_{l} Z_{l}$ where $Z_{l} \sim N (0, a^{2})$ in D-FTGPL algorithm (Algorithm 2).

Motivation

Under the optimism of face in uncertainty, D-LinUCB [Russac et al., 2019] chooses an action by maximizing the upper confidence bound of expected regret based on ${^θ}_{t}^{w l s}$ and confidence level $a$ is replaced by a random variable $Z$ in non-stationary variant of RandUCB algorithm Vaswani et al. [2019] with the comparable theoretical guarantee,

	$D-LinUCB : X_{t}$	$= arg max x \in X_{t} ⟨ x, {^θ}_{t}^{w l s} ⟩ + a \cdot ∥ x ∥_{V_{t}^{- 1}}$
	$RandUCB-Non : X_{t}$	$= arg max x \in X_{t} ⟨ x, {^θ}_{t}^{w l s} ⟩ + Z \cdot ∥ x ∥_{V_{t}^{- 1}} .$

D-FTGPL algorithm is proposed from two motivations that it is approximately equivalent to non-stationary variant of RandUCB algorithm as

x^{T} V_{t}^{- 1} t - 1 \sum l = 1 γ^{- l} X_{l} Z_{l} ≃ Z \cdot ∥ x ∥_{V_{t}^{- 1}}

where $Z_{l}, Z \sim N (0, a^{2})$ , but its innate perturbation allows this algorithm to have an arg-max oracle access ( $A . M . O .$ ) differently from D-LinUCB and SW-LinUCB. Therefore, D-FTPGL algorithm can be efficient in computation though infinite action set $| X_{t} | = \infty$ is considered.

Input:

λ > 0

0 < γ < 1

, and

a > 0

Initialize

W_{1} = λ I_{d}

and

{¯ b}_{1} = 0

for

t = 1

T

t \leq d

then

Randomly play

X_{t} \in X_{t}

and receive reward

Y_{t}

else

Obtain

{~ θ}_{t} = W_{t, λ}^{- 1} {¯ b}_{t}

Oracle :

X_{t} = arg {max}_{x \in X_{t}} ⟨ x, {~ θ}_{t} ⟩

Play action

X_{t}

and receive reward

Y_{t}

end if

Update

W_{t + 1, λ} = γ W_{t, λ} + X_{t} X_{t}^{T} + (1 - γ) λ I_{d}

Update

{¯ b}_{t + 1} = γ {¯ b}_{t} + X_{t} (Y_{t} + Z_{t})

where

Z_{t} \sim N (0, a^{2})

end for

Algorithm 2 Discounted Follow The Gaussian Perturbed Leader (D-FTGPL)

3.5 Analysis

The framework of regret analysis in Section 2.4 is extended to non-stationary setting where true parameter $θ_{t}^{⋆}$ changes within total variation $B_{T}$ . Dynamic regret is decomposed into surrogate regret and bias arising from total variation.

	$E [R (T)] = T \sum t = 1 E [⟨ x_{t}^{⋆} - X_{t}, θ_{t}^{⋆} ⟩]$
	$= T \sum t = 1 E [⟨ x_{t}^{⋆} - X_{t}, {¯ θ}_{t} ⟩] + T \sum t = 1 E [⟨ x_{t}^{⋆} - X_{t}, θ_{t}^{⋆} - {¯ θ}_{t} ⟩]$
	$\leq T \sum t = 1 E [⟨ x_{t}^{⋆} - X_{t}, {¯ θ}_{t} ⟩] + 2 T \sum t = 1 ∥ θ_{t}^{⋆} - {¯ θ}_{t} ∥_{2}$

3.5.1 Surrogate Instantaneous Regret

To bound the surrogate instantaneous regret $⟨ x_{t}^{⋆} - X_{t}, {¯ θ}_{t} ⟩$ , we newly define three events $E^{w l s}, E_{t}^{c o n c}$ , and $E_{t}^{a n t i}$ :

	$Ewls={∀x∈Xt,t∈[T];\|⟨x,^θwlst−¯θt⟩\|≤c1∥x∥V−1t},$
	$Econct={∀x∈Xt;\|~ft(x)−⟨x,^θwlst⟩\|≤c2∥x∥V−1t},$
	$Eantit={\|~ft(x⋆t)−⟨x⋆t,^θwlst⟩\|>c1∥x⋆t∥V−1t}.$

Theorem 8.

Assume we have $c_{1}, c_{2} \geq 1$ satisfying $P (E^{w l s}) \geq 1 - p_{1}$ , $P (E_{t}^{c o n c}) \geq 1 - p_{2}$ , and $P (E_{t}^{a n t i}) \geq p_{3}$ , and $c_{3} = 2 d log (\frac{1}{γ}) + 2 \frac{d}{T} log (1 + \frac{1}{d λ (1 - γ)})$ . Let $A$ be an algorithm that chooses arm $X_{t} = arg {max}_{X_{t}} {~ f}_{t} (x)$ at time $t$ . Then the expected surrogate instantaneous regret of $A$ , $E [⟨ x_{t}^{⋆} - X_{t}, {¯ θ}_{t} ⟩]$ is bounded by

p_{2} + (c_{1} + c_{2}) (1 + \frac{2}{p_{3} - p_{2}}) E_{t} [min (1, ∥ X_{t} ∥_{V_{t}^{- 1}})] .

Proof.

Define $Δ_{x} = ⟨ x_{t}^{⋆} - x, {¯ θ}_{t} ⟩$ . Given $H_{t - 1}$ where $E^{w l s}$ holds, let ${¯ S}_{t} = {x \in X_{t} : (c_{1} + c_{2}) ∥ x ∥_{V_{t}^{- 1}} \geq Δ_{x} and Δ_{x} \geq 0}$ be the set of arms that are under-sampled and worse than $x_{t}^{⋆}$ given ${¯ θ}_{t}$ in round $t$ . Among them, let $U_{t} = arg {min}_{x \in {¯ S}_{t}} ∥ x ∥_{V_{t}^{- 1}}$ be the least uncertain under-sampled arm in round $t$ . By definition, $x_{t}^{⋆} \in {¯ S}_{t}$ . The set of sufficiently sampled arms is defined as $S_{t} = {x \in X_{t} : (c_{1} + c_{2}) ∥ x ∥_{V_{t}^{- 1}} \leq Δ_{x} % a n d Δ_{x} \geq 0}$ and let $c = c_{1} + c_{2}$ .

The proof is different from that of Theorem 15 in that actions $x \in X_{t}$ with $Δ_{x} < 0$ can be neglected since the regret induced by these actions is upper bounded by zero. $U_{t}$ is deterministic term while $X_{t}$ is random because of innate perturbation in ${~ f}_{t}$ . Thus surrogate instantaneous regret can be bounded as,

	$Δ_{X_{t}} = Δ_{U_{t}} + ⟨ U_{t}, {¯ θ}_{t} ⟩ - ⟨ X_{t}, {¯ θ}_{t} ⟩$
	$\leq Δ_{U_{t}} + {~ f}_{t} (U_{t}) - {~ f}_{t} (X_{t}) + c ∥ X_{t} ∥_{V_{t}^{- 1}} + c ∥ U_{t} ∥_{V_{t}^{- 1}}$
	$\leq c ∥ X_{t} ∥_{V_{t}^{- 1}} + 2 c ∥ U_{t} ∥_{V_{t}^{- 1}} .$

Thus, the expected surrogate instantaneous regret can be bounded as,

	$E_{t} [Δ_{X_{t}}]$	$= E_{t} [Δ_{X_{t}} I {E_{t}^{c o n c}}] + E_{t} [Δ_{X_{t}} I {{¯ E}_{t}^{c o n c}}]$
		$\leq c E_{t} [∥ X_{t}$

Near-optimal Oracle-efficient Algorithms for Stationary and Non-Stationary Stochastic Linear Bandits

Related Research

Online Non-Convex Learning: Following the Perturbed Leader is Optimal

Non-stationary Linear Bandits Revisited

Combinatorial Semi-Bandit in the Non-Stationary Environment

Follow the Perturbed Leader: Optimism and Fast Parallel Algorithms for Smooth Minimax Games

Follow-the-Perturbed-Leader for Adversarial Markov Decision Processes with Bandit Feedback

Kiefer Wolfowitz Algorithm is Asymptotically Optimal for a Class of Non-Stationary Bandit Problems

Generalized Policy Elimination: an efficient algorithm for Nonparametric Contextual Bandits

Code Repositories

NonstationaryLB

1 Introduction

Contributions

2 Stationary Stochastic Linear Bandit

2.1 Preliminary

2.2 Open Problem : The Best of Three Worlds

2.3 Follow The Gaussian Perturbed Leader (FTGPL)

Motivation

Oracle point of view

Definition 1 (Offline Optimization Oracle).

Perspective of Linear Thompson Sampling

2.4 Analysis

2.4.1 General Framework for Regret Analysis

Theorem 2 (Expected Regret).

2.4.2 Expected Regret of FTGPL

Corollary 3 (Expected Regret of FTGPL).

Lemma 4 (Least-Squares Confidence Ellipsoid, Theorem 2 [Abbasi-Yadkori et al., 2011]).

Lemma 5 (Concentration).

Proof.

Lemma 6 (Anti-concentration).

Proof.

3 Non-Stationary Stochastic Linear Bandit

3.1 Preliminary

3.2 Open Problem

3.3 Weighted Least-Squares Estimator

Lemma 7 (Weighted Least-Sqaures Confidence Ellipsoid, Theorem 1 [Russac et al., 2019]).

3.4 Discounted Follow The Gaussian Perturbed Leader (D-FTGPL)

Motivation

3.5 Analysis

3.5.1 Surrogate Instantaneous Regret

Theorem 8.

Proof.