# Make the Minority Great Again: First-Order Regret Bound for Contextual Bandits

Regret bounds in online learning compare the player's performance to L^*, the optimal performance in hindsight with a fixed strategy. Typically such bounds scale with the square root of the time horizon T. The more refined concept of first-order regret bound replaces this with a scaling √(L^*), which may be much smaller than √(T). It is well known that minor variants of standard algorithms satisfy first-order regret bounds in the full information and multi-armed bandit settings. In a COLT 2017 open problem, Agarwal, Krishnamurthy, Langford, Luo, and Schapire raised the issue that existing techniques do not seem sufficient to obtain first-order regret bounds for the contextual bandit problem. In the present paper, we resolve this open problem by presenting a new strategy based on augmenting the policy space.

• 39 publications
• 48 publications
• 63 publications
01/27/2017

### The Price of Differential Privacy For Online Learning

We design differentially private algorithms for the problem of online li...
03/03/2020

### MOTS: Minimax Optimal Thompson Sampling

Thompson sampling is one of the most widely used algorithms for many onl...
09/15/2012

### Further Optimal Regret Bounds for Thompson Sampling

Thompson Sampling is one of the oldest heuristics for multi-armed bandit...
02/02/2019

### First-Order Regret Analysis of Thompson Sampling

We address online combinatorial optimization when the player has a prior...
05/26/2017

### Online Auctions and Multi-scale Online Learning

We consider revenue maximization in online auctions and pricing. A selle...
02/11/2022

### A PDE-Based Analysis of the Symmetric Two-Armed Bernoulli Bandit

This work addresses a version of the two-armed Bernoulli bandit problem ...
06/11/2020

### Bandits with Partially Observable Offline Data

We study linear contextual bandits with access to a large, partially obs...

## 1 Introduction

The contextual bandit problem is an influential extension of the classical multi-armed bandit. It can be described as follows. Let be the number of actions, a set of experts (or “policies”), the time horizon, and denote . At each time step ,

• The player receives from each expert an “advice” .

• Using advices and previous feedbacks, the player selects a probability distribution

.

• The adversary selects a loss function

.

• The player plays an action at random from (and independently of the past).

• The player’s suffered loss is , which is also the only feedback the player receives about the loss function .

The player’s performance at the end of the rounds is measured through the regret with respect to the best expert:

 RTdef=maxe∈E{E[T∑t=1ℓt(at)−⟨ξet,ℓt⟩]}=maxe∈E{E[T∑t=1⟨pt−ξet,ℓt⟩]}. (1.1)

A landmark result by Auer et al. (2002) is that a regret of order is achievable in this setting. The general intuition captured by regret bounds is that the player’s performance is equal to the best expert’s performance up to a term of lower order. However the aforementioned bound might fail to capture this intuition if . It is thus natural to ask whether one could obtain a stronger guarantee where is essentially replaced by . This question was posed as a COLT 2017 open problem Agarwal et al. (2017). Such bounds are called first-order regret bounds, and they are known to be possible with full information Auer et al. (2002), as well as in the multi-armed bandit setting Allenberg et al. (2006) (see also Foster et al. (2016) for a different proof) and the semi-bandit framework Neu (2015); Lykouris et al. (2017). Our main contribution is a new algorithm for contextual bandit, which we call MYGA (see Section 2), and for which we prove the following first-order regret bound, thus resolving the open problem.

###### Theorem 1.1.

For any loss sequence such that one has that MYGA with and satisfies

 RT≤O(√Klog(|E|T)L∗+Klog(|E|T)).

## 2 Algorithm Description

In this section we describe the MYGA algorithm.

### 2.1 Truncation

We introduce a truncation operator that takes as input an index and a threshold . Then, treating the first arms as “majority arms” and the last arms as “minority arms,” redistributes “multiplicatively” the probability mass of all minority arms below threshold to the majority arms.

###### Definition 2.1.

For and , the truncation operator is defined as follows. Given any , then we set

 Tksq(i)=⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩0,i>k and q(i)≤s;q(i),i>k and q(i)>s;q(i)⋅(1+∑j:j>k∧q(j)≤sq(j)∑j≤kq(j)),i≤k.

Equivalently one can define for the majority arms with the following implicit formula:

 Tksq(i)=q(i)∑j≤kq(j)∑j≤kTksq(j). (2.1)

To see this it suffices to note that the amount of mass in the majority arms is given by

 ∑j≤kTksq(j)=1−∑j>kTksq(j)=1−∑j:j>k∧q(j)>sq(j)=∑j≤kq(j)+∑j:j>k∧q(j)≤sq(j).
###### Example 2.2.

If , then simply adds into if .

###### Example 2.3.

An example with and is as follows:

 q =( 0.2 0.1 0.2 0.1 0.1 0.1 0.05 0.05 0.04 0.03 0.03) T30.02q =( 0.2 0.1 0.2 0.1 0.1 0.1 0.05 0.05 0.04 0.03 0.03) T30.03q =( 0.224 0.112 0.224 0.1 0.1 0.1 0.05 0.05 0.04 0 0) T30.04q =( 0.24 0.12 0.24 0.1 0.1 0.1 0.05 0.05 0 0 0) T30.05q =( 0.28 0.14 0.28 0.1 0.1 0.1 0 0 0 0 0) T30.1q =( 0.4 0.2 0.4 0 0 0 0 0 0 0 0) T30.2q =( 0.4 0.2 0.4 0 0 0 0 0 0 0 0) T30.5q =( 0.4 0.2 0.4 0 0 0 0 0 0 0 0)

### 2.2 Informal description

MYGA is parameterized by two parameters: a classical learning rate , and a thresholding parameter . Also let

At a high level, a key feature of MYGA is to introduce a set of auxiliary experts, one for each . More precisely, in each round , after receiving expert advices , MYGA calculates a distribution for each . Then, MYGA uses the standard exponential weight updates on with learning rate , to calculate a weight function —see (2.3). Then, it computes

• , the weighted average of expert advices in : .

• , the weighted average of expert advices in : .

Using these information, MYGA calculates the probability distribution from which the arm is played at round .

Let us now explain how and , are defined. First we remark that in the contextual bandit setting, the arm index has no real meaning since in each round we can permute the arms by some

and permute the expert’s advices and the loss vector by the same

. For this reason, throughout this paper, we shall assume

 ζt(1)≥ζt(2)≥⋯ζt(K).

Let us define the “pivot” index . Then, in order to perform truncation, MYGA views the first arms as “majority arms” and the last arms as “minority arms” of the current round . At a high level we will have:

• the distribution to play from is .

• Each auxiliary expert is defined by .

We now give a more precise description in Algorithm 1.

## 3 Preliminaries

###### Definition 3.1.

For analysis purpose, let us define the truncated loss , so that

 Eat[⟨˜ℓt,pt⟩]=⟨¯ℓt,pt⟩=⟨ℓt,pt⟩

We next derive two lemmas that will prove useful to isolate the properties of the truncation operator that are needed to obtain a first-order regret bound.

###### Lemma 3.2.

Let and assume that for all , for some universal constant , and that . Then one has

 (1−cKγ)LT−L∗T≤log(|E′|)η+η2ET∑t=1∥¯ℓt∥22 . (3.1)
###### Proof.

Using , , and , we have

 (1−cKγ)LT−L∗T≤maxe∈E′ET∑t=1⟨(1−cKγ)pt−ξet,¯ℓt⟩≤maxe∈E′ET∑t=1⟨qt−ξet,¯ℓt⟩.

The rest of the proof follows from standard argument to bound the regret of Exp4, see e.g., [Theorem 4.2, Bubeck and Cesa-Bianchi (2012)] (with the minor modification that the assumption on implies that ). ∎

The next lemma is straightforward.

###### Lemma 3.3.

In addition to the assumptions in Lemma 3.2, assume that there exists some numerical constants such that

 γ ET∑t=1∥¯ℓt∥22≤2 c′ (η+γ) K LT+2 c′′ log(|E′|)η . (3.2)

Then one has

We now see that it suffices to show that MYGA satisfies the assumptions of Lemma 3.2 and Lemma 3.3 for , and (assume that is known), in which case one obtains a bound of order .

In fact the assumption of Lemma 3.2 will be easily verified, and the real difficulty will be to prove (3.2). We observe that the standard trick of thresholding the arms with probability below would yield (3.2) with the right hand side replaced by , and in turn this leads to a regret of order . Our goal is to improve over this naive argument.

## 4 Proof of the 2-Armed Case

The goal of this section is to explain how our MYGA algorithm arises naturally. To focus on the main ideas we restrict to the case . The complete formal proof of Theorem 1.1 is given in Section 5.

Recall we have assumed without loss of generality that for each round . This implies because . In this simple case, for , we abbreviate our truncation operator as , and it acts as follows. Given

 if q(2)≤s we have Tsq=(1,0);  and if q(2)>s we have Tsq=q.

In particular, we have and for all . We refer to arm as the majority arm and arm as the minority arm. We denote as the loss of the majority arm and as the loss of the minority arm.

Since and , we have

 E∑Tt=1∥¯ℓt∥22≤E∑Tt=1¯ℓt(1)+¯ℓt(2)=M+m . (4.1)

Observe also that one always has (indeed ), and thus the whole game to prove (3.2) is to upper bound the minority’s loss .

### 4.1 When the minority suffers small loss

Assume that for some constant . Then, because , one can directly obtain (3.2) from (4.1) with . In words, when the minority arm has a total loss comparable to the majority arm, simply playing from would satisfy a first-order regret bound.

Our main idea is to somehow enforce this relation between the minority and majority losses, by “truncating” probabilities appropriately. Indeed, recall that if after some truncation we have , then it satisfies so the minority loss can be improved.

### 4.2 Make the minority great again

Our key new insight is captured by the following lemma which is proved using an integral averaging argument.

###### Definition 4.1.

For each , let be the expected loss if the truncated strategy is played at each round.

###### Lemma 4.2.

As long as ,

 ∃s∈(γ,1/2]:m−M≤LT−LsTγ .

In words, if is large, then it must be that was a much better threshold compared to , that is is large.

###### Proof of Lemma 4.2.

For any , define the function

 f(s)def=E∑Tt=11{qt(2)≤s}(¯ℓt(1)−¯ℓt(2)).

Let us pick to minimize , and breaking ties by choosing the smaller value of . We make several observations:

• because for any with we must have .

• .

• because .

Let us define the points

 s0def=γ  and  {s1<…

Note that the tie-breaking rule for the choice of ensures (if then it must satisfy giving a contradiction).

Using the identity

 T∑t=1⟨Tsqt,¯ℓt⟩=T∑t=1⟨qt,¯ℓt⟩+1{qt(2)≤s}qt(2)(¯ℓt(1)−¯ℓt(2)), (4.2)

we calculate that

 LT−LsT =ET∑t=1⟨Tγqt−Tsqt,ℓt⟩=ET∑t=1⟨Tγqt−Tsqt,¯ℓt⟩ =ET∑t=1(1{qt(2)≤γ}−1{qt(2)≤s})qt(2)(¯ℓt(1)−¯ℓt(2)) =ET∑t=1m∑i=1−si1{qt(2)=si}(¯ℓt(1)−¯ℓt(2)) =m∑i=1si(f(si−1)−f(si))=m−1∑i=1(si+1−si)f(si)+s1f(s0)−smf(sm) .

Since , and , we conclude that

 LT−LsT≥(sm−s1)f(sm)−smf(sm)=−s1f(sm)≥γ(m−M) .\qed

Given Lemma 4.2, a very intuitive strategy start to emerge. Suppose we can somehow get an upper bound of the form

 LT−LsT≤O(log(|E′|)η+η(m+M)+γLT) . (4.3)

Then, putting this into Lemma 4.2 and using , we have for any ,

 γm≤O(log(|E′|)η+γLT) .

In words, the minority arm also suffers from a small loss (and thus is great again!) Putting this into (4.1), we immediately get (3.2) as desired and finish the proof of Theorem 1.1 in the case .

Thus, we are left with showing (4.3). The main idea is to add the truncated strategy as an additional auxiliary expert. If we can achieve this, then (4.3) can be obtained from the regret formula in Lemma 3.2.

### 4.3 Expanding the set of experts

Assume for a moment that we somehow expand the set of experts into

so that:

 ∀s∈(γ,1/2],∃e∈E′ such that for all t∈[T],ξet=Tsqt . (4.4)

Then clearly (4.3) would be satisfied using Lemma 3.2, (4.1) and (the loss of an expert should be no better than the loss of the best expert ).

There are two issues with condition (4.4): first, it self-referential, in the sense that it assumes satisfies a certain form depending on while is defined via (recall (2.2)); and second, it potentially requires to have an infinite number of experts (one for each ).

Let us first deal with the second issue via discretization.

###### Lemma 4.3.

In the same setting as Lemma 4.2, there exists such that

 m−M≤1+LT−LsTγ .
###### Proof.

For let be the smallest element in . For any we can rewrite (4.2) as (note that )

 ⟨Tsqt,¯ℓt⟩=⟨qt,¯ℓt⟩+1{qt(2)–––––≤s}qt(2)–––––(¯ℓt(1)−¯ℓt(2))+εt,s ,

where . Using the same proof of Lemma 4.2, and redefining

 f(s)def=E∑Tt=11{qt(2)–––––≤s}(¯ℓt(1)−¯ℓt(2)).

we get that there exists and such that

 LT−LsT=ε+m∑i=1si(f(si−1)−f(si)) .

The rest of the proof now follows from the same proof of Lemma 4.2, except that we minimize over instead of . ∎

Thus, instead of (4.4), we only need to require

 ∀s∈S,∃e∈E′ such that for all t∈[T],ξet=Tsqt . (4.5)

We now resolve the self-referentiality of (4.5) by defining simultaneously and as follows. Consider the map defined by:

 Ft(x)=1∑e∈Ewt(e)+∑s∈Swt(s)(∑e∈Ewt(e)ξet(2)+∑s∈Swt(s)x1{x>s}) .

It suffices to find a fixed point : indeed, setting

 qtdef=(1−x,x)  and  ξst(2)def=x1{x>s}=Tsqt for s∈S,

we have both (4.5) holds and is the correct weighted average of expert advices in

Finally, has a fixed point since it is a nondecreasing function from a closed interval to itself. It is also not hard to find such a point algorithmically.

This concludes the (slightly informal) proof for . We give the complete proof for arbitrary in the next section.

## 5 Proof of Theorem 1.1

In this section, we assume satisfies (2.2) and we defer the constructive proof of finding to Section 6. Recall the arm index has no real meaning so without loss of generality we have permuted the arms so that

 ζt(1)≥ζt(2)≤…≥ζt(K)for each t=1,2,…,T.

We refer to the set of majority arms and the set of minority arms at round .111We stress that in the -arm setting, although is the minimum index such that , it may not be the minimum index so that . We let and respectively be the total loss of the majority and minority arms. We again have

 E∑Tt=1∥¯ℓt∥22≤E∑Tt=1∑i∈[K]¯ℓt(i)=M+m . (5.1)

Thus, the whole game to prove (3.2) is to upper bound and .

### 5.1 Useful properties

We state a few properties about and its truncations.

###### Lemma 5.1.

In each round , if satisfies (2.2), then

 for every s∈S and i≤kt:\quadξst(i)=ζt(i)∑j≤kζt(j)⋅(1−∑j>kξst(j))
###### Proof.

Let and . By (2.1) and since one has

 ξst(i)=qt(i)∑j≤kqt(j)∑j≤kξst(j).

Moreover is a mixture of and truncated versions of so similarly using (2.1) one has

 qt(i)=ζt(i)∑j≤kζt(j)∑j≤kqt(j).

Putting the two above displays together concludes the proof. ∎

###### Lemma 5.2.

In each round , if satisfies (2.2), then

• for every minority arm it satisfies , and

• for every majority arm it satisfies .

###### Proof.

For sake of notation we drop the index in this proof. Recall .

• For every minority arm , every , we have according to Definition 2.1. Therefore, we must have .

• For every majority arm , we have (using Lemma 5.1)

 ξe(i)=ζ(i)∑j≤kζ(j)⋅(1−∑j>kξs(j))≥ζ(i)∑j≤kζ(j)⋅(1−∑j>kζ(j))=ζ(i)

From the definition of , we can also conclude . This is because . ∎

The next lemma shows that setting satisfies the assumption of Lemma 3.2.

###### Lemma 5.3.

If satisfies (2.2), and , then for every arm :

 (1−2Kγ)pt(i)≤qt(i)andpt(i)≠0⇒pt(i)≥qt(i).
###### Proof.

For sake of notation we drop the index in this proof.

By Definition 2.1 and Lemma 5.2, we have for every :

 p(i)≤q(i)(1+∑j:j>k∧q(j)≤γq(j)∑j≤kq(j))≤q(i)(1+∑j:q(j)≤γq(j)∑j≤kζ(j))≤q(i)(1+2Kγ).

The other statement follows because whenever , Definition 2.1 says it must satisfy . ∎

### 5.2 Bounding m and M

We first upper bound and then upper bound .

###### Lemma 5.4.

If satisfies (2.2), then .

###### Proof.

Using Lemma 5.2 we have for any . Also, for every satisfying (owing to Definition 3.1 and Lemma 5.3). Therefore,

 M=T∑t=1E∑i≤kt¯ℓt(i) ≤2KT∑t