I Introduction
MultiArmed Bandits (MAB) is a sequential decisionmaking framework addressing the explorationexploitation dilemma [1, 2, 3, 4]. A player explores a finite set of arms sequentially, and pulling each of them results in a reward to the player. The problem has been studied for different rewards models. In the stochastic MAB, the rewards of each arm are assumed to be i.i.d. samples of an unknown, fixed distribution. Then, the player’s goal is to exploit the arm with largest expected reward as many times as possible to maximize the gain. In the literature, this objective has been translated to minimizing the cumulative regret, a comparison measure between the actual performance of the player versus a clairvoyant knowing the best arm in advance. Early studies on MAB dates back to a few decades ago, but the problem has surged a lot of renewed interest due to its modern applications. Though generally addressing sequential decision making problems, MAB has been studied in several engineering contexts such as web search and advertising, wireless cognitive radios, multichannel communication systems, and BigData streaming (see e.g. [5, 6, 7, 8, 9, 10, 11] and references therein).
Departing from its classical setting (explorationexploitation), many researchers have studied MAB in the pureexploration framework. In this case, the player aims to minimize the simple
regret which can be related to finding the best arm with a high probability
[12]. As a result, the bestarm identification problem has received a considerable attention in the literature of machine learning
[13, 14, 15, 16, 17, 12, 18, 19]. The problem has been viewed from two main standpoints: (i) the fixedconfidence setting, where the objective is to minimize the number of trials to find the best arm with a certain confidence, and (ii) the fixedbudget setting, where the player attempts to maximize the probability of correct identification given a fixed number of arm pulls. Bestarm identification has several applications including channel allocation as originally proposed in [12]. Consider the problem of channel allocation for mobile phone communication. Before the outset of communication, a cellphone (player) can explore the set of channels (arms) to find the best one to operate. Each channel feedback is noisy, and the number of trials (budget) is limited. The problem is hence an instance of bestarm identification, and minimizing the cumulative regret is not the right approach to the problem.While both pureexploration and explorationexploitation setups are concerned with finding the best arm, they are quite different problems in nature. In fact, Bubeck et al. [17] establish that methods designed to minimize the cumulative regret (explorationexploitation) can perform poorly for the simpleregret minimization (pureexploration). More specifically, they proved that upper bounds on the cumulative regret yield lower bounds on the simple regret, i.e., the smaller the cumulative regret, the larger the simple regret. Therefore, one must adopt different strategies for optimal bestarm recommendation.
Ia Our Contribution
In this paper, we address the bestarm identification problem in the fixedbudget setting. We restrict our attention to a class of algorithms that work based on sequential elimination. Recently, it is proved in [20] that some existing strategies based on the sequential elimination of the arms are optimal. However, the notion of optimality is defined with respect to the worstcase allocation of the reward distributions. The main focus of this paper is not the worstcase scenario. On the contrary, given certain regimes for rewards, our goal is to propose an algorithm outperforming the stateoftheart in these regimes. We characterize these reward allocations and prove the superiority of our algorithm both theoretically and empirically.
Of particular relevance to the current study is the works of [12, 18], where two iterative algorithms are proposed for sequential elimination: Successive Rejects (SuccRej) [12] and Sequential Halving (SeqHalv) [18]. In both algorithms, the player must sample the arms in rounds to discard the arms sequentially, until a single arm is left, one that is perceived as the best arm. However, we recognize two distinctions between the algorithms: (i) SuccRej eliminates one arm at each round, until it is left with a single arm, whereas SeqHalv discards roughly half of the remaining arms at each round to identify the best arm. (ii) At each round, SeqHalv samples the remaining arms uniformly (excluding previous rounds), whereas SuccRej samples them uniformly once including the previous rounds.
Inspired by these works, our first contribution is to propose a general framework to bring sequential elimination algorithms (including SuccRej and SeqHalv) under the same umbrella. Our analysis reveals a novel performance bound which relies on the sampling design as well as the number of eliminated arms at each around. Following this general framework, we extend SuccRej to an algorithm that divides the budget by a nonlinear function of remaining arms at each round, unlike SuccRej that does so in a linear fashion. We prove theoretically that we can gain advantage from the nonlinearity. In particular, we consider several wellstudied reward regimes and exhibit the suitable nonlinearity for each environment. Benefiting from the nonlinearity, our algorithm outperforms SuccRej and SeqHalv in these regimes. Interestingly, our numerical experiments support our theoretical results, while showing that our algorithm is competitive with UCBE [12] which requires prior knowledge of a problemdependent parameter.
Finally, we consider sequential elimination in the presence of side observations. In this model, a graph encodes the connection between the arms, and pulling one arm reveals the rewards of all neighboring arms [21]. While the impact of side observations is wellknown for explorationexploitation [22, 23], we consider the model in the pureexploration setting. Given a partition of arms to a few blocks, we propose an algorithm that eliminates blocks consecutively and selects the best arm from the final block. Naturally, we provide an improved theoretical guarantee comparing to the full bandit setting where there is no side observation.
IB Related Work
Pureexploration in the PAClearning setup was studied in [13], where Successive Elimination for finding an optimal arm with probability (fixedconfidence setting) was proposed. Seminal works of [14, 15] provide matching lower bounds for the problem, which present a sufficient number of arm pulls to reach the confidence . Many algorithms for pureexploration are inspired by the classical UCB1 for explorationexploitation [2]. For instance, Audibert et al. [12] propose UCBE, which modifies UCB1 for pureexploration. UCBE needs prior knowledge of a problemdependent parameter, so the authors also propose its adaptive counterpart AUCBE to address the issue. In addition, Jamieson et al. [24] propose an optimal algorithm for the fixed confidence setting, inspired by the law of the iterated logarithm. We refer the reader to [25] and the references therein for recent advances in the fixedconfidence setting, while remarking that Gabillon et al. [19] present a unifying approach for fixedbudget and fixedconfidence settings. As another interesting direction, various works in the literature introduce informationtheoretic measures for bestarm identification. Kaufmann et al. [26]
study the identification of multiple top arms using KLdivergencebased confidence intervals. The authors of
[27] investigate both settings to show that the complexity of the fixedbudget setting may be smaller than that of the fixedconfidence setting. Recently, Russo [28] develops three Bayesian algorithms to examine asymptotic complexity measure for the fixedconfidence setting. There also exists extensive literature on identification of multiple top arms in MAB (see e.g. [29, 30, 26, 31, 32, 27]). Finally, we remark that simpleregret minimization has been successfully used in the context of MonteCarlo Tree Search [33, 34] as well.IC Organization
The rest of the paper is organized as follows. Section II is dedicated to nomenclature, problem formulation, and a summary of our results. In Sections III and IV, we discuss our main theoretical results and their consequences, while we extend these results to sideobservation model in Section V. In Section VI, we describe our numerical experiments, and the concluding remarks are provided in Section VII. We include the proofs in the Appendix (Section VIII).
Ii Preliminaries
Notation: For integer , we define to represent the set of positive integers smaller than or equal to . We use to denote the cardinality of the set
. Throughout, the random variables are denoted in bold letters.
SuccRej  SeqHalv  Our algorithm  

Iia Problem Statement
Consider the stochastic Multiarmed Bandit (MAB) problem, where a player explores a finite set of arms. When the player samples an arm, only the corresponding payoff or reward of that arm is observed. The reward sequence for each arm corresponds to i.i.d samples of an unknown distribution whose expected value is . We assume that the distribution is supported on the unit interval , and the rewards are generated independently across the arms. Without loss of generality, we further assume that the expected value of the rewards are ordered as
(1) 
and therefore, arm is the unique best arm. We let denote the gap between arm and arm 1, measuring the suboptimality of arm . We also represent by the average reward obtained from pulling arm for times.
In this paper, we address the bestarm identification setup, a pureexploration problem in which the player aims to find the arm with the largest expected value with a high confidence. There are two wellknown scenarios for which the problem has been studied: fixed confidence and fixed budget. In the fixedconfidence setting, the objective is to minimize the number of trials needed to achieve a fixed confidence. However, in this work, we restrict our attention to the fixedbudget, formally described as follows:
Problem 1.
Given a total budget of arm pulls, minimize the probability of misidentifying the best arm.
It is wellknown that classical MAB techniques in the explorationexploitation setting, such as UCB1, are not efficient for the identification of the best arm. In fact, Bubeck et al. have proved in [17]
that upper bounds on the cumulative regret yield lower bounds on the simple regret, i.e., the smaller the cumulative regret, the larger the simple regret. In particular, for all Bernoulli distributions on the rewards, a constant
, and a function , they have proved if an algorithm satisfiesafter rounds, we have
Misidentification Probability  
Expected Simple Regret 
for two positive constants and . Given that for optimal algorithms in the explorationexploitation setting, we have , these algorithms decay polynomially fast for bestarm identification. However, a carefully designed bestarm identification algorithm achieves an exponentially fast decay rate (see e.g. [12, 18]).
The underlying intuition is that in the explorationexploitation setting, only playing the best arm matters. For instance, playing the second best arm for a long time can result in a dramatically large cumulative regret. Therefore, the player needs to minimize the exploration time to focus only on the best arm. On the contrary, in the bestarm identification setting, player must recommend the best arm at the end of the game. Hence, exploring the suboptimal arms “strategically” during the game helps the player to make a better recommendation. In other words, the performance is measured by the final recommendation regardless of the time spent on the suboptimal arms. We focus on sequential elimination algorithms for the bestarm identification in the next section.
IiB Previous Performance Guarantees and Our Result
In this work, we examine sequentialelimination type algorithms in the fixed budget setting. We propose a general algorithm that unifies sequential elimination methods, including celebrated SuccRej [12] and SeqHalv [18]. We then use a special case of this general framework to develop an algorithm called Nonlinear Sequential Elimination. We show that this algorithm is more efficient than SuccRej and SeqHalv in several problem scenarios.
Any sequential elimination algorithm samples the arms based on some strategy. It then discards a few arms at each round and stops when it is only left by one arm. In order to integrate sequential elimination algorithms proceeding in rounds, we use the following key observation: let any such algorithm play each (remaining) arm for times (in total) by the end of round . If the algorithm needs to discard arms at round , it must satisfy the following budget constraint
(2) 
since the arms eliminated at round have been played times, and the surviving arm has been played times. Alternatively, letting denote the number of remaining arms at the start of round , one can pose the budget constraint as
(3) 
Our first contribution is to derive a generic performance measure for such algorithm (Theorem 2), relating the algorithm efficiency to as well as the sampling scheme determining . Note that SuccRej satisfies for rounds, whereas SeqHalv is characterized via with for . While it is shown in [14] that in many settings for MAB, the quantity in the following plays a key role,
(4) 
the performance of both SuccRej and SeqHalv is described via the complexity measure , which is equal to up to logarithmic factors in [12]. In particular, for each algorithm the bound on the probability of misidentification can be written in the form of , where and are given in Table I in which .
In SuccRej, at round , the remaining arms are played proportional to the whole budget divided by which is a linear function of . Motivated by the fact that this linear function is not necessarily the best sampling rule, as our second contribution, we specialize our general framework to an algorithm which can be cast as a nonlinear extension of SuccRej. This algorithm is called Nonlinear Sequential Elimination (NSeqEl), where the term “nonlinear” refers to the fact that at round , the algorithm divides the budget by the nonlinear function for a positive real (an input value). We prove (Proposition 3) that the performance of our algorithm depends on the following quantities
(5) 
as described in Table I. We do not use as a subscript in the definition of to avoid confusion over the fact that due to definition (4). Indeed, NSeqEl with recovers SuccRej, but we show that in many regimes for arm gaps, provides better theoretical results (Corollary 4). We also illustrate this improvement in the numerical experiments in Section VI, where we observe that can outperform SuccRej and SeqHalv in many settings considered in [12, 18]. Since the value of is received as an input, we remark that our algorithm needs tuning to perform well; however, the tuning is more qualitative rather than quantitative, i.e., the algorithm maintains a reasonable performance as long as is in a certain interval, and therefore, the value of needs not be specific. We will discuss this in the next section in more details.
Iii Nonlinear Sequential Elimination
In this section, we propose our generic sequential elimination method and analyze its performance in the fixed budget setting. Consider a sequential elimination algorithm given budget of arm pulls. The algorithm maintains an active set initialized by the arms, and it proceeds for rounds to discard the arms sequentially until it is left by a single arm. Let us use to denote the ceiling function. Then, for a constant and a decreasing, positive sequence , set at round , let the algorithm sample the remaining arms for times, and calculate the empirical average of rewards for each arm. If the algorithm dismisses arms with lowest average rewards, and we impose the constraint on the sequence , the algorithm outputs a single arm, the one that it hopes to be the best arm. This general point of view is summarized in Fig. 1 on the left side.
The choice of in the algorithm (see Fig. 1) must warrant that the budget constraint (3) holds. When we substitute into (3), we get
where in the last line we used the condition . The following theorem provides an upper bound on the error probability of the algorithm.
Theorem 2.
Consider the General Sequential Elimination algorithm outlined in Fig. 1. For any , let denote the number of arms that the algorithm eliminates at round . Let also be the number of remaining arms at the start of round . Given a fixed budget of arm pulls and the input sequence , setting and as described in the algorithm, the misidentification probability satisfies the bound,
It is already quite wellknown that the suboptimality of each arm plays a key role in the identification quality; however, an important subsequent of Theorem 2 is that the performance of any sequential elimination algorithm also relies on the choice of which governs the constant . In [12], SuccRej employs , i.e., at each round the remaining arms are played equally often in total. This results in being of order .
We now use the abstract form of the generic algorithm to specialize it to Nonlinear Sequential Elimination delineated in Fig. 1. The algorithm works with and for a given , and it is called “nonlinear” since is not necessarily equal to one. The choice of reduces the algorithm to SuccRej. In Section IV, we prove that in many regimes for arm gaps, provides better theoretical results, and we further exhibit the efficiency in the numerical experiments in Section VI. The following proposition encapsulates the theoretical guarantee of the algorithm.
Proposition 3.
The performance of the algorithm does depend on the parameter , but the choice is more qualitative rather than quantitative. For instance, if the suboptimal arms are almost the same, i.e. for , noting the definition of and in (5), we observe that performs better than . In general, larger values for increase and decrease . Therefore, there is a tradeoff in choosing . We elaborate on this issue in Sections IV and VI, where we observe that a wide range of values for can be used for tuning, and the tradeoff can be addressed using either or .
We remark that in the proof of Theorem 2, the constant behind the exponent can be improved if one can avoid the union bounds. To do so, one needs to assume some structure in the sequence . For instance, the authors in [18] leverage the fact that to avoid union bounds in SeqHalv. The idea can be extended to when the ratio of is a constant independent of .
Finally, while the result of Proposition 3 provides a general performance bound with respect to reward regimes, in the next section, we provide two important corollaries of the proposition to study several regimes for rewards, where we can simplify the bound and compare our result with other algorithms.
Iv Performance in Several Suboptimality Regimes
Inspired by numerical experiments carried out in the previous works [12, 18], we consider a few instances for suboptimality of arms in this section, and we demonstrate how NSeqEl fares in these cases. We would like to distinguish three general regimes that encompass interesting settings for suboptimality and determine the values of for which we expect NSeqEl to achieve faster identification rates:

Arithmetic progression of gaps: In this case, for , where is a constant.

A large group of competitive, suboptimal arms: We recognize this case as when there exists a constant^{1}^{1}1The choice of must be constant with respect to and . such that for arms , where grows linearly as a function of , and for , .

A small group of competitive, suboptimal arms: This case occurs when there exists a constant such that for , where is of constant order with respect to , and for , .
We now state the following corollary of Proposition 3, which proves to be useful for our numerical evaluations. Note that the orders are expressed with respect to .
Corollary 4.
Consider the Nonlinear Sequential Elimination algorithm described in Fig. 1. Let constants and be chosen such that and . Then, for the three settings given above, the bound on the misidentification probability presented in Proposition 3 satisfies
Regime 1  Regime 2  Regime 3 

We can now compare our algorithm with SuccRej and SeqHalv using the result of Corollary 4. Returning to Table I and calculating for Regimes to , we can observe the following table,
SuccRej  SeqHalv  Our algorithm  

Regime 1  
Regime 2  
Regime 3 
which shows that with a right choice of for NSeqEl, we can save a factor in the exponential rate comparing to other methods. Though we do not have a prior information on gaps to categorize them into one of the Regimes to (and then choose ), the choice of is more qualitative rather than quantitative. Roughly speaking: if the suboptimal arms are almost the same performs better than , and if there are a few real competitive arms, outperforms . Therefore, the result of Corollary 4 is of practical interest, and we will show using numerical experiments (Section VI) that a wide range of values for can potentially result in efficient algorithms.
One should observe that the number of competitive arms is a key factor in tuning . We now provide another corollary of Proposition 3, which presents the suitable range of parameter given the growth rate of competitive arms as follows.
Corollary 5.
Let the number of competitive arms be an arbitrary function of total arms, i.e., there exists a constant such that for , where , and for , . Then, there exists a suitable choice of for which NSeqEl outperforms SuccRej and SeqHalv in the sense that the misidentification probability decays with a faster rate, i.e., we have . For different conditions on the growth of , the suitable choice of parameter is presented in Table III.
Condition on  Suitable Range of Parameter 

The corollary above indicates that a side information in the form of the number of competitive arms (in the order) can help us tune the input parameter
. The corollary exhibits a smooth interpolation between when the competitive arms are small versus when they are large. Perhaps, the most interesting regime is the middle row, where the choice of
is given as a function of the number of arms . Consider the following example where we calculate the choice of for , where .As we can see in the table above, since the condition on depends in logarithmic orders on and , we have flexibility to tune even for very large . However, we only use for the numerical experiments in Section VI, since the timecomplexity of Monte Carlo simulations is prohibitive on large number of arms.
V Side Observations
In the previous sections, we considered a scenario in which pulling an arm yields only the reward of the chosen arm. However, there exist applications where pulling an arm can additionally result in some side observations. For a motivative example, consider the problem of web advertising, where an ad placer offers an ad to a user and receives a reward only if the user clicks on the ad. In this example, if the user clicks on a vacation ad, the ad placer receives the side information that the user could have also clicked on ads for rental cars. The value of side observations in the stochastic MAB was studied in [22] for explorationexploitation setting. The sideobservation model is described via an undirected graph that captures the relationship between the arms. Once an arm is pulled, the player observes the reward of the arm as well as its neighboring arms. In explorationexploitation settings, the analysis of MAB with side observations relies on the cliques of the graph [21, 22]. In this section, we would like to consider the impact in the pureexploration setting.
In the pureexploration, we minimize the simple regret rather than the cumulative regret, and therefore, the player’s best bets are the most connected arms resulting in more observations. Now consider a partition of the set into blocks such that each contains a star graph, i.e., there exists an arm in connected to all other arms. Given such partition, the player can follow a simple rule to leverage side observations and eliminate blocks one by one. The idea is to sample the central arm in each block, which reveals the rewards for all the arms in that block. At round , sample the remaining blocks (the central arms of each block) for times, and find the arm with largest average reward in each block. With the best arms of all blocks at hand, remove the block whose best arm is the worst comparing to other blocks. Continue the sampling until only one block is left, and output the arm with largest average reward in the final block as the best arm. The algorithm, called Sequential Block Elimination, is summarized in Fig. 2, and its performance is characterized in the following theorem.
Theorem 6.
Let the Sequential Block Elimination algorithm in Fig. 2 run given the input sequence and blocks . Define to be the maximum cardinality among the blocks. Then, the misidentification probability satisfies the bound
In particular, the choice of for a given , yields
where .
The theorem indicates that once we can partition the arms into blocks, the complexity measure need not be maximized over arms. Instead, it is maximized over the top arms. The improvement must be more visible in the settings that arms are roughly as competitive as each other, since we can slow down the linear growth of by .
As a final note, recovers the fully bandit (no side observation) setting with , and we observe that in such case, we recover the result of Proposition 3 from Theorem 6 in the rate of exponential decay. However, the constant behind the exponential would change from in Proposition 3 to in Theorem 6. This is the artifact of an extra term contributing to the upper bound, which can be removed for the case .
Vi Numerical Experiments
In this section, we empirically evaluate our algorithm on the settings perviously studied in [12, 18]. In these experiments, we compare NSeqEl with SuccRej, SeqHalv, and UCBE. Though we include UCBE proposed in [12] as a benchmark, we remark that the algorithm requires a prior knowledge of a parameter that depends on defined in (4). The adaptive version of UCBE was also developed in [12]. The algorithm (called AUCBE) does not need prior knowledge of , and it calculates the parameter online. The experiments in [18] suggest that for , SuccRej, SeqHalv, and UCBE outperform AUCBE, and it is not surprising that the prior knowledge of must give UCBE an advantage over AUCBE.
We consider Bernoulli distribution on the rewards, assuming that the expected value of Bernoulli distribution for the best arm is . In what follows, we use the notation to denote integers in . We examine the following setups for two values of arm numbers :

One group of suboptimal arms: .

Two groups of suboptimal arms: , , and .

Three groups of suboptimal arms: , , , and .

Arithmetic progression: for .

Geometric progression: for .

One real competitive arm: and .
We run experiments for each setup with specific , and the misidentification probability is averaged out over the experiment runs. The budget considered in each setup is equal to in the corresponding setup following [12, 18]. Fig. 3 illustrates the performance of each algorithm in different setups. The height of each bar depicts the misidentification probability, and the index guideline is as follows: (i) indices 14: NSeqEl with parameter . (ii) index 5: SuccRej. (iii) index 6: SeqHalv. (iv) indices 79: UCBE with parameter , for . The legends are the same for all the plots, and we remove some of them to avoid clutter.
The results are consistent with Corollary 4, and the following comments are in order:

Setup 1 perfectly relates to Regime 2 in Corollary 4 (). While any choice of gives an rate, the rate deteriorates to by choosing . Therefore, only the the choice of is appropriate for NSeqEl. This choice joined with SuccRej (which amounts to our algorithm with ) outperform others even UCBE.

Setup 4 defines Regime 1 in Corollary 4. Accordingly, choice of outperforms SuccRej and SeqHalv, and prove to be competitive to UCBE.

In Setups 23, the number of competitive arms grows with , but they can be considered close to Regime 3 in Corollary 4 as the growth is sublinear. Also, Setup 6 is the ideal match for Regime 3 in Corollary 4. Therefore, any choice of should be suitable in these cases. We observe that for , NSeqEl outperforms SuccRej and SeqHalv, while being quite competitive to UCBE in Setups 23 for .

In Setup 5, we were expecting to observe good performance for . While we do see that for , NSeqEl outperforms SuccRej and SeqHalv with , the performance is not quite satisfactory for (only performs well). A potential reason is that we want to keep the rewards bounded in , thereby choosing the geometric rate for so slow that it is dominated by rate in . To support our argument, we present a complementary evaluation of this case for a small , where we can use faster geometric growth.

As a final remark, note that in consistent with Table II, for relevant cases we observe an improvement of performance when increases. For instance, in Setup 1 with , the ratio of misidentification probability of NSeqEl to SeqHalv increases from to , or in Setup 4 with , the ratio of misidentification probability of NSeqEl to SuccRej increases from to .
Via Complementary Numerical Experiment for Geometric Suboptimality Setup
As we remarked in the very last comment of numerical experiments, choosing small number of arms, we can space the expected value of arm rewards such that in (5) the suboptimality term dominates for . We propose a simple experiment for arms, where and in Fig. 4. As expected, NSeqEl tuned with is competitive, and it achieves its best performance with , winning the competition against others, while almost equalizing with UCBE for .
Vii Conclusion
We considered bestarm identification in the stochastic multiarmed bandits, where a player is given a certain budget to explore a finite number of arms. The player’s objective is to detect the arm with largest expected reward. We contribute to the literature of bestarm identification by (i) unifying sequential elimination algorithms under a general framework, which introduces a novel performance metric for this class of algorithms, (ii) developing a nonlinear sequential elimination algorithm with provable theoretical and practical guarantees, (iii) and establishing a theoretical result on the value of side observations in the pureexploration setting.
Having established that we gain advantage from nonlinear budget allocation, an important future direction is to propose a method that starts with a specific nonlinear rate and finetunes the rate according to the problem environment. The main challenge is that the quantity should perhaps be timevarying, and its value needs to be cautiously controlled by the algorithm, so that the algorithm does not overspend the budget.
Acknowledgments
We gratefully acknowledge the support of DARPA under grant numbers N6600115C4028 and W911NF1410508. We thank Prof. Masahiro Yukawa (Keio University) for his helpful comments. We also thank Dr. Ahmad Beirami and Dr. Hamed Farhadi for many helpful discussions.
Viii Appendix
Fact 1.
(Hoeffding’s inequality) Let be independent random variables with support on the unit interval with probability one. If , then for all , it holds that
Proof of Theorem 2
Recall that denotes the set of arms not eliminated by the start of round with being its cardinality. Also, represents the set of arms that we decide to discard after playing round with denoting its cardinality. It evidently holds that for . Therefore,
(6) 
since the sets of removed arms at each round are disjoint, i.e. for . We can then write
(7) 
Now for any particular , consider the worst arms, i.e., the bottom arms when arms are ordered in terms of (true) expected value. If the best arm (arm ) is set to be eliminated at the end of round , its empirical average must be less than at least one of these arms. In the case that , the bottom arms would be . Therefore, recalling that denotes the average reward of pulling arm for times, and using Hoeffding’s inequality (Fact 1), we get
(8) 
where the last step is due to the fact that . In any other case for the best of the worst arms cannot be better than arm . As a result, combining (7) and (8), we obtain
Then, in view of (6) we derive
Noting the fact that , we can use above to conclude that
which completes the proof. ∎
Proof of Proposition 3
We point out that the algorithm is a special case of General Sequential Elimination where , , , and . The proof then follows immediately from the result of Theorem 2. ∎
Proof of Corollary 4
The proof follows by substituting each case in (5). We need to understand the order of and for different regimes of . Let us start by
and noting that for any , is a convergent sum when . Therefore, for the regime , the sum is a constant, i.e., . On the other hand, consider , and note that the sum is divergent, and for large we have . Now, let us analyze
For Regime 1, since , for , we have
which is of constant order with respect to . Therefore, the product .
For Regime 2, we have
and the maximum order can be achieved as the number of arms close to the second best arm grows linearly in . Combining with , the product .
For Regime 3, if , we have
since the cardinality of is of constant order with respect to . On the other hand, since , we have
Therefore, is of constant order, and combining with , the product . ∎
Proof of Corollary 5
According to calculations in the previous proof, we have that for , and for . In order to find out the order of and , we should note that
and we simply have . Now returning to Table I, we must compare and for all three the rows given in Table III.
Row 1: In the case that , for any choice of , since , we always have
Row 3: In the case that , for any choice of , since , we have
Row 2: In this case, we have that . To prove the claim in the Table III, we have to break down the analysis into two cases:
Row 2 – Case 1: First, consider the case . Since , we have that
Row 2 – Case 2: Second, consider the case . Since , we have that
Therefore, for all of the conditions on , we showed proper choice of which guarantees . ∎
Proof of Theorem 6
Recall that the elements of are the set of blocks not eliminated by the start of round , and only contains the block we decide to discard after playing round . Also, recall that arm is the best arm, located in the block , and denotes the maximum cardinality of the blocks.
If the algorithm does not output the best arm, the arm is either eliminated with block in one the rounds in , or not selected at the final round . Therefore,
Comments
There are no comments yet.