The use of multi-armed bandit (MAB) problems to represent sequential decision-making processes has received increasing attention in the literature, with contributions ranging from novel algorithms to strong theoretical results. The fundamental problem which MABs address, that of sequential finite resource allocation, has found application in diverse fields including control theory, advertising, and portfolio allocation. Our movivation arises from a novel dataset in the area of online video advertising, where the content provider seeks to optimize clickthrough rates from advertisements both within and around video content. Given features of the user such as their country and browser, we wish to display the advertisement with the highest probability of the user clicking through. Further, early investigations suggest that advertising effectiveness changes over time, as such we must account for time-varying clickthrough rates.
The classic MAB problem, and the origin of its moniker, is to imagine one is sitting in front of a potentially infinite set of slot machines, each with a different expected reward, and at each time one must decide which machine’s arm to pull. If one finds an arm that initially performs well, is it better to continue pulling the arm of that machine or exploring to find alternative, potentially higher payout, arms? MAB problems address this issue of balancing exploitation of a strategy you know to be best given your current knowledge vs. exploration to test alternative strategies.
Some simple, though naive, approaches for MAB problems are the full exploration strategy, which equally allocates actions across all of the arms regardless of past performance, and a Markovian strategy, which replays the current arm if success is achieved and otherwise picks an alternative arm at random (Robbins, 1956). Both of these strategies ignore the full history of past rewards, and hence ignore relevant information for finding an optimal strategy. An alternative is to be greedy – after an initial exploration period choose the arm at each time which has had the highest observed average reward. Problematically, new arms or those which due to randomness had small initial reward in the exploration period will not be subsequently sampled, leading to possible convergence to a suboptimal strategy. A variant on the greedy approach is an -greedy method, which starts with a greedy strategy but with probability chooses a different arm at each time.
More probabilistic approaches also exist, such as those which create confidence intervals for the expected reward from each arm and select arms based on features of this interval, such as choosing the arm with the highest upper bound (Auer et al., 2002)
. Another alternative is Thompson sampling, also termed probability matching(Scott, 2010)
, which selects an arm according to its probability of being optimal according to some underlying probability distribution. For binary (success/failure) rewards, the simple underlying model is to treat, the probability of success in arm
, as a Beta distribution, which through Bayes rule can be updated based on a sequence of observed successes and failures for each arm. Results using probability matching are highly promising(Graepel et al., 2010; Granmo, 2010; May and Leslie, 2011; Chapelle and Li, 2011), with corresponding theoretical results highlighting the strengths of this approach (Agrawal and Goyal, 2011; May et al., 2012).
In many situations, we are not presented with just the arms, but also some covariates . As an example, when deciding which ad to display to a user in online advertising, we know which browser and operating system is being used as well as an approximate location based on geolocated IP addresses. As such, under binary rewards might be a function of these covariates, . MABs with covariate information, also called contextual bandits, have been studied in, for example, Yang and Zhu (2002). Pavlidis et al. (2008) show that several commonly-studied empirical approaches, such as greedy, -greedy, and confidence bound methods can be fit into the contextual situation. Several problems remain with these empirical strategies, however, such as an inability to borrow strength by allowing for hierarchical structure on the covariates.
Some other issues which commonly arise in MAB problems are dynamic expected rewards, also called restless bandits, where the may change over time (Whittle, 1988), and arm-acquisition, where new arms can enter the system. Additionally, other issues can arise including penalties for switching arms, the ability to simultaneously pull multiple arms, or the presence of side observations (Mahajan and Teneketzis, 2008; Caron et al., 2012). Further, the MAB problem can be embedded within other methods and algorithms, such as to tune Monte Carlo algorithms (Wang and de Freitas, 2011). This paper presents a coherent, flexible approach for solving bandit problems in all of these circumstances. Through sequential Monte Carlo (Doucet et al., 2001), we are able to achieve this flexibility while maintaining efficient, scalable inference.
2 The Bayesian Bandit Framework
Assume we have observations which are the sequence of observed rewards, and is the strategy, or arm selected, at each time step. Let be a collection of parameters which control such features as the respective rewards for each arm or the impact of covariates; here is a function of of . Let be the likelihood function, or reward distribution, and be the prior distribution. As an example, in the case of binary rewards, if
where is the probability of success in arm , and for each we assign a prior,
then the resulting posterior after observing successes in trials of arm is . The fundamental idea behind Thompson sampling is to select a sample from each of these posterior distributions, and pull the arm with the highest-valued sample (Thompson, 1933, 1935).
2.1 A General, Hierarchical Bandit Structure
As noted by Scott (2010) and others, the Thompson sampling framework need not be constrained to the above Beta-Binomial conjugate setup. However, as more elaborate and
are chosen, the resulting posteriors will often require approximation methods such as Markov chain Monte Carlo or variational methods. Momentarily ignoring this hurdle, if our data are binary we can use a logit, probit, or other link function withinto connect the probability of success to the model parameters . The prior can be used to impose structure on the parameters , such as inducing dependence, hierarchical structure, or hard and soft constraints through some higher-level parameters .
This simple Bayesian framework provides a natural solution to address many of the bandit problem variants. For instance, contextual bandits with binary rewards can be modeled by assuming is binomial with probability , and learning the regression parameters
akin to Bayesian logistic regression. More generally, the wealth of knowledge about Bayesian online learning and generalized linear models can be brought in to create a highly flexible class of bandit models. Following the above example, we might suspect that the parametersare related with some overall mean , so the prior could be extended through a hierarchy, for example
and further hierarchies could be imposed on , or other variables (Gelman and Hill, 2006).
In general, probability matching selects an arm according its probability of having the highest expected rewards. In the binary reward case, arm is selected with probability
In simple cases, one can find these distributions analytically. However, beyond these few simple cases, approximations must be made to find these distributions. If we generate multiple samples from for each arm , we can compare the sample sets to determine the probability of selecting each arm. The origins of Thompson sampling, however, is to take this idea further, and to generate sample from each of . The largest sample is then selected as the strategy at time .
We are interested in learning about the underlying parameters as data is collected, specifically through the posterior . Consistent with earlier discussion, is the set of arm-specific parameters, which are related through some hierarchical parameters , while are parameters shared across arms. Figure 1
displays the model with and without the parameter space separated into components. Several bandit problem extensions can be fit into this model. For example, arms being added and removed are automatically included in the model; in practice, one need only expand the parameter vector to include the additional parameters. Further, costs for switching arms may be included by directly modeling a penalty parameter which reduces the expected reward of all arms other than the most recently chosen.
3 Efficient Inference
In restricted conjugate cases, the above model could be solved in closed form. However, more generally some approximation of is required for each . A simple, if computationally expensive, approach is to draw samples using MCMC or related methods for each . While such an approach is intuitively straightforward, in practice the computational cost of sampling from increases with . As an example, in the probit regression case Gibbs sampling is possible through sampling of a latent variable for each binary (Albert and Chib, 1993). As a more efficient alternative, we propose to use sequential Monte Carlo (SMC) methods to transition through the sequence of distributions in an efficient manner (Doucet et al., 2001). Intuitively, the approximation at time is leveraged to quickly and efficiently make an approximation at time .
SMC methods were originally designed for sequences of distributions of increasing dimension, with applications to state-space models (Doucet et al., 2000) and target tracking (Liu and Chen, 1998) among others, though they have recently been shown to provide flexibility and efficiency in static-dimensional problems as well (Chopin, 2002; Del Moral et al., 2006; Bornn et al., 2010). The goal of SMC methods is to sample from this sequence of distributions sequentially; because SMC borrows information from adjacent distributions, it will typically be computationally cheaper than MCMC even if we can sample from each distribution using MCMC.
For each time , SMC collects weighted samples (often called particles) , approximating . Expectations with respect to this posterior may be calculated with the weighted samples using . Through importance sampling, these (weighted) particles can then be reweighted to approximate the subsequent distribution. To ensure the sample does not become degenerate, the effective sample size (ESS),
is monitored and the particles are resampled when the ESS drops below some threshold ; often
. While particle filters are often criticized for problems of degeneracy, our interest is in each individual distribution rather than the entire joint distribution over the whole sample path. As such, resampling and related tools are available to straightforwardly mitigate this issue(Doucet et al., 2001).
SMC methods have been employed in sequential decision-making problems previously. Specifically, Yi et al. (2009) used SMC to emulate human performance on beta-binomial restless bandit problems. In another related area, Coquelin et al. (2008) explored a partially observable Markov decision problem (POMDP), where SMC was used alongside a policy gradient approach to optimize the decision at each point in time.
In the dynamic case discussed later, diversity is naturally incorporated into the samples through the prior dynamics. However, in the static case, we also move the particles with a Markov kernel of invariant distribution when we resample. In cases where the kernel is known to mix slowly, one may wish to move the particles multiple times at each resampling step. Of course, in these situations MCMC will similarly suffer due to the slow mixing of the kernel. Algorithm 1 shows the SMC bandits algorithm for static parameters:
SMC bandits has, at its core, a vanilla SMC algorithm. Based on the large literature on SMC methods, more advanced alternatives exist. For example, sampling from the prior may be inefficient, in which case we can design more effective proposals, as in Van der Merwe et al. (2001). An important side benefit from this algorithm is that we have, automatically, samples from the posterior distribution at every iteration. As a result, Thompson sampling amounts to randomly selecting one of the particles according to the weights
, and finding the arm with highest expected reward according to the value of that particle. Using all of the particles, one could also perform probability matching through a Monte Carlo estimate of.
3.1 Contextual Bandit Comparison
While sequential Monte Carlo bandits (hereafter SMC bandits) will naturally accommodate hierarchical model structure, dependence between parameters, and other structure, existing approaches for bandit problems do not naturally handle such issues. As such, we choose to compare SMC bandits in an arena where alternatives, such as -greedy and upper confidence bound (UCB) methods, are naturally suited, namely vanilla contextual bandits. Additionally, due to the additional parametric assumptions inherent in SMC bandits, we use diffuse prior information to simulate real-world modeling ignorance. The simulation data-generating mechanism is as follows:
where is the normal CDF and we have arms. We simulate data of length where , and take averages over 50 repeated simulations of the above system. For SMC bandits, we use the same prior for all elements of , namely . This diffuse prior covers the generated data, but has roughly
times the variance, representing apriori uncertainty about the model parameters.
We compare SMC bandits to contextual -Greedy and a UCB methods (Li et al., 2010; Filippi et al., 2010). For both we use the same probit link as in the data-generation mechanism. For -greedy we try both and , and for UCB we explore and confidence intervals.
Given the covariate set, we know the ground truth optimal strategy at each time step. As such, we compare the cumulative regret from each method, which is the cumulative difference between strategies of the methods vs. the optimal strategy. Figure 2 shows the cumulative regret, where we notice that SMC bandits have the lowest cumulative regret at each time point, even in this simple problem. In contrast, the -Greed and UCB methods’ performance is quite sensitive to the selection of or the confidence level, respectively. It is worth reiterating that the SMC bandits framework allows one to model hierarchies, constraints, and other structure naturally, while the other methods do not naturally extend to these cases.
Additionally, we also compare the use of SMC in the Bayesian hierarchical bandit framework to instead using MCMC repeated at each iteration. Figure 3 shows the time taken for each method, using samples for each. We see that SMC provides a significant reduction in computational time in comparison to repeating MCMC at each iteration. The intuition is that due to the efficient reweighting mechanism of the SMC algorithm, it only occasionally needs to move samples with a Markov kernel, and as such saves considerable computational cost. The improvements are similar to those from using SMC rather than MCMC for the tasks of prior sensitivity analyis and cross-validation (Bornn et al., 2010).
4 Contextual Dynamic Bandits
The case where the reward is changing over time is often referred to as restless, or dynamic, bandits. Intuitively, the difficulty arises in that if a given action is not taken for a period of time, it is possible that the unobserved action has had an increase in its expected reward; as a result, strategies for restless bandits will generally not converge to a single action, as on some regular basis other arms must be “checked” to monitor for changes in expected reward.
The framework proposed here provides a natural representation of restless bandits as a hidden Markov model (Figure4), where the hidden states which determine the expected reward vary in time (Gupta et al., 2011). We notate these time-varying parameters as .
Due to their probabilistic nature, SMC bandits will not only estimate the evolving parameters , but also their corresponding uncertainty. As such, when a given arm is not sampled for a period of time, its estimated uncertainty grows; more specifically, its posterior variance increases, and as a result over time that arm will be re-selected due to the Thompson sampling mechanism. Numerically, the prior dynamics increase the variance of the particles for the unsampled arm; as such, when the Thompson sampling mechanism selects a given particle, it becomes increasingly more likely that that arm has the largest value. We demonstrate this with a simulated example.
4.1 Dynamic Bandit Comparison
We study the ability of SMC bandits to track dynamic rewards, simulating data in the following manner
where we have arms. We impose the following model structure for the dynamic SMC bandit:
Figure 5 shows the tracking of the instantaneous probabilities , which control the dynamic expected reward at time . We notice that the arm with highest expected reward (as indicated by larger parameter values) is sampled more often, and posterior variance of the unsampled arm grows until it is next chosen. In contrast, the optimal arm is sampled more frequently, and hence the posterior mean better tracks the true reward. In Figure 6, we plot the cumulative regret for the dynamic SMC method, as well as several other algorithms. We see that the dynamic SMC method has lowest regret; the figure also shows that as the optimal arm changes around iteration , the regret of all the methods increases, with the dynamic SMC method adapting to the change and hence showing the least increase in regret.
Interestingly, and somewhat paradoxically, the dynamic SMC bandit method is generally faster than the static case due to the prior dynamics inducing diversity, rather than the static SMC bandit’s use of a Markov step. While one might be tempted to model static problems in the dynamic framework, this is equivalent to assuming your (static) parameters vary stochastically. As a result, arms which you establish with high certainty to be low-reward early on will come back into play later in the sequence due to the artificial dynamics induced on the parameters. As a result, the computational gains will come with the cost of incurring additional regret.
5 Empirical Results
An ideal performance evaluation of a bandit algorithm would include testing the algorithm online on a real system. Because online testing is expensive and typically unfeasable in most situations, we evaluate the empirical performance of the SMC bandit offline.
More precisely, we assume that our empirical data consists of a sequence of arm selections , as well as the reward and optional contexts . Crucially, the reward is only observed for the arm that was chosen uniformly at random, at time .
The proposed offline bandit evaluator is shown in Algorithm 3 (Li et al., 2011). The method requires bandit algorithm as well as empirical data , containing realizations of the bandit data . At each time , we keep a history , which contains a subset of the overall data presented to the bandit algorithm. At time , is empty. For each sample in , we ask the bandit algorithm for a recommended arm to play. If, given the current history , the algorithm selects the same arm as the arm described by the data, then the data sample is retained (added to the history) and the total reward earned by the bandit algorithm is updated. Otherwise, if selects a different arm, the data sample is ignored and the bandit algorithm proceeds to the next data sample without any change in its state.
We assume that the original data contains arms that are chosen uniformly at random, so each data sample is retained by the bandit algorithm with probability . Therefore the retained samples have the same distribution as the original data set and the evaluator in Algorithm 3
is functionally equivalent to (an unbiased estimator of) an online evaluation of(Li et al., 2011).
5.1 Online Advertising Recommendation
Using the offline evaluation technique described in Section 6.1, we evaluate the performance of our SMC bandit algorithm on real-world advertising data. The data is sourced from an online video advertising campaign in 2011, which marketed an online dating service for older singles.
In particular, the dataset contains 93,264 impressions, where an impression describes the event of displaying an advertisement to a particular user. There exists a set of four advertisements from which an advertisement can be shown to the user. Each impression consists of the advertisement shown to the user, whether the user clicked on the advertisement, and relevant contextual information, including geo-location (latitude and longitude of the user’s location in the United States) and datetime (time of day and day of week the advertisement was shown).
We formulate this data into a 4-armed bandit problem by modeling the advertisements as arms, the clicks as binary rewards, and adopting a probit link function to map the linear function of our contextual information to the binary reward distribution, a.k.a. the probit SMC bandit. We test the performance of two variations of the probit SMC bandit, the static and dynamic case, using the offline evaluation technique described in Section 6.1. Each SMC algorithm is evaluated 100 times (to reduce variance of our performance estimate) over all 665,321 impressions, initialized with particles and ESS threshold set to . Similarly, we test the performance of the contextual -greedy and UCB algorithms, as well as a random arm selector, as a baseline comparison. Results are shown in Table 1.
|Dynamic SMC||0.0088||+ 11.4||0.01|
We notice that, out of the various bandit algorithms tested, the dynamic SMC bandit performs the best, at about a improvement over the baseline random arm selector. Interestingly, while the performance of the dynamic SMC bandit is the best out of the group of algorithms, the static SMC bandit has middling performance. This is a strong indication that the reward distributions for each of the advertisements in the online advertising dataset move dynamically, which makes intuitive sense, because customer preferences for different advertisements likely vary over time. Nonstationary reward distributions prove to be a problem for the MCMC step in the static SMC bandit, which may be the cause of the decreased average reward. In fact, we also see that the lin--greedy algorithm with performs second best - another indication that the bandit arms may be moving dynamically. Recall that higher values of in -greedy algorithms force the algorithm to perform more random exploration. A bandit algorithm that performs random exploration at a high rate is particularly suited for dynamic rewards, as the algorithm can better track the random distributional movements of the arms it explores. This is likely why the greedy algorithm slightly outperforms the greedy algorithm.
A simple binning of the overall clickthrough rate of each advertisement in our dataset confirms our suspicion that the arm reward distributions are moving dynamically (Figure 7). Figure 7 shows the SMC bandit estimated probability of displaying each arm at every pageview, using the evaluation method described in Algorithm 3. Note the dynamic nature of the empirical clickthrough rates, as well as the similarity of the SMC estimated means to the empirical rates. This analysis provides conclusive evidence of the dynamic nature of the advertising data, as well as the efficacy and robustness of the SMC algorithm, especially in the dynamic case. While the other popular bandit algorithms exhibiti similar performance to random arm selection, the dynamic SMC bandit show significant improvement over the other arm allocation methods.
These results provide promising indications of the applicability of the SMC bandit in real-world situations. Especially in an application like online advertising, in which millions of advertisements are shown to users every day, a improvement in the average clicks per user can potentially lead to an extraordinary increase in revenue for advertisers.
In conclusion, we have proposed a flexible and scalable, yet simple, method for learning in multi-armed bandit problems. The proposed framework naturally handles contexts, dynamic rewards, the additional and removal of arms, and other features common to bandit problems. Through a hierarchical Bayesian model, the method is highly adaptable in its specification, with the user able to adjust hierarchical structure, hard and soft constraints, and other features through the prior specification of the parameter set. In addition, this additional structure does not come at a significant cost, as the sequential Monte Carlo algorithm proposed allows for scalability in time, but also as model complexity increases.
While using Bayesian inferential methods for modeling multi-armed bandit problems is not new, the hierarchical and dynamic structure proposed here provides for significantly increased flexibility. The framework allows for borrowing of the strength of the large literature on hierarchical Bayesian methods as well as the literature on sequential Monte Carlo to allow natural extendability.
- Agrawal and Goyal (2011) Agrawal, S. and Goyal, N. (2011). Analysis of Thompson sampling for the multi-armed bandit problem. arXiv preprint arXiv:1111.1797.
- Albert and Chib (1993) Albert, J. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association, 88(422):669–679.
- Auer et al. (2002) Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235–256.
- Bornn et al. (2010) Bornn, L., Doucet, A., and Gottardo, R. (2010). An efficient computational approach for prior sensitivity analysis and cross-validation. Canadian Journal of Statistics, 38(1):47–64.
Caron et al. (2012)
Caron, S., Kveton, B., Lelarge, M., and Bhagat, S. (2012).
Leveraging side observations in stochastic bandits.
Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence (UAI), pages 142–151.
- Chapelle and Li (2011) Chapelle, O. and Li, L. (2011). An empirical evaluation of Thompson sampling. In Neural Information Processing Systems (NIPS).
- Chopin (2002) Chopin, N. (2002). A sequential particle filter method for static models. Biometrika, 89(3):539–552.
- Coquelin et al. (2008) Coquelin, P.-A., Deguest, R., Munos, R., et al. (2008). Particle filter-based policy gradient for pomdps. Advances in Neural Information Processing Systems.
- Del Moral et al. (2006) Del Moral, P., Doucet, A., and Jasra, A. (2006). Sequential Monte Carlo samplers. Journal of the Royal Statistical Society: Series B, 68(3):411–436.
- Doucet et al. (2001) Doucet, A., de Freitas, N., and Gordon, N. (2001). Sequential Monte Carlo Methods in Practice. Springer, New York.
- Doucet et al. (2000) Doucet, A., Godsill, S., and Andrieu, C. (2000). On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and computing, 10(3):197–208.
- Filippi et al. (2010) Filippi, S., Cappé, O., Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case. Advances in Neural Information Processing Systems, 23:1–9.
- Gelman and Hill (2006) Gelman, A. and Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models.
- Graepel et al. (2010) Graepel, T., Candela, J. Q., Borchert, T., and Herbrich, R. (2010). Web-scale Bayesian click-through rate prediction for sponsored search advertising in Microsoft’s Bing search engine. In Proc. 27th Internat. Conf. on Machine Learning. Morgan Kaufmann, San Francisco, CA.
- Granmo (2010) Granmo, O.-C. (2010). Solving two-armed Bernoulli bandit problems using a Bayesian learning automaton. International Journal of Intelligent Computing and Cybernetics, 3(2):207–234.
- Gupta et al. (2011) Gupta, N., Granmo, O.-C., and Agrawala, A. (2011). Thompson sampling for dynamic multi-armed bandits. In Machine Learning and Applications and Workshops (ICMLA), 2011 10th International Conference on, volume 1, pages 484–489. IEEE.
- Li et al. (2010) Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661–670. ACM.
- Li et al. (2011) Li, L., Chu, W., Langford, J., and Wang, X. (2011). Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In 4th ACM Intl. Conf. on Web Search and Data Mining (WSDM).
- Liu and Chen (1998) Liu, J. and Chen, R. (1998). Sequential Monte Carlo methods for dynamic systems. Journal of the American Statistical Association, 93(443):1032–1044.
- Mahajan and Teneketzis (2008) Mahajan, A. and Teneketzis, D. (2008). Multi-armed bandit problems. Foundations and Applications of Sensor Management, pages 121–151.
- May et al. (2012) May, B. C., Korda, N., Lee, A., and Leslie, D. S. (2012). Optimistic Bayesian sampling in contextual-bandit problems. The Journal of Machine Learning Research, 98888:2069–2106.
- May and Leslie (2011) May, B. C. and Leslie, D. S. (2011). Simulation studies in optimistic Bayesian sampling in contextual-bandit problems. Technical report, Technical Report 11: 02, Statistics Group, Department of Mathematics, University of Bristol.
- Pavlidis et al. (2008) Pavlidis, N. G., Tasoulis, D. K., and Hand, D. J. (2008). Simulation studies of multi-armed bandits with covariates. In Computer Modeling and Simulation, 2008. UKSIM 2008. Tenth International Conference on, pages 493–498. IEEE.
- Robbins (1956) Robbins, H. (1956). A sequential decision problem with a finite memory. Proceedings of the National Academy of Sciences of the United States of America, 42(12):920.
- Scott (2010) Scott, S. L. (2010). A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26(6):639–658.
- Thompson (1933) Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, pages 285–294.
- Thompson (1935) Thompson, W. R. (1935). On the theory of apportionment. American Journal of Mathematics, 57(2):450–456.
- Van der Merwe et al. (2001) Van der Merwe, R., Doucet, A., De Freitas, N., and Wan, E. (2001). The unscented particle filter. Advances in Neural Information Processing Systems, pages 584–590.
- Wang and de Freitas (2011) Wang, Z. and de Freitas, N. (2011). Predictive adaptation of hybrid monte carlo with bayesian parametric bandits. NIPS.
- Whittle (1988) Whittle, P. (1988). Restless bandits: Activity allocation in a changing world. Journal of applied probability, pages 287–298.
- Yang and Zhu (2002) Yang, Y. and Zhu, D. (2002). Randomized allocation with nonparametric estimation for a multi-armed bandit problem with covariates. The Annals of Statistics, 30(1):100–121.
- Yi et al. (2009) Yi, M. S., Steyvers, M., and Lee, M. (2009). Modeling human performance in restless bandits with particle filters. The Journal of Problem Solving, 2(2):5.