noregretgames
Code associated with the paper: NoRegret Learning in Unknown Games with Correlated Payoffs, P.G. Sessa, I. Bogunovic, M. Kamgarpour, A. Krause. NeurIPS 2019.
view repo
We consider the problem of learning to play a repeated multiagent game with an unknown reward function. Single player online learning algorithms attain strong regret bounds when provided with full information feedback, which unfortunately is unavailable in many realworld scenarios. Bandit feedback alone, i.e., observing outcomes only for the selected action, yields substantially worse performance. In this paper, we consider a natural model where, besides a noisy measurement of the obtained reward, the player can also observe the opponents' actions. This feedback model, together with a regularity assumption on the reward function, allows us to exploit the correlations among different game outcomes by means of Gaussian processes (GPs). We propose a novel confidencebound based bandit algorithm GPMW, which utilizes the GP model for the reward function and runs a multiplicative weight (MW) method. We obtain novel kerneldependent regret bounds that are comparable to the known bounds in the full information setting, while substantially improving upon the existing bandit results. We experimentally demonstrate the effectiveness of GPMW in random matrix games, as well as realworld problems of traffic routing and movie recommendation. In our experiments, GPMW consistently outperforms several baselines, while its performance is often comparable to methods that have access to full information feedback.
READ FULL TEXT VIEW PDFCode associated with the paper: NoRegret Learning in Unknown Games with Correlated Payoffs, P.G. Sessa, I. Bogunovic, M. Kamgarpour, A. Krause. NeurIPS 2019.
Many realworld problems, such as traffic routing leblanc1975 , market prediction fainmesser2012 , and social network dynamics skyrms2009 , involve multiple learning agents that interact and compete with each other. Such problems can be described as repeated games, in which the goal of every agent is to maximize her cumulative reward. In most cases, the underlying game is unknown to the agents, and the only way to learn about it is by repeatedly playing and observing the corresponding game outcomes.
The performance of an agent in a repeated game is often measured in terms of regret. For example, in traffic routing, the regret of an agent quantifies the reduction in travel time had the agent known the routes chosen by the other agents. Noregret algorithms for playing unknown repeated games exist, and their performance depends on the information available at every round. In the case of full information feedback, the agent observes the obtained reward, as well as the rewards of other nonplayed actions. While these algorithms attain strong regret guarantees, such full information feedback is often unrealistic in realworld applications. In traffic routing, for instance, agents only observe the incurred travel times and cannot observe the travel times for the routes not chosen.
In this paper, we address this challenge by considering a more realistic feedback model, where at every round of the game, the agent plays an action and observes the noisy reward outcome. In addition to this bandit feedback, the agent also observes the actions played by other agents. Under this feedback model and further regularity assumptions on the reward function, we present a novel noregret algorithm for playing unknown repeated games. The proposed algorithm alleviates the need for full information feedback while still achieving comparable regret guarantees.
Hedge freund1997  Exp3 auer2003  GPMW [this paper]  
Feedback  rewards for all actions  obtained reward  obtained reward + opponents’ actions 
Regret 
)) captures the degrees of freedom in the reward function.
Related Work. In the full information setting, multiplicativeweights (MW) algorithms littlestone1994 such as Hedge freund1997 attain optimal regret, where is the number of actions available to agent . In the case of convex action sets in , and convex and Lipschitz rewards, online convex optimization algorithms attain optimal regret zinkevich2003 . By only assuming Lipschitz rewards and bounded action sets, regret follows from maillard2010 , while in hazan2017 the authors provide efficient gradientbased algorithms with ‘local’ regret guarantees. Full information feedback requires perfect knowledge of the game and is unrealistic in many applications. Our proposed algorithm overcomes this limitation while achieving comparable regret bounds.
In the more challenging bandit setting, existing algorithms have a substantially worse dependence on the size of the action set. For finite actions, Exp3 auer2003 and its variants ensure optimal regret. In the case of convex action sets, and convex and Lipschitz rewards, bandit algorithms attain regret bubeck2017 , while in the case of Lipschitz rewards regret can be obtained slivkins2014 . In contrast, our algorithm works in the noisy bandit setting and requires the knowledge of the actions played by other agents. This allows us to, under some regularity assumptions, obtain substantially improved performance. In Table 1, we summarize the regret and feedback model of our algorithm together with the existing noregret algorithms.
The previously mentioned online algorithms reduce the unknown repeated game to a single agent problem against an adversarial and adaptive environment that selects a different reward function at every time step cesabianchi2006 . A fact not exploited by these algorithms is that in a repeated game, the rewards obtained at different time steps are correlated through a static unknown reward function. In syrgkanis2015 the authors use this fact to show that, if every agent uses a regularized noregret algorithm, their individual regret grows at a lower rate of , while the sum of their rewards grows only as . In contrast to syrgkanis2015 , we focus on the singleplayer viewpoint, and we do not make any assumption on opponents strategies^{1}^{1}1In fact, they are allowed to be adaptive and adversarial.. Instead, we show that by observing opponents’ actions, the agent can exploit the structure of the reward function to reduce her individual regret.
Contributions. We propose a novel noregret bandit algorithm GPMW for playing unknown repeated games. GPMW combines the ideas of the multiplicative weights update method littlestone1994 , with GP upper confidence bounds, a powerful tool used in GP bandit algorithms (e.g., srinivas2010 ; bogunovic2016truncated ). When a finite number of actions is available to player
, we provide a novel highprobability regret bound
, that depends on a kerneldependent quantity srinivas2010 . For common kernel choices, this results in a sublinear regret bound, which grows only logarithmically in . In the case of infinite action subsets of and Lipschitz rewards, via a discretization argument, we obtain a highprobability regret bound of . We experimentally demonstrate that GPMW outperforms existing bandit baselines in random matrix games and traffic routing problems. Moreover, we present an application of GPMW to a novel robust Bayesian optimization setting in which our algorithm performs favourably in comparison to other baselines.We consider a repeated static game among noncooperative agents, or players. Each player has an action set and a reward function . We assume that the reward function is unknown to player . At every time , players simultaneously choose actions and player obtains a reward , which depends on the played action and the actions of all the other players. The goal of player is to maximize the cumulative reward . After time steps, the regret of player is defined as
(1) 
i.e., the maximum gain the player could have achieved by playing the single best fixed action in case the sequence of opponents’ actions and the reward function were known in hindsight. An algorithm is noregret for player if as for any sequence .
First, we consider the case of a finite number of available actions , i.e., . To achieve noregret, the player should play mixed strategies cesabianchi2006
, i.e., probability distributions
over . With fullinformation feedback, at every time playerobserves the vector of rewards
. With bandit feedback, only the reward is observed by the player. Existing full information and bandit algorithms freund1997 ; auer2003 , reduce the repeated game to a sequential decision making problem between player and an adaptive environment that, at each time , selects a reward function . In a repeated game, the reward that player observes at time is a static fixed function of , i.e., , and in many practical settings similar game outcomes lead to similar rewards (see, e.g., the traffic routing application in Section 4.2). In contrast to existing approaches, we exploit such correlations by considering the feedback and reward function models described below.Feedback model. We consider a noisy bandit feedback model where, at every time , player observes a noisy measurement of the reward where is subGaussian, i.e., for all , with independence over time. The presence of noise is typical in realworld applications, since perfect measurements are unrealistic, e.g., measured travel times in traffic routing.
Besides the standard noisy bandit feedback, we assume player also observes the played actions of all the other players. In some applications, the reward function depends only indirectly on through some aggregative function . For example, in traffic routing leblanc1975 , represents the congestion in the network’s edges, while in network games jackson2015 , it represents the strategies of player ’s neighbours. In such cases, it is sufficient for the player to observe instead of .
Regularity assumption on rewards. In this work, we assume the unknown reward function has a bounded norm in a reproducing kernel Hilbert space (RKHS) associated with a positive semidefinite kernel function , that satisfies for all . The RKHS norm measures the smoothness of with respect to the kernel function , while the kernel encodes the similarity between two different outcomes of the game . Typical kernel choices are polynomial, Squared Exponential, and Matérn:
where , is the modified Bessel function, and
are kernel hyperparameters
(rasmussen2005, , Section 4). This is a standard smoothness assumption used in kernelized bandits and Bayesian optimization (e.g., srinivas2010 ; chowdhury2017 ). In our context it allows player to use the observed history of play to learn about and predict unseen game outcomes. Our results are not restricted to any specific kernel function, and depending on the application at hand, various kernels can be used to model different types of reward functions. Moreover, composite kernels (see e.g., krause2011 ) can be used to encode the differences in the structural dependence of on and .It is well known that Gaussian Process models can be used to learn functions with bounded RKHS norm srinivas2010 ; chowdhury2017 . A GP is a probability distribution over functions , specified by its mean and covariance functions and , respectively. Given a history of measurements at points with and , the posterior distribution under a
prior is also Gaussian, with mean and variance functions:
(2)  
(3) 
where , , and is the kernel matrix.
At time , an upper confidence bound on can be obtained as:
(4) 
where is a parameter that controls the width of the confidence bound and ensures , for all and , with high probability srinivas2010 . We make this statement precise in Theorem 3.
Due to the above regularity assumptions and feedback model, player can use the history of play to compute an upper confidence bound of the unknown reward function by using (4). In the next section, we present our algorithm that makes use of to simulate full information feedback.
We now introduce GPMW, a novel noregret bandit algorithm, which can be used by a generic player (see Algorithm 1). GPMW maintains a probability distribution (or mixed strategy) over and updates it at every time step using a multiplicativeweight (MW) subroutine (see (6)) that requires full information feedback. Since such feedback is not available, GPMW builds (in (5)) an optimisticestimate of the true reward of every action via the upper confidence bound of . Moreover, since rewards are bounded in , the algorithm makes use of . At every time step , GPMW plays an action sampled from , and uses the noisy reward observation and actions played by other players to compute the updated upper confidence bound .
Input: Set of actions , GP prior , parameters
(5) 
(6) 
In Theorem 3, we present a highprobability regret bound for GPMW while all the proofs of this section can be found in the supplementary material. The obtained bound depends on the maximum information gain, a kerneldependent quantity defined as:
It quantifies the maximal reduction in uncertainty about after observing outcomes and the corresponding noisy rewards. The result of srinivas2010 shows that this quantity is sublinear in , e.g., in the case of , and in the case of , where is the total dimension of the outcomes , i.e., .
Fix and assume ’s are subGaussian with independence over time. For any such that , if player plays actions from , , according to GPMW with and , then with probability at least ,
The proof of this theorem follows by the decomposition of the regret of GPMW into the sum of two terms. The first term corresponds to the regret that player incurs with respect to the sequence of computed upper confidence bounds. The second term is due to not knowing the true reward function . The proof of Theorem 3 then proceeds by bounding the first term using standard results from adversarial online learning cesabianchi2006 , while the second term is upper bounded by using regret bounding techniques from GP optimization srinivas2010 ; bogunovic2018 .
Theorem 3 can be made more explicit by substituting bounds on . For instance, in the case of the squared exponential kernel, the regret bound becomes . In comparison to the standard multiarmed bandit regret bound (e.g., auer2003 ), this regret bound does not depend on , similarly to the ideal full information setting.
In this section, we consider the case when is a (continuous) compact subset of . In this case, further assumptions are required on and to achieve sublinear regret. Hence, we assume a bounded set and to be Lipschitz continuous in . Under the same assumptions, existing regret bounds are and in the full information maillard2010 and bandit setting slivkins2014 , respectively. By using a discretization argument, we obtain a high probability regret bound for GPMW.
Let and be subGaussian with independence over time. Assume , , and is Lipschitz in its first argument, and consider the discretization with such that for every , where is the closest point to in . If player plays actions from according to GPMW with and , then with probability at least ,
By substituting bounds on , our bound becomes in the case of the SE kernel (for fixed ). Such a bound has a strictly better dependence on than the existing bandit bound from slivkins2014 . Similarly to slivkins2014 ; maillard2010 , the algorithm resulting from Corollary 3 is not efficient in high dimensional settings, as its computational complexity is exponential in .
In this section, we consider random matrix games and a traffic routing model and compare GPMW with the existing algorithms for playing repeated games. Then, we show an application of GPMW to robust BO and compare it with existing baselines on a movie recommendation problem.
We consider a repeated matrix game between two players with actions and payoff matrices . At every time step, each player receives a payoff , where indicates the th entry of matrix . We select and generate random matrices with , where with . We set the noise to , and use . For every game, we distinguish between two settings:
Against random opponent. In this setting, player2 plays actions uniformly at random from at every round , while player1 plays according to a noregret algorithm. In fig:random_opponent, we compare the timeaveraged regret of player1 when playing according to Hedge freund1997 , Exp3.P auer2003 , and GPMW. Our algorithm is run with the true function prior while Hedge receives (unrealistic) noiseless full information feedback (at every round ) and leads to the lowest regret. When only the noisy bandit feedback is available, GPMW significantly outperforms Exp3.P.
GPMW vs Exp3.P. Here, player1 plays according to GPMW while player2 is an adaptive adversary and plays using Exp3.P. In Figure 0(b), we compare the regret of the two players averaged over the game instances. GPMW outperforms Exp3.P and ensures player1 a smaller regret.
one standard deviation.
We consider the SiouxFalls road network leblanc1975 ; website_transp_test , a standard benchmark model in the transportation literature. The network is a directed graph with 24 nodes and 76 edges (). In this experiment, we have agents and every agent seeks to send some number of units from a given origin to a given destination node. To do so, agent can choose among possible routes consisting of network edges . A route chosen by agent corresponds to action with in case belongs to the route and otherwise. The goal of each agent is to minimize the travel time weighted by the number of units . The travel time of an agent is unknown and depends on the total occupancy of the traversed edges within the chosen route. Hence, the travel time increases when more agents use the same edges. The number of units for every agent, as well as travel time functions for each edge, are taken from leblanc1975 ; website_transp_test . A more detailed description of our experimental setup is provided in Appendix C.
We consider a repeated game, where agents choose routes using either of the following algorithms:
[leftmargin=1em]
Hedge. To run Hedge, each agent has to observe the travel time incurred had she chosen any different route. This requires knowing the exact travel time functions. Although these assumptions are unrealistic, we use Hedge as an idealized benchmark.
Exp3.P. In the case of Exp3.P, agents only need to observe their incurred travel time. This corresponds to the standard bandit feedback.
GPMW. Let be the total occupancy (by other agents) of edges at time . To run GPMW, agent needs to observe a noisy measurement of the travel time as well as the corresponding .
QBRI (Qlearning Better Replies with Inertia algorithm monderer1996 ). This algorithm requires the same feedback as GPMW and is proven to asymptotically converge to a Nash equilibrium (as the considered game is a potential game chapman2013 ). We use the same set of algorithm parameters as in chapman2013 .
For every agent to run GPMW, we use a composite kernel such that for every , , where is a linear kernel and is a polynomial kernel of degree .


First, we consider a random subset of agents that we refer to as learning agents. These agents choose actions (routes) according to the aforementioned noregret algorithms for game rounds. The remaining nonlearning agents simply choose the shortest route, ignoring the presence of the other agents. In Figure 2 (top plots), we compare the average regret (expressed in hours) of the learning agents when they use the different noregret algorithms. We also show the associated average congestion in the network (see (13) in Appendix C for a formal definition). When playing according to GPMW, agents incur significantly smaller regret and the overall congestion is reduced in comparison to Exp3.P and QBRI.
In our second experiment, we consider the same setup as before, but we vary the number of learning agents. In Figure 2 (bottom plots), we show the final (when ) average regret and congestion as a function of the number of learning agents. We observe that GPMW systematically leads to a smaller regret and reduced congestion in comparison to Exp3.P and QBRI. Moreover, as the number of learning agents increases, both Hedge and GPMW reduce the congestion in the network, while this is not the case with Exp3.P or QBRI (due to a slower convergence).
In this section, we apply GPMW to a novel robust Bayesian Optimization (BO) setting, similar to the one considered in bogunovic2018 . The goal is to optimize an unknown function (under the same regularity assumptions as in Section 2) from a sequence of queries and corresponding noisy observations. Very often, the actual queried points may differ from the selected ones due to various input perturbations, or the function may depend on external parameters that cannot be controlled (see bogunovic2018 for examples).
This scenario can be modelled via a two player repeated game, where a player is competing against an adversary. The unknown reward function is given by . At every round of the game, the player selects a point , and the adversary chooses . The player then observes the parameter and a noisy estimate of the reward: . After time steps, the player incurs the regret
Note that both the regret definition and feedback model are the same as in Section 2.
In the standard (nonadversarial) Bayesian optimization setting, the GPUCB algorithm srinivas2010 ensures noregret. On the other hand, the StableOpt algorithm bogunovic2018 attains strong regret guarantees against the worstcase adversary which perturbs the final reported point . Here instead, we consider the case where the adversary is adaptive at every time , i.e., it can adapt to past selected points . We note that both GPUCB and StableOpt fail to achieve noregret in this setting, as both algorithms are deterministic conditioned on the history of play. On the other hand, GPMW is a noregret algorithm in this setting according to Theorem 3 (and Corollary 3).
Next, we demonstrate these observations experimentally in a movie recommendation problem.
Movie recommendation. We seek to recommend movies to users according to their preferences. A priori it is unknown which user will see the recommendation at any time . We assume that such a user is chosen arbitrarily (possibly adversarially), simultaneously to our recommendation.
We use the MovieLens100K dataset movielens which provides a matrix of ratings for movies rated by users. We apply nonnegative matrix factorization with latent factors on the incomplete rating matrix and obtain feature vectors for movies and users, respectively. Hence, represents the rating of movie by user . At every round , the player selects , the adversary chooses (without observing ) a user index , and the player receives reward . We model via a GP with composite kernel where is a linear kernel and is a diagonal kernel.
We compare the performance of GPMW against the ones of GPUCB and StableOpt when sequentially recommending movies. In this experiment, we let GPUCB select , while StableOpt chooses at every round . Both algorithms update their posteriors with measurements at with in the case of GPUCB and for StableOpt. Here, represents a lower confidence bound on (see bogunovic2018 for details).
In Figure 2(a), we show the average regret of the algorithms when the adversary chooses users uniformly at random at every . In our second experiment (Figure 2(b)), we show their performance when the adversary is adaptive and selects according to the Hedge algorithm. We observe that in both experiments GPMW is noregret, while the average regrets of both GPUCB and StableOpt do not vanish.
We have proposed GPMW, a noregret bandit algorithm for playing unknown repeated games. In addition to the standard bandit feedback, the algorithm requires observing the actions of other players after every round of the game. By exploiting the correlation among different game outcomes, it computes upper confidence bounds on the rewards and uses them to simulate unavailable full information feedback. Our algorithm attains high probability regret bounds that can substantially improve upon the existing bandit regret bounds. In our experiments, we have demonstrated the effectiveness of GPMW on synthetic games, and realworld problems of traffic routing and movie recommendation.
This work was gratefully supported by Swiss National Science Foundation, under the grant SNSF _, and by the European Union’s Horizon 2020 ERC grant .
Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing
, STOC 2017, pages 72–85, 2017.International Conference on Machine Learning (ICML)
, 2017.Handbook of Game Theory with Economic Applications
, volume 4, chapter 3, pages 95–163. Elsevier, 2015.We make use of the following wellknown confidence lemma. [Confidence Lemma] Let be a RKHS with underlying kernel function . Consider an unknown function in such that , and the sampling model where is subGaussian (with independence between times). By setting
the following holds with probability at least :
where and are given in (2)(3). Lemma A follows directly from [2, Theorem 3.11 and Remark 3.13] as well as the definition of the maximum information gain .
We can now prove Theorem 3. Recall the definition of regret
Defining , can be rewritten as
By Lemma A and since rewards are in , with probability the true unknown reward function can be upper and lower bounded as:
(7) 
The statement of the theorem then follows by standard probability arguments:
To show (8), define the function . Note that if (7) holds, since , hence . Using such definition, the left hand side of (8) can be upper bounded as:
(9) 
Observe that the right hand side of (9) is precisely the regret which player incurrs in an adversarial online learning problem with reward functions . The actions , moreover, are exactly chosen by the Hedge [11] algorithm which receives the full information feedback . Note that the original version of Hedge works with losses instead of rewards, but the same happens in GPMW since the mixed strategies are updated with . Therefore, by [7, Corollary 4.2], with probability ,
Note that according to [7, Remark 4.3], the functions can be chosen by an adaptive adversary depending on past actions , but not on the current action . This applies to our setting, since depends only on and not on . ∎
A function is Lipschitz continuous with constant (or Lipschitz) if
Define . The fact that is Lispschitz in its first argument implies that
(10) 
Moreover, recall the discrete set with such that , where is the closest point to in . An example of such a set can be obtained for instance by a uniform grid of points in .
As in the proof of Theorem 3, let . Moreover, let be the closest point to in . We have:
We prove the corollary by bounding and separately.
To bound , note that