Re-evaluating evaluation

06/07/2018 ∙ by David Balduzzi, et al. ∙ 0

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agent-vs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation -- since there is no harm (computational cost aside) from including all available tasks and agents.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 16

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Evaluation is a key driver of progress in machine learning, with e.g. ImageNet

deng:09 and the Arcade Learning Environment BellemareNVB13

enabling subsequent breakthroughs in supervised and reinforcement learning

krizhevsky:12 ; mnih:15 . However, developing evaluations has received little systematic attention compared to developing algorithms. Immense amounts of compute is continually expended smashing algorithms and tasks together – but the results are almost never used to evaluate and optimize evaluations. In a striking asymmetry, results are almost exclusively applied to evaluate and optimize algorithms.

The classic train-and-test paradigm on common datasets, which has served the community well donoho:15 , is reaching its limits. Three examples suffice. Adversarial attacks have complicated evaluation, raising questions about which attacks to test against szegedy:13 ; tramer:18 ; kurakin:18 ; uesato:18 . Training agents far beyond human performance with self-play means they can only really be evaluated against each other silver:17 ; silver:17a . The desire to build increasingly general-purpose agents has led to a proliferation of environments: Mujoco, DM Lab, Open AI Gym, Psychlab and others todorov:12 ; beattie:16 ; brockman:16 ; leibo:18 .

In this paper we pause to ask, and partially answer, some basic questions about evaluation: Q1. Do tasks test what we think they test? Q2. When is a task redundant? Q3. Which tasks (and agents) matter the most? Q4. How should evaluations be evaluated?

We consider two scenarios: agent vs task (AvT), where algorithms are evaluated on suites of datasets or environments; and agent vs agent (AvA), where agents compete directly as in Go and Starcraft. Our goal is to treat tasks and agents symmetrically – with a view towards, ultimately, co-optimizing agents and evaluations. From this perspective AvA, where the task is (beating) another agent, is especially interesting. Performance in AvA is often quantified using Elo ratings Elo78 or the closely related TrueSkill herbrich:07 . There are two main problems with Elo. Firstly, Elo bakes-in the assumption that relative skill is transitive; but Elo is meaningless – it has no predictive power – in cyclic games like rock-paper-scissors. Intransitivity has been linked to biodiversity in ecology, and may be useful when evolving populations of agents frean:01 ; kerr:02 ; laird:06 ; szolnoki:14 . Secondly, an agent’s Elo rating can be inflated by instantiating many copies of an agent it beats (or conversely). This can cause problems when Elo guides hyper-optimization methods like population-based training jaderberg:17b . Similarly, the most important decision when constructing a task-suite is which tasks to include. It is easy, and all too common, to bias task-suites in favor of particular agents or algorithms.

1.1 Overview

Section 2 presents background information on Elo and tools for working with antisymmetric matrices, such as the Schur decomposition and combinatorial Hodge theory. A major theme underlying the paper is that the fundamental algebraic structure of tournaments and evaluation is antisymmetric dgm:18 . Techniques specific to antisymmetric matrices are less familiar to the machine learning community than approaches like PCA that apply to symmetric matrices and are typically correlation-based.

Section 3 presents a unified approach to representing evaluation data, where agents and tasks are treated symmetrically. A basic application of the approach results in our first contribution: a multidimensional Elo rating (mElo) that handles cyclic interactions. We also sketch how the Schur decomposition can uncover latent skills and tasks, providing a partial answer to Q1. We illustrate mElo on the domain of training an AlphaGo agent DSilverHMGSDSAPL16 .

The second contribution of the paper is Nash averaging, an evaluation method that is invariant to redundant tasks and agents, see section 4. The basic idea is to play a meta-game on evaluation data lanctotl:17 . The meta-game has a unique maximum entropy Nash equilibrium. The key insight of the paper is that the maxent Nash adapts automatically to the presence of redundant tasks and agents. The maxent Nash distribution thus provides a principled answer to Q2 and Q3: which tasks and agents do and do not matter is determined by a meta-game. Finally, expected difficulty of tasks under the Nash distribution on agents yields a partial answer to Q4. The paper concludes by taking a second look at the performance of agents on Atari. We find that, under Nash averaging, human performance ties with the best agents, suggesting better-than-human performance has not yet been achieved.

1.2 Related work

Legg and Hutter developed a definition of intelligence which, informally, states “intelligence measures an agent’s ability to achieve goals in a wide range of environments” legg:05 ; legg:13 . Formally, they consider all computable tasks weighted by algorithmic complexity solomonoff:64 ; kolmogorov:65 ; chaitin:66 . Besides being incomputable, the distribution places (perhaps overly) heavy weight on the simplest tasks.

A comprehensive study of performance metrics for machine learning and AI can be found in ferri:09 ; hernandez:12 ; hernandez:17 ; hernandez:17a ; Olson2017

. There is a long history of psychometric evaluation in humans, some of which has been applied in artificial intelligence

spearman:04 ; woolley:10 ; bringsjord:11 . Bradley-Terry models provide a general framework for pairwise comparison hunter:04 . Researchers have recently taken a second look at the arcade learning environment BellemareNVB13 and introduced new performance metrics Machado17a . However, the approach is quite particular. Recent work using agents to evaluate games has somewhat overlapping motivation with this paper liapis:13 ; horn:14 ; nielsen:15 ; silva:17 ; volz:18 . Item response theory is an alternative, and likely complementary, approach to using agents to evaluate tasks hambleton:91 that has recently been applied to study the performance of agents on the Arcade Learning Environment martinez:17 .

Our approach draws heavily on work applying combinatorial Hodge theory to statistical ranking jiang:11

and game theory

candogan:11 ; candogan:13 ; candogan:13a . We also draw on empirical game theory Walsh03 ; Wellman06 , by using a meta-game to “evaluate evaluations”, see section 4. Empirical game theory has been applied to domains like poker and continuous double auctions, and has recently been extended to asymmetric games PhelpsPM04 ; PhelpsCMNPS07 ; PonsenTKR09 ; BloembergenTHK15 ; Tuyls18 . von Neumann winners in the dueling bandit setting and NE-regret are related to Nash averaging dudik:15 ; balsubramani:16 ; Jordan:07 ; Jordan:10 .

2 Preliminaries

Notation.Vectors are column vectors. and denote the constant vectors of zeros and ones respectively. We sometimes use subscripts to indicate dimensions of vectors and matrices, e.g. or and sometimes their entries, e.g. or ; no confusion should result. The unit vector with a 1 in coordinate is . Proofs and code are in the appendix.

2.1 The Elo rating system

Suppose agents play a series of pairwise matches against each other. Elo assigns a rating to each player based on their wins and losses, which we represent as an -vector

. The predicted probability of

beating given their Elo ratings is

(1)

The constant is not important in what follows, so we pretend . Observe that only the difference between Elo ratings affects win-loss predictions. We therefore impose that Elo ratings sum to zero, , without loss of generality. Define the loss,

(2)

and is the true probability of beating . Suppose the match pits player against , with outcome if wins and if loses. Online gradient descent on obtains

(3)

Choosing learning rate or recovers the updates introduced by Arpad Elo in Elo78 .

The win-loss probabilities predicted by Elo ratings can fail in simple cases. For example, rock, paper and scissors will all receive the same Elo ratings. Elo’s predictions are for all – and so Elo has no predictive power for any given pair of players (e.g. paper beats rock with ).

What are the Elo update’s fixed points?

Suppose we batch matches to obtain empirical estimates of the probability of player

beating : . As the number of matches approaches infinity, the empirical estimates approach the true probabilities .

Proposition 1.

Elo ratings are at a stationary point under batch updates iff the matrices of empirical probabilities and predicted probabilities have the same row-sums (or, equivalently the same column-sums):

(4)

Many different win-loss probability matrices result in identical Elo ratings. The situation is analogous to how many different joint probability distributions can have the same marginals. We return to this topic in section 

3.1.

2.2 Antisymmetric matrices

We recall some basic facts about antisymmetric matrices. Matrix is antisymmetric if . Antisymmetric matrices have even rank and imaginaryeigenvalues . Any antisymmetric matrix admits a real Schur decomposition:

(5)

where is orthogonal and consists of zeros except for diagonal-blocks of the form:

(6)

The entries of are real numbers, found by multiplying the eigenvalues of by .

Proposition 2.

Given matrix with rank

and singular value decomposition

. Construct antisymmetric matrix

(7)

Then the thin Schur decomposition of is where the nonzero pairs in are the singular values in and

(8)

Combinatorial Hodge theory is developed by analogy with differential geometry, see jiang:11 ; candogan:11 ; candogan:13 ; candogan:13a . Consider a fully connected graph with vertex set . Assign a flow to each edge of the graph. The flow in the opposite direction is , so flows are just antisymmetric matrices. The flow on a graph is analogous to a vector field on a manifold.

The combinatorial gradient of an -vector is the flow: . Flow is a gradient flow if for some , or equivalently if for all . The divergence of a flow is the -vector . The divergence measures the contribution to the flow of each vertex, considered as a source. The curl

of a flow is the three-tensor

. Finally, the rotation is .

Theorem (Hodge decomposition, jiang:11 ).

(i) for any satisfying .
(ii) for any flow . (iii) for any vector .
(iv) The vector space of antisymmetric matrices admits an orthogonal decomposition

(9)

with respect to the standard inner product . Concretely, any antisymmetric matrix decomposes as

(10)

Sneak peak. The divergence recovers Elo ratings or just plain average performance depending on the scenario. The Hodge decomposition separates transitive (captured by averages or Elo) from cyclic interactions (rock-paper-scissors), and explains when Elo ratings make sense. The Schur decomposition is a window into the latent skills and tasks not accounted for by Elo and averages.

3 On the algebraic structure of evaluation

The Schur decomposition and combinatorial Hodge theory provide a unified framework for analyzing evaluation data in the AvA and AvT scenarios. In this section we provide some basic tools and present a multidimensional extension of Elo that handles cyclic interactions.

3.1 Agents vs agents (AvA)

In AvA, results are collated into a matrix of win-loss probabilities based on relative frequencies. Construct with . Matrix is antisymmetric since .

When can Elo correctly predict win-loss probabilities? The answer is simple in logit space:

Proposition 3.

(i) If probabilities are generated by Elo ratings then the divergence of its logit is . That is,

(11)

(ii) There is an Elo rating that generates probabilities iff . Equivalently, iff for all .

Elo is, essentially, a uniform average in logit space. Elo’s predictive failures are due to the cyclic component that uniform averaging ignores.

Multidimensional Elo (mElo). Elo ratings bake-in the assumption that relative skill is transitive. However, there is no single dominant strategy in games like rock-paper-scissors or (arguably) StarCraft. Rating systems that can handle intransitive abilities are therefore necessary. An obvious approach is to learn a feature vector and a rating vector per player, and predict . Unfortunately, this reduces to the standard Elo rating since is a scalar.

Handling intransitive abilities requires learning an approximation to the cyclic component . Combining the Schur and Hodge decompositions allows to construct low-rank approximations that extend Elo. Note, antisymmetric matrices have even rank. Consider

(12)

where the rows of are orthogonal to each other, to , and to . The larger , the better the approximation. Let mElo assign each player Elo rating and -dimensional vector . Vanilla Elo uses . The mElo win-loss prediction is

(13)

Online updates can be computed by gradient descent, see section E, with orthogonality enforced.

3.2 Application: predicting win-loss probabilities in Go

Elo ratings are widely used in Chess and Go. We compared the predictive capabilities of Elo and the simplest extension mElo on eight Go algorithms taken from extended data table 9 in DSilverHMGSDSAPL16 : seven variants of AlphaGo, and Zen. The Frobenius norms and logistic losses are and for Elo vs the empirical probabilities and and for mElo.

To better understand the difference, we zoom in on three algorithms that were observed to interact non-transitively in Tuyls18 : with value net, with policy net, and Zen. Elo’s win-loss predictions are poor (Table Elo: Elo incorrectly predicts both that likely beats and likely beats Zen), whereas mElo (Table mElo) correctly predicts likely winners in all cases (Table empirical), with more accurate probabilities:

Elo Zen
- 0.41 0.58
0.59 - 0.67
Zen 0.42 0.33 -
empirical Zen
- 0.7 0.4
0.3 - 1.0
Zen 0.6 0.0 -
mElo Zen
- 0.72 0.46
0.28 - 0.98
Zen 0.55 0.02 -

3.3 Agents vs tasks (AvT)

In AvT, results are represented as an matrix : rows are agents, columns are tasks, entries are scores (e.g. accuracy or total reward). Subtract the total mean, so the sum of all entries of is zero. We recast both agents and tasks as players and construct an antisymmetric -matrix. Let and be the average skill of each agent and the average difficulty of each task. Define . Let be the concatenation of and . We construct the antisymmetric matrix

(14)

The top-right block of is agent performance on tasks; the bottom-left is task difficulty for agents. The top-left block compares agents by their average skill on tasks; the bottom-right compares tasks by their average difficulty for agents. Average skill and difficulty explain the data if the score of agent on task is , the agent’s skill minus the task’s difficulty, for all . Paralleling proposition 3, averages explain the data, , iff .

The failure of averages to explain performance is encapsulated in and . By proposition 2, the SVD of and Schur decomposition of are equivalent. If the SVD is then the rows of represent the latent abilities exhibited by agents and the rows of represent the latent problems posed by tasks.

4 Invariant evaluation

Evaluation is often based on metrics like average performance or Elo ratings. Unfortunately, two (or two hundred) tasks or agents that look different may test/exhibit identical skills. Overrepresenting particular tasks or agents introduces biases into averages and Elo – biases that can only be detected post hoc

. Humans must therefore decide which tasks or agents to retain, to prevent redundant agents or tasks from skewing results. At present,

evaluation is not automatic and does not scale. To be scalable and automatic, an evaluation method should always benefit from including additional agents and tasks. Moreover, it should adjust automatically and gracefully to redundant data.

Definition 1.

An evaluation method maps data to a real-valued function on players (that is, agents or agents and tasks):

(15)

Desired properties. An evaluation method should be:

  1. Invariant: adding redundant copies of an agent or task to the data should make no difference.

  2. Continuous: the evaluation method should be robust to small changes in the data.

  3. Interpretable: hard to formalize, but the procedure should agree with intuition in basic cases.

Elo and uniform averaging over tasks are examples of evaluation methods that invariance excludes.

4.1 Nash averaging

This section presents an evaluation method satisfying properties . We discuss AvA here, see section D for AvT. Given antisymmetric logit matrix , define a two-player meta-game with payoffs and for the row and column meta-players, where . The game is symmetric because and zero-sum because .

The row and column meta-players pick ‘teams’ of agents. Their payoff is the expected log-odds of their respective team winning under the joint distribution. If there is a dominant agent that has better than even odds of beating the rest, both players will pick it. In rock-paper-scissors, the only unbeatable-on-average team is the uniform distribution. In general, the value of the game is zero and the Nash equilibria are teams that are unbeatable in expectation.

A problem with Nash equilibria (NE) is that they are not unique, which forces the user to make choices and undermines interpretability vonneumann:44 ; nash:50 . Fortunately, for zero-sum games there is a natural choice of Nash:

Proposition 4 (maxent NE).

For antisymmetric there is a unique symmetric Nash equilibrium solving with greater entropy than any other Nash equilibrium.

Maxent Nash is maximally indifferent between players with the same empirical performance.

Definition 2.

The maxent Nash evaluation method for AvA is

(16)

where is the maxent Nash equilibrium and is the Nash average.

Invariance to redundancy is best understood by looking at an example; for details see section C.

Example 1 (invariance).

Consider two logit matrices, where the second adds a redundant copy of agent to the first:

0.0 4.6 -4.6
-4.6 0.0 4.6
4.6 -4.6 0.0

and  0.0 4.6 -4.6 -4.6 -4.6 0.0 4.6 4.6 4.6 -4.6 0.0 0.0 4.6 -4.6 0.0 0.0

The maxent Nash for is . It is easy to check that is Nash for for any and thus the maxent Nash for is . Maxent Nash automatically detects the redundant agents and distributes ’s mass over them equally.

Uniform averaging is not invariant to adding redundant agents; concretely whereas , falsely suggesting agent is superior. In contrast, and (the zero-vectors have different sizes because there are different numbers of agents). Nash averaging correctly reports no agent is better than the rest in both cases.

Figure 1: (A) The Nash assigned to agents; (B) the Nash assigned to environments.
Theorem 1 (main result for AvA111The main result for AvT is analogous, see section D. ).

The maxent NE has the following properties:

  1. Invariant: Nash averaging, with respect to the maxent NE, is invariant to redundancies in .

  2. Continuous: If is a Nash for and then is an -Nash for .

  3. Interpretable: (i) The maxent NE on is the uniform distribution, , iff the meta-game is cyclic, i.e. . (ii) If the meta-game is transitive, i.e. , then the maxent NE is the uniform distribution on the player(s) with highest rating(s) – there could be a tie.

See section C for proof and formal definitions. For interpretability, if then the transitive rating is all that matters: Nash averaging measures performance against the best player(s). If then no player is better than any other. Mixed cases cannot be described in closed form.

The continuity property is quite weak: theorem 1.2 shows the payoff is continuous: a team that is unbeatable for is -beatable for nearby

. Unfortunately, Nash equilibria themselves can jump discontinuously when

is modified slightly. Perturbed best response converges to a more stable approximation to Nash hofbauer:02 ; sandholm:10 that unfortunately is not invariant.

Example 2 (continuity).

Consider the cyclic and transitive logit matrices

(17)

The maxent Nash equilibria and Nash averages of are

(18)

The maxent Nash is the uniform distribution over agents in the cyclic case (), and is concentrated on the first player when it dominates the others (). When the optimal team has most mass on the first and last players. Nash jumps discontinuously at .

4.2 Application: re-evaluation of agents on the Arcade Learning Environment

To illustrate the method, we re-evaluate the performance of agents on Atari BellemareNVB13 . Data is taken from results published in wang:16 ; vanhasselt:16 ; ostrovski:17 ; hessel:17 . Agents include rainbow, dueling networks, prioritized replay, pop-art, DQN, count-based exploration and baselines like human, random-action and no-action. The 20 agents evaluated on 54 environments are represented by matrix . It is necessary to standardize units across environments with quite different reward structures: for each column we subtract the and divide by the so scores lie in .

We introduce a meta-game where row meta-player picks aims to pick the best distribution on agents and column meta-player aims to pick the hardest distribution on environments, see section D for details. We find a Nash equilibrium using an LP-solver; it should be possible to find the maxent Nash using the algorithm in ortiz:06 ; ortiz:07 . The Nash distributions are shown in figure 1. The supports of the distributions are the ‘core agents’ and the ‘core environments’ that form unexploitable teams. See appendix for tables containing all skills and difficulties. panel B.

Figure 2A shows the skill of agents under uniform and Nash averaging over environments; panel B shows the difficulty of environments under uniform and Nash averaging over agents. There is a tie for top between the agents with non-zero mass – including human. This follows by the indifference principle for Nash equilibria: strategies with support have equal payoff.

Our results suggest that the better-than-human performance observed on the Arcade Learning Environment is because ALE is skewed towards environments that (current) agents do well on, and contains fewer environments testing skills specific to humans. Solving the meta-game automatically finds a distribution on environments that evens out the playing field and, simultaneously, identifies the most important agents and environments.

Figure 2: Comparison of uniform and Nash averages. (A) Skill of agents by uniform and Nash averaging over environments. (B) Difficulty of environments under uniform and Nash averaging over agents. Agents and environments are sorted by Nash-averages.

5 Conclusion

A powerful guiding principle when deciding what to measure is to find quantities that are invariant to naturally occurring transformations. The determinant is computed over a basis – however, the determinant is invariant to the choice of basis since

for any invertible matrix

. Noether’s theorem implies the dynamics of a physical system with symmetries obeys a conservation law. The speed of light is fundamental because it is invariant to the choice of inertial reference frame.

One must have symmetries in mind to talk about invariance. What are the naturally occurring symmetries in machine learning? The question admits many answers depending on the context, see e.g. diaconis:88 ; lecun:98 ; kondor:03 ; kondor:08 ; zaheer:17 ; hartford:18 ; kondor:18

. In the context of evaluating agents, that are typically built from neural networks, it is unclear

a priori

whether two seemingly different agents – based on their parameters or hyperparameters – are

actually different. Further, it is increasingly common that environments and tasks are parameterized – or are learning agents in their own right, see self-play silver:17 ; silver:17a , adversarial attacks szegedy:13 ; tramer:18 ; kurakin:18 ; uesato:18 , and automated curricula sukhbaatar:17 . The overwhelming source of symmetry when evaluating learning algorithms is therefore redundancy: different agents, networks, algorithms, environments and tasks that do basically the same job.

Nash evaluation computes a distribution on players (agents, or agents and tasks) that automatically adjusts to redundant data. It thus provides an invariant approach to measuring agent-agent and agent-environment interactions. In particular, Nash averaging encourages a maximally inclusive approach to evaluation: computational cost aside, the method should only benefit from including as many tasks and agents as possible. Easy tasks or poorly performing agents will not bias the results. As such Nash averaging is a significant step towards more objective evaluation.

Nash averaging is not always the right tool. Firstly, it is only as good as the data: garbage in, garbage out. Nash decides which environments are important based on the agents provided to it, and conversely. As a result, the method is blind to differences between environments that do not make a difference to agents and vice versa. Nash-based evaluation is likely to be most effective when applied to a diverse array of agents and environments. Secondly, for good or ill, Nash averaging removes control from the user. One may have good reason to disagree with the distribution chosen by Nash. Finally, Nash is a harsh master. It takes an adversarial perspective and may not be the best approach to, say, constructing automated curricula – although boosting is a related approach that works well freund:96 ; schapire:12 . It is an open question whether alternate invariant evaluations can be constructed, game-theoretically or otherwise.

Acknowledgements. We thank Georg Ostrovski, Pedro Ortega, José Hernández-Orallo and Hado van Hasselt for useful feedback.

References

  • (1) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, 2009.
  • (2) M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, “The arcade learning environment: An evaluation platform for general agents,” J. Artif. Intell. Res., vol. 47, pp. 253–279, 2013.
  • (3)

    A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in

    Advances in Neural Information Processing Systems (NIPS), 2012.
  • (4) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, 02 2015.
  • (5)

    D. Donoho, “50 years of Data Science,” in

    Based on a presentation at the Tukey Centennial workshop, 2015.
  • (6) C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in arXiv:1312.6199, 2013.
  • (7) F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble Adversarial Training: Attacks and Defenses,” in ICLR, 2018.
  • (8) A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, C. Xie, J. Wang, Z. Zhang, Z. Ren, A. Yuille, S. Huang, Y. Zhao, Y. Zhao, Z. Han, J. Long, Y. Berdibekov, T. Akiba, S. Tokui, and M. Abe, “Adversarial Attacks and Defences Competition,” in arXiv:1804.00097, 2018.
  • (9) J. Uesato, B. O’Donoghue, A. van den Oord, and P. Kohli, “Adversarial Risk and the Dangers of Evaluating Against Weak Attacks,” in ICML, 2018.
  • (10) D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, “Mastering the game of Go without human knowledge,” Nature, vol. 550, pp. 354–359, 2017.
  • (11) D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” in arXiv:1712.01815, 2017.
  • (12) E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in IROS, 2012.
  • (13) C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, and S. Petersen, “DeepMind Lab,” in arXiv:1612.03801, 2016.
  • (14) G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” 2016.
  • (15) J. Z. Leibo, C. de Masson d’Autume, D. Zoran, D. Amos, C. Beattie, K. Anderson, A. G. Castañeda, M. Sanchez, S. Green, A. Gruslys, S. Legg, D. Hassabis, and M. M. Botvinick, “Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents,” in arXiv:1801.08116, 2018.
  • (16) A. E. Elo, The Rating of Chess players, Past and Present. Ishi Press International, 1978.
  • (17) R. Herbrich, T. Minka, and T. Graepel, “TrueSkill: a Bayesian skill rating system,” in NIPS, 2007.
  • (18) M. Frean and E. R. Abraham, “Rock-scissors-paper and the survival of the weakest,” Proc. R. Soc. Lond. B, no. 268, pp. 1323–1327, 2001.
  • (19) B. Kerr, M. A. Riley, M. W. Feldman, and B. J. M. Bohannan, “Local dispersal promotes biodiversity in a real-life game of rock–paper–scissors,” Nature, no. 418, pp. 171–174, 2002.
  • (20) R. A. Laird and B. S. Schamp, “Competitive Intransitivity Promotes Species Coexistence,” The American Naturalist, vol. 168, no. 2, 2006.
  • (21) A. Szolnoki, M. Mobilia, L.-L. Jiang, B. Szczesny, A. M. Rucklidge, and M. Perc, “Cyclic dominance in evolutionary games: a review,” J R Soc Interface, vol. 11, no. 100, 2014.
  • (22) M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, and K. Kavukcuoglu, “Population based training of neural networks,” CoRR, vol. abs/1711.09846, 2017.
  • (23) D. Balduzzi, S. Racanière, J. Martens, J. Foerster, K. Tuyls, and T. Graepel, “The mechanics of -player differentiable games,” in ICML, 2018.
  • (24) D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of Go with deep neural networks and tree search,” Nature, vol. 529, no. 7587, pp. 484–489, 2016.
  • (25) M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Perolat, D. Silver, and T. Graepel, “A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning,” in NIPS, 2017.
  • (26) S. Legg and M. Hutter, “A universal measure of intelligence for artificial agents,” in IJCAI, 2005.
  • (27) S. Legg and J. Veness, “An Approximation of the Universal Intelligence Measure,” in Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence, 2013.
  • (28) R. J. Solomonoff, “A formal theory of inductive inference I, II,” Inform. Control, vol. 7, no. 1-22, 224-254, 1964.
  • (29) A. N. Kolmogorov, “Three approaches to the quantitative definition of information,” Problems Inform. Transmission, vol. 1, no. 1, pp. 1–7, 1965.
  • (30) G. J. Chaitin, “On the length of computer programs for computing finite binary sequences,” J Assoc. Comput. Mach., vol. 13, pp. 547–569, 1966.
  • (31) C. Ferri, J. Hernández-Orallo, and R. Modroiu, “An experimental comparison of performance measures for classification,” Pattern Recognition Letters, no. 30, pp. 27–38, 2009.
  • (32) J. Hernández-Orallo, P. Flach, and C. Ferri, “A Unified View of Performance Metrics: Translating Threshold Choice into Expected Classification Loss,” JMLR, no. 13, pp. 2813–2869, 2012.
  • (33) J. Hernández-Orallo, The Measure of All Minds: Evaluating Natural and Artificial Intelligence. Cambridge University Press, 2017.
  • (34) J. Hernández-Orallo, “Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement,” Artificial Intelligence Review, vol. 48, no. 3, pp. 397–447, 2017.
  • (35) R. S. Olson, W. La Cava, P. Orzechowski, R. J. Urbanowicz, and J. H. Moore, “PMLB: a large benchmark suite for machine learning evaluation and comparison,” BioData Mining, vol. 10, p. 36, Dec 2017.
  • (36) C. Spearman, “‘General Intelligence,’ objectively determined and measured,” Am. J. Psychol., vol. 15, no. 201, 1904.
  • (37) A. Woolley, C. Fabris, A. Pentland, N. Hashmi, and T. Malone, “Evidence for a Collective Intelligence Factor in the Performance of Human Groups,” Science, no. 330, pp. 686–688, 2010.
  • (38) S. Bringsjord, “Psychometric artificial intelligence,” Journal of Experimental & Theoretical Artificial Intelligence, vol. 23, no. 3, pp. 271–277, 2011.
  • (39) D. R. Hunter, “MM algorithms for generalized Bradley-Terry models,” Annals of Statistics, vol. 32, no. 1, pp. 384–406, 2004.
  • (40) M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling, “Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents,” Journal of Artificial Intelligence Research (JAIR), vol. 61, pp. 523–562, 2018.
  • (41) A. Liapis, G. N. Yannakakis, and J. Togelius, “Towards a Generic Method of Evaluating Game Levels,” in Artificial Intelligence in Digital Interactive Entertainment (AIIDE), 2013.
  • (42) B. Horn, S. Dahlskog, N. Shaker, G. Smith, and J. Togelius, “A Comparative Evaluation of Procedural Level Generators in the Mario AI Framework,” in Foundations of Digital Games, 2014.
  • (43) T. S. Nielsen, G. Barros, J. Togelius, and M. J. Nelson, “General video game evaluation using relative algorithm performance profiles,” in EvoApplications, 2015.
  • (44) F. de Mesentier Silva, S. Lee, J. Togelius, and A. Nealen, “AI-based Playtesting of Contemporary Board Games,” in Foundations of Digital Games (FDG), 2017.
  • (45) V. Volz, J. Schrum, J. Liu, S. M. Lucas, A. M. Smith, and S. Risi, “Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Network,” in GECCO, 2018.
  • (46) R. K. Hambleton, H. Swaminathan, and H. J. Rogers, Fundamentals of item response theory. Sage Publications, 1991.
  • (47) F. Martínez-Plumed and J. Hernández-Orallo, “ AI results for the Atari 2600 games: difficulty and discrimination using IRT,” in Workshop on Evaluating General-Purpose AI (EGPAI at IJCAI), 2017.
  • (48) X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye, “Statistical ranking and combinatorial Hodge theory,” Math. Program., Ser. B, vol. 127, pp. 203–244, 2011.
  • (49) O. Candogan, I. Menache, A. Ozdaglar, and P. A. Parrilo, “Flows and Decompositions of Games: Harmonic and Potential Games,” Mathematics of Operations Research, vol. 36, no. 3, pp. 474–503, 2011.
  • (50) O. Candogan, A. Ozdaglar, and P. A. Parrilo, “Near-Potential Games: Geometry and Dynamics,” ACM Trans Econ Comp, vol. 1, no. 2, 2013.
  • (51) O. Candogan, A. Ozdaglar, and P. A. Parrilo, “Dynamics in near-potential games,” Games and Economic Behavior, vol. 82, pp. 66–90, 2013.
  • (52)

    W. E. Walsh, D. C. Parkes, and R. Das, “Choosing samples to compute heuristic-strategy nash equilibrium,” in

    Proceedings of the Fifth Workshop on Agent-Mediated Electronic Commerce, 2003.
  • (53) M. P. Wellman, “Methods for empirical game-theoretic analysis,” in Proceedings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, pp. 1552–1556, 2006.
  • (54) S. Phelps, S. Parsons, and P. McBurney, “An Evolutionary Game-Theoretic Comparison of Two Double-Auction Market Designs,” in Agent-Mediated Electronic Commerce VI, Theories for and Engineering of Distributed Mechanisms and Systems, AAMAS Workshop, pp. 101–114, 2004.
  • (55) S. Phelps, K. Cai, P. McBurney, J. Niu, S. Parsons, and E. Sklar, “Auctions, Evolution, and Multi-agent Learning,” in AAMAS and 7th European Symposium on Adaptive and Learning Agents and Multi-Agent Systems (ALAMAS), pp. 188–210, 2007.
  • (56) M. Ponsen, K. Tuyls, M. Kaisers, and J. Ramon, “An evolutionary game-theoretic analysis of poker strategies,” Entertainment Computing, vol. 1, no. 1, pp. 39–45, 2009.
  • (57) D. Bloembergen, K. Tuyls, D. Hennes, and M. Kaisers, “Evolutionary dynamics of multi-agent learning: A survey,” J. Artif. Intell. Res. (JAIR), vol. 53, pp. 659–697, 2015.
  • (58) K. Tuyls, J. Perolat, M. Lanctot, J. Z. Leibo, and T. Graepel, “A Generalised Method for Empirical Game Theoretic Analysis ,” in AAMAS, 2018.
  • (59) M. Dudik, K. Hofmann, R. E. Schapire, A. Slivkins, and M. Zoghi, “Contextual Dueling Bandits,” in COLT, 2015.
  • (60) A. Balsubramani, Z. Karnin, R. E. Schapire, and M. Zoghi, “Instance-dependent Regret Bounds for Dueling Bandits,” in COLT, 2016.
  • (61) P. R. Jordan, C. Kiekintveld, and M. P. Wellman, “Empirical game-theoretic analysis of the TAC supply chain game,” in 6th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2007), Honolulu, Hawaii, USA, May 14-18, 2007, p. 193, 2007.
  • (62) P. R. Jordan, Practical Strategic Reasoning with Applications in Market Games. PhD thesis, 2010.
  • (63) J. von Neumann and O. Morgenstern, Theory of Games and Economic Behavior. Princeton University Press, Princeton NJ, 1944.
  • (64) J. F. Nash, “Equilibrium Points in -Person Games,” Proc Natl Acad Sci U S A, vol. 36, no. 1, pp. 48–49, 1950.
  • (65) J. Hofbauer and W. H. Sandholm, “On the global convergence of stochastic fictitious play,” Econometrica, vol. 70, no. 6, pp. 2265–2294, 2002.
  • (66) W. H. Sandholm, Population Games and Evolutionary Dynamics. MIT Press, 2010.
  • (67) Z. Wang, T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas, “Dueling Network Architectures for Deep Reinforcement Learning,” in ICML, 2016.
  • (68) H. van Hasselt, A. Guez, M. Hessel, V. Mnih, and D. Silver, “Learning values across many orders of magnitude,” in NIPS, 2016.
  • (69) G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos, “Count-Based Exploration with Neural Density Models,” in ICML, 2017.
  • (70) M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, “Rainbow: Combining Improvements in Deep Reinforcement Learning,” in AAAI, 2018.
  • (71) L. E. Ortiz, R. E, Schapire, and S. M. Kakade, “Maximum entropy correlated equilibrium,” in Technical Report TR-2006-21, CSAIL MIT, 2006.
  • (72) L. E. Ortiz, R. E. Schapire, and S. M. Kakade, “Maximum entropy correlated equilibria,” in AISTATS, 2007.
  • (73) P. Diaconis, Group Representations in Probability and Statistics. Institute of Mathematical Statistics, 1988.
  • (74) Y. LeCun, , L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  • (75) R. Kondor and T. Jebara, “A kernel between sets of vectors,” in ICML, 2003.
  • (76) R. Kondor, “Group theoretical methods in machine learning,” in PhD dissertation, 2008.
  • (77) M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, “Deep Sets,” in NIPS, 2017.
  • (78) J. Hartford, D. R. Graham, K. Leyton-Brown, and S. Ravanbakhsh, “Deep Models of Interactions Across Sets,” in ICML, 2018.
  • (79) R. Kondor, Z. Lin, and S. Trivedi, “Clebsch–Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network,” in NIPS, 2018.
  • (80) S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus, “Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play,” in ICLR, 2017.
  • (81) Y. Freund and R. E. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” Journal of Computer and System Sciences, 1996.
  • (82) R. Schapire and Y. Freund, Boosting: Foundations and Algorithms. MIT Press, 2012.
  • (83) R. J. Vandenberg and C. E. Lance, “A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research,” Organizational Research Methods, vol. 3, no. 1, pp. 4–70, 2000.

A Invariance: Further motivation

Consider agents A, B, C evaluated on a benchmark suite comprising three tasks:

task 1 task 2 task 3 average rank
agent A 89 93 76 86
agent B 85 85 85 85
agent C 79 74 99 84

On average, agent A performs best and agent C performs worst. Consider a second benchmark suite containing an additional fourth task. On the second benchmark suite, agent A performs worst and agent C performs best on average. However, a closer look at the second suite reveals that the additional task is a minor variant of one of the original three tasks.

task 1 task 2 task 3a task 3b average rank
agent A 89 93 76 77 84
agent B 85 85 85 84 85
agent C 79 74 99 98 88

Measuring performance by uniformly averaging over tasks in a benchmark suite is sensitive to the set of tasks that are included in the suite. Including redundant tasks, whether consciously or not, can easily skew average performance in favor of or against particular agents.

The problem becomes more serious when agent performance is measured against other agents, as in games such as Go, Chess or StarCraft. It is easy to manipulate the measured performance of agents by tweaking the composition of the population used to evaluate them. Consider the following example:

agent A agent B agent C Elo
agent A 0.5 0.9 0.1 0
agent B 0.1 0.5 0.9 0
agent C 0.9 0.1 0.5 0

The three agents exhibit rock-paper-scissors dynamics; their Elo ratings (normalized to sum to zero) are all zero. However, adding a second copy of agent C decreases the Elo rating of agent A and increases the Elo rating of agent B:

agent A agent B agent C agent C Elo
agent A 0.5 0.9 0.1 0.1 -63
agent B 0.1 0.5 0.9 0.9 63
agent C 0.9 0.1 0.5 0.5 0
agent C 0.9 0.1 0.5 0.5 0

That is, the Elo ratings of agents A and B are easily manipulated by changing the structure of the population.

The examples above suggest it is important to find evaluation metrics that are

invariant to redundant changes in the population of agents or suite of tasks.

Related work.

A different notion of measurement invariance has been proposed in the psychometric and consumer research literatures vandenberg:00 . There, measurement invariance refers to the statistical property that a measurement measures the same construct across a predefined set of groups. For example, whether or not a question in an IQ test is measurement invariant has to do with whether or not the question is interpreted in the same way by individuals with different cultural backgrounds.

B Proofs of propositions

Proof of proposition 1.

Proposition 1.

Batch Elo updates are stable if the matrices of empirical probabilities and predicted probabilities have the same row-sums (or, equivalently the same column-sums):

(19)
Proof.

Player ’s weights are updated in one batch after observing win-loss probabilities for each player . Observe that

(20)

The result follows. ∎

Proof of proposition 2.

Proposition 2.

Given matrix with rank and singular value decomposition . Construct antisymmetric matrix

(21)

Then the thin Schur decomposition of is where the eigenpairs in the are the singular values in and

(22)
Proof.

Direct computation; multiply out the matrices. ∎

Proof of proposition 3.

Proposition 3.

(i) If probabilities are generated by Elo ratings then the divergence of its logit is . That is,

(23)

(ii) There is an Elo rating that generates probabilities iff . Alternatively, iff for all .

Proof.

For the first claim, apply definitions and recall . For the second, apply the Hodge decomposition from section 2.2. ∎

Proof of proposition 4.

Proposition 4 (maxent NE).

The game , where is antisymmetric, has a unique symmetric Nash equilibrium , with greater entropy than any other Nash equilibrium.

Proof.

The Nash equilibria in a two-player zero-sum are rectangular: if and are Nash equilibria then so are and . Further, they form a convex polytope. Since , the set of Nash equilibria is also symmetric: if is a Nash equilibrium then so is . The entropy is strictly concave and therefore achieves a unique maximum on the compact, convex, symmetric set of Nash equilibria. ∎

C Proof of theorem 1

Theorem (main result for AvA).

The maxent NE has the following properties:

  1. Invariant: Nash averaging, with respect to the maxent NE, is invariant to redundancies in .

  2. Continuous: If is a Nash for and then is an -Nash for .

  3. Interpretable: (i) The maxent NE on is the uniform distribution, , iff the meta-game is cyclic, i.e. . (ii) If the meta-game is transitive, i.e. , then the maxent NE is the uniform distribution on the player(s) with highest rating(s) – there could be a tie.

c.1 Proof of theorem 1.1

First, we more precisely formalize invariance to redundancy.

Definition 3 (invariance to copying the last row and column).

Given antisymmetric matrix , denote the right-most column by . Assume the right-most column (and bottom row) of differs from all other columns. Construct antisymmetric matrix

(24)

by adding an additional copy of the right-most column (and bottom row) to . A family of functions

(25)

is invariant to adding a row and column according to (24) if

(26)

If the copied row is not unique and receives positive mass under maxent Nash, then maxent Nash will already be spreading mass across the copies. In that case, adding yet another copy will result in the maxent Nash on the larger mass spreading mass evenly across all copies.

Lemma 1.

Suppose is antisymmetric with Nash equilibrium . Construct from by adding a redundant copy of the right-most column and bottom row according (24). Then

(27)

is a Nash equilibrium for for all . Conversely, if is a Nash equilibrium for then

(28)

is a Nash equilibrium for .

Proof.

Since the value of the game on is zero, it follows that is a Nash equilibrium iff all the coordinates of are nonnegative, i.e. . Direct computation shows that , which implies is a Nash equilibrium for . The converse follows similarly. ∎

Finally, we prove theorem 1.2.

Proof.

The definition and lemma are stated in the particular case where the last column and row are copied. This is simply for notational convenience. They generalize trivially to copying arbitrary row/columns into arbitrary positions, and can be applied inductively to the cases where is constructed from by inserting multiple redundant copies. ∎

c.2 Proof of theorem 1.2

Recall the max-norm on matrices is .

Definition 4 (-Nash equilibrium).

A joint strategy is an -Nash equilibrium for if the benefit from either player deviating, separately, is at most :

(29)

We are now ready to prove

  1. Continuous: If is a Nash for and then is an -Nash for .

Proof.

Suppose is a Nash equilibrium for the antisymmetric matrix . Observe that

(30)

for any distribution because since is antisymmetric. It follows that

(31)

The first term on the right-hand-side is since is a Nash equilibrium for and the value of the game is zero. The second term on the right-hand-side is because and and are probability distributions. ∎

Note that since for any , the divergence from Nash is also controlled by how well approximates in the operator or Frobenius norms.

The proof is adapted from the proof of the following lemma in Tuyls18 .

Lemma 2 (Nash on approximate games).

Suppose is a Nash equilibrium for and that . Then is a -Nash for .

Our result is slightly sharper because we specialize to antisymmetric matrices.

c.3 Proof of theorem 1.3

Proof.

(i) If then and implying the uniform distribution is a Nash equilibrium because there is no incentive to deviate from . The uniform distribution also has maximum entropy. Conversely, suppose . Then has at least one positive and one negative coordinate because we know by antisymmetry. It follows that if the row player chooses then the column player is incentivized to choose a distribution with more mass on the positive coordinate and less on the negative. In other words, the column player will not play the uniform distribution, and the uniform distribution is therefore not a Nash equilibrium.

(ii) By assumption , so decouples into an independent maximization problem with respect to and minimization problem with respect to . It follows that the optimal distribution concentrates mass on the maximal coordinate(s) of and so does . That is, to be a Nash equilibrium, and must place their mass on the maximal coordinate if it is unique and can distribute it arbitrarily over the set of maximal coordinates if there is a tie. Adding the condition that the Nash equilibrium has maximum entropy entails placing the uniform distribution over the maximal coordinate(s). ∎

D Nash averaging for agent-vs-task

Given score matrix , construct antisymmetric matrix

(32)

Note this differs from the antisymmetrization used in section 3.3, see next remark.

Remark 1.

The graph structure underlying AvT is bipartite: agents interact with tasks and tasks with agents, but there are no direct agent-agent or task-task interactions. When done in full generality, the definitions of , and take into account the graph structure, see jiang:11 . In particular, , and are computed differently on bipartite graphs than fully connected graphs – note the definitions in section 2.2 are specific to fully connected graphs. Working in full generality is overkill for our purposes. It suffices to introduce slightly ad hoc notation to handle the specific case of AvT.

Introduce the notation

(33)

where measures uniform average skill of agents on tasks and measures uniform average difficulty of tasks for agents. Let

(34)

Define a two-player zero-sum meta-game

(35)

The setup is the same as for AvA in the main text except the row and column meta-players each play two distributions: one on agents and one on tasks/environments. The same argument as in proposition 4 shows there is a unique symmetric maxent Nash equilibrium .

Definition 5.

The maxent Nash evaluation method for AvT is

(36)

where and are the maxent Nash equilibria over agents and environments and and are the Nash averages quantifying difficulty of environments (Nash averaged over agents) and skill of agents (Nash averaged over environments) respectively.

Suppose decomposes as

(37)

where and . Observe that

(38)

If then the game reduces to

(39)

and so the row player maximizes its payoff by putting all its agent-mass on the most skillful agent(s) and all of its environment-mass on the most difficult task(s) – and similarly for the column player.

It is easy to check that the maxent Nash has a uniform distribution on all agents iff , and has a uniform distribution on tasks iff . We thus obtain

Theorem 2 (main result for AvT).

The maxent NE has the following properties:

  1. Invariant: Nash averaging is invariant to redundancies in .

  2. Continuous: If is a Nash for and then is an -Nash for .

  3. Interpretable: (i) The agent component of the maxent NE on is the uniform distribution on agents, , iff .
    (ii) The task component of the maxent NE on is the uniform distribution on tasks, , iff .
    (iii) If the meta-game is transitive, i.e. , then the maxent NE is the uniform distribution on the most skillful agent(s) and the uniform distribution on the most difficult task(s) – there could be ties.

E Code for computing mElo updates

The routine takes as input: a pair of players , the probability of player beating player (which could be 0 or 1 if only a single match is observed on the given round), the rating vector and the matrix quantifying non-transitive interactions. It returns updates to the and entries of and .