The Problem Problem
Pity the curious solipsist, for there is a limit to the knowledge they may acquire. To see why, consider a solitaire game played on a planar grid of size . The goal is to place black stones on the board so as to surround as much territory as possible. Obviously, the optimal solution is simply to place stones all along the edge of the grid. Imagine one learns by trial and error how to play this game. Once the optimal solution is discovered then there is nothing left to learn. The cleverness obtainable by practicing this game is bounded. Now introduce an additional player who places a white stone after each black stone. The white stones and the territories they enclose become barriers, preventing additional expansion of the black territory. Thus, the game of Go is born—a game with enough emergent complexity to occupy millions of minds for millenia (Fairbairn, 1995).
. This definition suggests that the ceiling of a solipsist’s intelligence may only be raised by providing more and more environments of ever increasing diversity and complexity. To that end, recent work in artificial intelligence has relied on rich 3D simulation environments (e.g.Kempka et al. (2016); Beattie et al. (2016)). The resulting
agents have achieved proficiency at a wide range of tasks, such as navigating virtual mazes (Mirowski et al., 2016; Savinov et al., 2018), foraging over rough, naturalistic terrain (Espeholt et al., 2018; Hessel et al., 2018), and tests drawn from the neuroscience and cognitive psychology literatures like visual search (Leibo et al., 2018) and memory recall (Wayne et al., 2018). As impressive as these results are, we think the research program they represent has fallen into a solipsistic trap. Just like the aforementioned Go-playing solipsist, the cleverness these agents may achieve is bounded by their adaptive environment(s). Advancing artificial intelligence by this route demands the creation of more and more environments, a laborious process similar to videogame design. Scaling up this process has become a bottleneck dubbed the problem problem (see Glossary).
How then did intelligence arise in nature? We propose that life solved its own version of the problem problem because it is a multi-agent system where any species’s innovation determines the environment to which others must adapt. For example, it was early photosynthesizers that were the main source of atmospheric oxygen, setting the stage for the subsequent evolution of all the many organisms that depend on it for energy (Kasting and Siefert, 2002). Likewise, human cultural evolution continually generates new “rules of the game”, demanding continuous adaptation just to avoid being left behind by a changing world (North, 1991; Gintis, 2000; Ostrom, 2005; Greif, 2006).
The argument has two main ingredients. First, adaptive units must learn implementation policies for their high-level strategies by sequencing low-level action primitives (Leibo et al., 2017). Second, the high level strategies themselves change over time in response to the strategic choices of others (Smith and Price, 1973; Schluter, 2000; North, 1991; Gintis, 2000). Taken together, these two processes induce a sequence of challenges for the adaptive process that we term an autocurriculum. The rest of this paper is concerned with clarifying the autocurriculum concept and explaining how it provides a useful lens through which to view phenomena in evolutionary biology and multi-agent reinforcement learning. To that end, we offer a classification of the various kinds of autocurricula by their underlying social interaction (competition or cooperation). In the final part of the paper we consider the conditions under which autocurricula might generate human-like accumulation of innovations.
Innovations Arise on All Levels of Life’s Hierarchies
Life on Earth is characterized by interactions between adaptive units. Each adaptive unit is composed of a set of interacting adaptive subunits, each of which is itself composed of interacting subsubunits and so on (Fig. 1-A). For example, eukaryotic cells are composed of interacting prokaryotic organelles and human communities are made up of interacting individuals (Maynard Smith and Szathmary, 1997; Gerber et al., 1999; Ostrom, 2005). Consider the great feats of human intelligence—composing symphonies, sending astronauts to the moon, developing agricultural technology to feed billions of people. To which adaptive units should we attribute such success? One perspective, which we adopt here, postulates that these phenomena occur at the level of groups rather than individuals, i.e. human community, culture, or civilization. The truly intelligent adaptive units are massive multifaceted multi-agent systems that, while being composed of humans (as humans are composed of cells), have their own internal logic and computational mechanisms different from those of individual humans.
Innovations expand an adaptive unit’s behavioral repertoire with new robust and repeatable problem solving abilities (Reader and Laland, 2003), and may arise on any level of the hierarchy. A number of crucial innovations have had outsize influence on the subsequent history of life, for example the emergence of eukaryotic cells, multi-cellular organisms, and perhaps language (Maynard Smith and Szathmary, 1997). Human technological innovations like agriculture (Kavanagh et al., 2018) and industry (Clark, 2008) profoundly altered human lifestyles around the world. On the scale of human societies, the scope and effectiveness of institutions aimed at promoting cooperation have increased steadily over time (Pinker, 2011). The origins of such institutions like corporations, labor unions, parliamentary democracies, and inter-government organizations are all innovations of higher level adaptive units.
Some populations of adaptive units only change over time via relatively slow processes like genetic evolution (genes), while others adapt via much faster processes like reinforcement learning and cultural evolution (agents). Often interactions between processes on both timescales are significant, as in cases of gene-culture co-evolution (Boyd and Richerson, 1988). Many of the principles remain the same regardless of whether evolution or learning dominate (Börgers and Sarin, 1997; Bloembergen et al., 2015; Such et al., 2017). We have adopted the unifying term “adaptive unit” to highlight the deep similarity between these processes, and to smoothly cross-apply insights originating in different fields.
Both evolution and reinforcement learning offer insights about how existing behaviors and solutions can be refined, for example by sharpening or stabilizing a desirable behavior. However, much less is known about how qualitatively novel innovations originate in the first place. The problem is that spontaneous generation of useful complex behavior is extraordinarily unlikely, yet necessary for the refinement processes of evolution and reinforcement learning to work their magic. Indeed the more complex the behavior, the lower the odds of its being generated spontaneously. This problem is exponentially exacerbated when two or more agents are required to jointly explore the set of solutions available to them as a group. Nevertheless, humans have walked on the moon, created the internet, and cured smallpox. But how?
Exploration by Exploitation
Innovation may be explained by considering that the environment to which units adapt can change over time. This causes old adaptations to lose their shine and thereby motivates exploration toward new innovative solutions. Researcher-controlled non-stationary dynamics in machine learning, known ascurricula, can facilitate the acquisition of complex behaviors that would not be learnable otherwise, e.g. Asmuth et al. (2008); Bengio et al. (2009); Zaremba and Sutskever (2015); Narvekar (2017); Czarnecki et al. (2018). The idea is to structure learning by changing the underlying environment over time.
We call such a change in the underlying environment dynamics a challenge. More precisely, we can think of a challenge to a policy as a change in its relative value compared to other policies. Notice that challenges may be positive as well as negative in nature. A previously successful predation strategy may diminish in effectiveness as prey species evolve countermeasures; or a chance dispersal to an uninhabited island may present an opportunity to apply foraging strategies that would not work on the mainland due to excessive competition.
Challenges motivate adaptive units to explore (and thus to learn) by following the gradient of their experience. That is, adaptive units explore because the true value of their current policy is shifting over time. We call this exploration by exploitation. In contrast to the traditional view in reinforcement learning based on the exploration-exploitation tradeoff, this view does not involve any deliberate trade-off between exploration and exploitation. An adaptive unit experiences new states not because it chooses to depart from exploitation, but because its underlying environment has changed.
Notice that a curriculum can be seen exactly as a sequence of challenges. We argue in this paper that certain kinds of curricula may emerge naturally from the non-stationary dynamics of social interaction processes, without any need for environmental engineering. We call such curricula autocurricula, since each challenge in the sequence is generated by the system itself. Adaptive social behavior of individuals continually reshapes the strategic landscape. We now seek to classify the various ways in which this can happen. The main distinguishing factor is whether the underlying challenge is endogenous or exogenous to the adaptive unit under consideration. This distinction underlies the contrasting and complementary dynamics of competition and cooperation.
Exogenous Challenges Generate Autocurricula
An exogenous challenge is a challenge that originates outside the adaptive unit under consideration. For example, consider a two-player zero-sum game. Player one experiences an exogenous challenge when player two changes their strategy in such a way as to induce a change to player one’s best response strategy. Viewed from the perspective of player one, this creates a change in its experienced adaptive landscape since its learning objective, implementing its best response strategy, has changed (Fig. 1-B). Player two may shift strategy again once player one has successfully learned to implement its best response, thereby inducing yet another change in adaptive landscape (see Box 1). This interaction may produce a long sequence of novel challenges, an “arms race” (Dawkins and Krebs, 1979) driving both players to improve their skills, i.e. an exogenous autocurriculum.
Box 1. Duality of Strategy and Implementation
The perspective developed here emphasizes situations where individuals must simultaneously learn not only what strategic choice to make, but also how to implement said choice by sequencing elementary actions. Thus, at any given time, the goal of each player’s learning is to find an implementation for a strategy that best responds to the strategy profile of the others. Of course, since co-players do not just wait around for others to learn how they may be exploited, they too learn a best response. Whenever they change strategy they create a new adaptive landscape. Each such change constitutes a challenge. This creates a feedback loop. Any change has a cascading effect as others adjust their own strategy in response. Thus, from the perspective of an individual learner, the problem is one of adapting to a sequence of challenges, i.e. an autocurriulum. Innovation occurs when following an autocurriculum leads implementation learning to escape local optima where it would otherwise have been trapped indefinitely.
Empirical game theoretic techniques (Walsh et al., 2002; Wellman, 2006) may be used to analyze behavior in terms of its strategic properties. Instead of assuming game rules are known a priori, these methods work backwards from outcome data to deduce properties of the game being played. For example, Tuyls et al. (2018) showed strategic intransitivities between AlphaGo versions and Zen, the previous state-of-the-art Go engine. A similar approach was taken for social dilemmas in Leibo et al. (2017). That work classified learned implementation policies by strategic properties such as “aggressiveness”. (Perolat et al., 2017; Hughes et al., 2018) extended its approach beyond the two-player case to analyze the strategic incentives underlying policies learned by reinforcement learning in common pool resource appropriation and public goods scenarios.
Empirical game theoretic techniques are not only useful for data analysis, they also formed a critical part of one recent general-purpose algorithm for multi-agent reinforcement learning called Policy Space Response Oracles (Lanctot et al., 2017). It works by incrementally building up the full normal form game table by iteratively adding to the table a best response to the mixed strategy equilibrium predicted for the table’s previous state.
However, there is no guarantee that novel challenges will continue to be generated in this way. Consider a game with intransitive strategic dynamics like rock-paper-scissors (Singh et al., 2000; Tuyls and Nowé, 2005). The dynamics of evolving populations with incentives described by such games are often oscillatory (e.g. Gilpin (1975)). That is, even though specific implementations may be different with each repetition, the same underlying challenges are continually repeated. This is an autocurriculum that endlessly chases its own tail, never breaking away to incentivize new innovations.
When do exogenous autocurricula break out of cycles and continually generate increasingly clever innovations? This question has been studied in multi-agent reinforcement learning in the framework of self-play algorithms for two-player zero-sum games. The idea behind this family of algorithms is that by continually training an adaptive unit to defeat past versions of itself it can always be paired with a partner of the appropriate difficulty, neither too strong nor too weak. Self-play ensures that the adaptive unit learns to exploit its own errors, thereby challenging itself to correct them the next time around. The algorithm TD-Gammon, which was the first to play competitively with human experts in the game of Backgammon, was an early example of this approach (Tesauro, 1995). Self-play remains a prominent approach in recent work. For example, Bansal et al. (2018) applied it to a 3D sumo wrestling game with simulated physics and Jaderberg et al. (2018) applied it to an egocentrically-viewed team Capture-the-flag game based on the Quake game engine.
However, in designing self-play algorithms, care must be taken to prevent forgetting of past policies. If old policies are forgotten, then a newer generation may become unable to defeat an older generation, creating an opening for long-extinct traits to re-appear in the future (Samothrakis et al., 2013). Thus forgetting may cause the induced autocurriculum to double back onto itself, just like it does in the rock-paper-scissors case, preventing the productive accumulation of new innovations. In practice, successful self-play algorithms generally play not just against the latest (and strongest) policy, but also against as large and diverse as possible a set of older policies (Lanctot et al., 2017).
In games with a small number of possible strategies, an exogenous autocurriculum is expected to approach a Nash equilibrium (Gintis, 2000). However, in more open games, or those with a huge but still finite space of possible strategies, like Go or Chess, then self-play really does seem to be able to continually generate novel innovations. For example, AlphaGo and its Chess/Shogi/Go-playing variant, AlphaZero (Silver et al., 2016, 2017, 2018) are based on self-play. Starting with adaptive policy and value estimators for game positions, they use Monte Carlo tree search to improve the current policy, then learn better estimators from the generated games. Interestingly, these algorithms show that some kinds of forgetting are not always harmful. Sometimes innovations are discovered at one point in the training process, and later on discarded in favor of others. AlphaGo Zero, for example, rediscovered several patterns known from human Go expertise called joseki, but some of them were discarded in favour of new variations later on in training (Silver et al., 2017). A similar phenomenon was observed in AlphaZero: its preferences towards certain Chess openings fluctuated in time; the most frequently played openings at certain intermediate points in training were no longer seen in its later stages (Silver et al., 2018). Presumably the algorithm discovered that the discarded strategies are suboptimal. Sometimes it goes beyond our human capacity to understand what it has found. For instance, AlphaZero, in Chess, makes surprising (to humans) sacrifices to gain positional advantage, a development that is now impacting human professional level Chess (Kasparov, 2018).
So far we have only discussed exogenous autocurricula in the context of two-player zero sum games. A recent paper describing an algorithm called Malthusian reinforcement learning considered them in more general settings (Leibo et al., 2019). Malthusian reinforcement learning extends self-play to allow for variable numbers of players to appear in each episode. Subpopulation sizes grow proportionally to their success. In games with limited resources this demographic expansion creates additional competitive pressure. That is, it induces an exogenous autocurriculum. Whenever a successful policy arises at any population size, its own success ensures that population will increase in the future. Thus the Malthusian reinforcement learning algorithm generates a continually changing adaptive landscape which functions to perturb old solutions that have grown too comfortable and thereby driving it to escape poor local optima where state-of-the-art single-agent methods cannot avoid becoming stuck.
Exogenous autocurricula also appear in some evolutionary models of human intelligence. In one theory, the main selection pressure on intelligence in human evolution is posited to be the need to manipulate others within the social group in order to climb a dominance hierarchy (Humphrey, 1976). Increasingly clever social partners motivate the need to evolve still greater manipulating intelligence, and so on, increasing intelligence up to the point where brain size could no longer expand for anatomical reasons (Byrne, 1996; Dunbar and Shultz, 2017). On the other hand, there is more to being human than competition. As we will see in the next section, the challenges of organizing collective action also yield autocurricula that may have structured the evolution of human cognitive abilities and motivated significant innovations.
Endogenous Challenges Generate Autocurricula
Autocurricula may emerge on any level of the hierarchy of adaptive units. When a level is atomic (indivisible), only exogenous challenges are possible. On all other levels, the adaptive units are made up of adaptive subunits. In such cases adaptation may also be driven in response to endogenous challenges to the collective’s integrity. For example, a collective-level adaptive unit will generally function best when it has suppressed most competition between its component subunits. In multi-cellular organisms, that suppression sometimes breaks down, freeing somatic cells to behave in their own short-sighted selfish interest (Frank and Nowak, 2004; Rankin et al., 2007). Cancerous cells often behave like unicellular organisms, even reverting to less efficient fermentation-based metabolism (the Warburg effect) (Vander Heiden et al., 2009) and activating other ancient cellular functions conserved in modern unicellular organisms (Trigos et al., 2018). Similar breakdowns of cooperation can occur on the level of a society. For example, eusocial insect colonies are vulnerable to exploitation by renegade worker reproduction (Beekman and Oldroyd, 2008).
Such situations are social dilemmas. They expose tensions between individual and collective rationality (Rapoport, 1974). One particularly well-studied type of social dilemma is called common-pool resource appropriation (Ostrom, 1990). For a common-pool resource like a common grazing pasture, fishery, or irrigation system, it is difficult or impossible for individuals to exclude one another’s access. But whenever an individual obtains a benefit from such a common-pool resource, the remaining amount available for appropriation by others is at least somewhat diminished. If each individual’s marginal benefit of appropriation exceeds their share of the cost of further depletion, then they are predicted to continue their appropriation until the resource becomes degraded. This situation is called the tragedy of the commons (Hardin, 1968; Ostrom, 1990). It is impossible for an individual acting unilaterally to escape this fate; since even if one were to restrain their appropriation, the effect would be too small to make a difference. Thus individual-level innovation is not sufficient to meet this challenge. Any innovation that resolves a social dilemma must involve changing the behavior of a critical fraction of the participants (Schelling, 1973).
One way to effect such a change in the joint behavior of many individuals is to originate an “institution” (Ostrom, 1990, 2005; Greif, 2006): a system of rules, norms, or beliefs. Institutions may structure individual-level adaptive processes to ensure the group as a whole achieves a socially beneficial outcome. They may be regarded as collective-level innovations. For example, consider an institution whereby individuals who over-exploit the common pool resource are sanctioned by the group. This institution changes the individual incentive structure such that over-exploiting is no longer the dominant strategy. We have seen hints of emergent institutions in recent multi-agent reinforcement learning models of common-pool resource appropriation situations (Leibo et al., 2017; Perolat et al., 2017; Hughes et al., 2018). For example. an institution for generating cooperation on the collective-level emerged in Hughes et al. (2018) when certain agents learned to sanction over-exploiters, effectively policing their behavior.
Box 2. No-Free-Lunch in Social Dilemmas
The literature contains several models that suggest that higher order social dilemmas can be evaded in various ways. Here we show that they depend on unrealistic assumptions, thereby sustaining the present argument that the no-free-lunch property of social dilemmas cannot generally be avoided.
For example, some models depend on asymmetries between altruistic cooperation and altruistic punishment (Boyd et al., 2003). This may hold in some situations, especially when the cost of monitoring for infractions is small, e.g. in agricultural land use. But it does not hold when monitoring costs are large, as in many real-world common-pool resource situations (Ostrom, 1990).
Other models depend on coordination of large numbers of individuals engaging in altruistic punishment (Boyd et al., 2010), but they do not take into account the costs of such coordination. Several of the case studies of effective community-level resource management described by Ostrom show that communities are willing to invest in complex and relatively costly institutional mechanisms for ensuring this coordination is effective so that sanctioning may be deemed legitimate by the group and no individual must bear the brunt of the cost (Ostrom, 1990).
Another intriguing idea is to link reputation across tasks (Panchanathan and Boyd, 2004)
. However, these mechanisms substantially increase pressure on institutions for assigning and communicating reputations. Thus they give rise to new attack vectors for the unscrupulous. Agents may try to cheat by finding ways to falsely inflate their reputations.
While these arguments may explain some instances of cooperation, especially when the costs of monitoring for infractions are low, they are insufficient to explain away the no-free-lunch principle that generates higher order social dilemmas.
A social dilemma may be resolved via the emergence of an institution that systematically changes payoffs experienced by adaptive units so as to eliminate socially deficient equilibria or nudge learning dynamics toward better equilibria. However, maintaining the institution itself still depends on interactions of those same participants. In many cases this yields a second order social dilemma because each individual would prefer others to shoulder a greater share of that burden (Axelrod, 1986; Yamagishi, 1988; Heckathorn, 1989). This is called the “second-order free rider problem”. As predicted by these models, there is evidence that pre-state societies sustain social norms that disapprove of second-order free riding, e.g. Mathew (2017). Second-order social dilemmas may themselves be resolved via the emergence of higher order institutions which, in turn, create their own still higher level successor dilemmas (Ostrom, 2000). Indeed we can say that social dilemma situations have a kind of “no-free lunch” property: once you resolve a social dilemma in one place then another one crops up somewhere else (see Box 2). A society may resolve a social dilemma by hiring watchmen, but then who watches the watchmen? These dynamics may generate a sequence of endogenous challenges growing steadily in scope and complexity, i.e. an autocurriculum.
Just as atomic individuals participate in social interactions with one another, communities interact with peer communities as competitors or allies. Exogenous challenges may also arise on the collective level. Their effects reverberate down the hierarchy, tending to resolve endogenous challenges by aligning the incentives of lower level entities (Maynard Smith and Szathmary, 1997; Henrich, 2004a; Wilson and Wilson, 2007). For example, eusocial insect colonies compete with other conspecific colonies. Those colonies that are better able to maintain internal cooperation, e.g., by establishing reproductive division of labor, are more likely to be successful in colony-level competition (Nowak et al., 2010). In this view, communities are treated much like atomic individuals. Just as atomic individuals are understood to seek to maximize the utility (food, shelter, mates, etc) they can obtain from social interaction, higher order adaptive units act to optimize a range of different higher order utility concepts. For example, populations of ant colonies optimize fitness in the sense of multi-level selection (Okasha, 2005), while human corporations optimize culturally-determined constructs like “shareholder value”. Analogous to nervous systems, communities weigh their options via decision-making institutions like parliaments and markets.
Accumulating Autocurricula and Human Uniqueness
In this paper we argued that interactions among multiple adaptive units across levels of the biological hierarchy give rise to sequences of challenges called autocurricula that perturb adaptive landscapes. This enables adaptive units to discover innovations by continually adapting to changing circumstances—exploration-by-exploitation (see Outstanding Questions). Might this mechanism be the key to solving the problem problem? Will autocurricula generate enough adaptive diversity? Perhaps not. Recall that autocurricula may be cyclic, repeatedly learning and unlearning the same information, innovations never accumulating or building on one another. Moreover, there is nothing about these mechanisms that suggests they do not apply equally strongly to non-human animals. We think the solution is as follows. Autocurricula do indeed exist throughout animal evolution. However, humans are unique among animals in their exceptionally long cultural memory. This allows intransitive cycles to be avoided, thereby promoting cumulative accumulation of innovation after innovation.
Why did this same accumulation not occur in other ape species? We highlight two possibilities, based on the structure of feedback loops within human societies, driven respectively by exogenous and endogenous challenges to group integrity. Both may be seen as the combination of a challenge-based loop, and a ratchet loop that serves the purpose of accumulating and distilling beneficial innovations.
The cumulative culture loop (Fig. 2-A). A growing population puts stress on human societies since more individuals must vie for access to the same limited resources. The resulting competitive pressure may give rise to exogenous challenges to each individual, motivating new innovation. However, since humans are adept social learners, any innovation that promotes survival will tend to quickly spread through the population and thereby increase population sizes in the future along the lines suggested by models like Enquist et al. (2011). There is a significant literature demonstrating that possibly uniquely human high fidelity social transmission of cultural elements allows innovations to build upon one another and increase in complexity over time (Boyd et al., 2011; Dean et al., 2012; Boyd et al., 2013; Muthukrishna and Henrich, 2016). Moreover, archaeological evidence and computational models suggest that larger populations can afford greater protection from random forgetting of cultural elements due to chance events like the death of a single master artisan, thus allowing a more significant opportunity for cumulative cultural evolution (Henrich, 2004b). Indeed, cultural evolution is made more effective by increasing the population size and increasing the population size increases the effectiveness of cultural learning. This is a feedback loop that, once started, could potentially increase until the population becomes limited by its environment’s carrying capacity in some way that further innovation cannot transcend.
The self-domestication loop (Fig. 2-B). A growing population need not lead to scarcity, provided that resource production is efficient. However, it does provide more opportunity for norm violators to free-ride on the contributions of others (Carpenter, 2007). One norm-violating behavior is reactive aggression, particularly in young males. Coordinated punishment for aggression in small-scale societies often takes the form of capital punishment (Boehm, 2011). The result is genetic evolution that gradually reduces the prevalence of aggressive individuals. This process has been called self-domestication (Wrangham, 2018) because it is similar to the selection process that may have been applied in creating domestic dogs from wolves, now applied by a species to itself. Self-domestication has the effect of increasing tolerance for non-kin because it reduces the likelihood of encountering aggressive individuals. This may have created opportunities for improving communication abilities, and ultimately, language. Conversely, improving communication between individuals in a society improves the effectiveness of institutions for sanctioning norm violators by facilitating coordinated punishment (Bochet et al., 2006; Janssen et al., 2010, 2014). Language also improves the accuracy of reputation information, e.g. as conveyed by gossip, informing the decision of which individuals deserve punishment (Nowak and Sigmund, 1998; Rockenbach and Milinski, 2006). Hence, the resultant feedback loop may accumulate institutions indefinitely.
In this paper we identified multi-agent interactions as key drivers of sustained innovation and, perhaps, increases in intelligence over the course of human evolution. As a consequence, a research program based only on replicating individual human cognitive abilities, e.g. attention, memory, planning, etc, is likely incomplete. It seems that intelligence researchers would do well to pay more attention to the ways in which multi-agent dynamics may structure both evolution and learning.
The authors would like to thank Edgar Duenez-Guzman, David Balduzzi, Aliya Amirova, Greg Wayne, Peter Sunehag, Joyce Xu, Martin Chadwick, Richard Everett, Vlad Firiou and Raphael Koster for very helpful comments and discussions during the drafting of this article. In addition, the first author would like to thank all the speakers, organizers, and attendees of the 2014 “Are there limits to evolution?” workshop at Cambridge where much of the thought process leading eventually to this article was first hatched.
- Asmuth et al. (2008) John Asmuth, Michael L Littman, and Robert Zinkov. Potential-based shaping in model-based reinforcement learning. In AAAI, pages 604–609, 2008.
- Axelrod (1986) Robert Axelrod. An evolutionary approach to norms. American political science review, 80(4):1095–1111, 1986.
- Bansal et al. (2018) Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representations, 2018.
- Beattie et al. (2016) Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03801, 2016.
- Beekman and Oldroyd (2008) Madeleine Beekman and Benjamin P Oldroyd. When workers disunite: intraspecific parasitism by eusocial bees. Annual review of entomology, 53, 2008.
- Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM, 2009.
- Bloembergen et al. (2015) Daan Bloembergen, Karl Tuyls, Daniel Hennes, and Michael Kaisers. Evolutionary dynamics of multi-agent learning: A survey. J. Artif. Intell. Res. (JAIR), 53:659–697, 2015.
- Bochet et al. (2006) Olivier Bochet, Talbot Page, and Louis Putterman. Communication and punishment in voluntary contribution experiments. Journal of Economic Behavior & Organization, 60(1):11–26, 2006.
- Boehm (2011) Christopher Boehm. Retaliatory violence in human prehistory. The British Journal of Criminology, 51(3):518–534, 2011.
- Börgers and Sarin (1997) Tilman Börgers and Rajiv Sarin. Learning through reinforcement and replicator dynamics. Journal of Economic Theory, 77(1):1–14, 1997.
- Boyd and Richerson (1988) Robert Boyd and Peter J Richerson. Culture and the evolutionary process. University of Chicago press, 1988.
- Boyd et al. (2003) Robert Boyd, Herbert Gintis, Samuel Bowles, and Peter J Richerson. The evolution of altruistic punishment. Proceedings of the National Academy of Sciences, 100(6):3531–3535, 2003.
- Boyd et al. (2010) Robert Boyd, Herbert Gintis, and Samuel Bowles. Coordinated punishment of defectors sustains cooperation and can proliferate when rare. Science, 328(5978):617–620, 2010.
- Boyd et al. (2011) Robert Boyd, Peter J Richerson, and Joseph Henrich. The cultural niche: Why social learning is essential for human adaptation. Proceedings of the National Academy of Sciences, 108(Supplement 2):10918–10925, 2011.
- Boyd et al. (2013) Robert Boyd, Peter J Richerson, and Joseph Henrich. The cultural evolution of technology: Facts and theories. Cultural evolution: society, technology, language, and religion, 12:119, 2013.
- Byrne (1996) Richard W Byrne. Machiavellian intelligence. Evolutionary Anthropology: Issues, News, and Reviews: Issues, News, and Reviews, 5(5):172–180, 1996.
- Carpenter (2007) Jeffrey Carpenter. Punishing free-riders: How group size affects mutual monitoring and the provision of public goods. Games and Economic Behavior, 60(1):31–51, 2007.
- Clark (2008) Gregory Clark. A farewell to alms: a brief economic history of the world, volume 27. Princeton University Press, 2008.
- Czarnecki et al. (2018) W. M. Czarnecki, S. M. Jayakumar, M. Jaderberg, L. Hasenclever, Y. Whye Teh, S. Osindero, N. Heess, and R. Pascanu. Mix and Match - Agent Curricula for Reinforcement Learning. ArXiv e-prints, June 2018.
- Dawkins and Krebs (1979) Richard Dawkins and John Richard Krebs. Arms races between and within species. Proc. R. Soc. Lond. B, 205(1161):489–511, 1979.
- Dean et al. (2012) Lewis G Dean, Rachel L Kendal, Steven J Schapiro, Bernard Thierry, and Kevin N Laland. Identification of the social and cognitive processes underlying human cumulative culture. Science, 335(6072):1114–1118, 2012.
- Dunbar and Shultz (2017) RIM Dunbar and Susanne Shultz. Why are there so many explanations for primate brain evolution? Phil. Trans. R. Soc. B, 372(1727):20160244, 2017.
- Enquist et al. (2011) Magnus Enquist, Stefano Ghirlanda, and Kimmo Eriksson. Modelling the evolution and diversity of cumulative culture. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 366(1563):412–423, 2011.
- Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.
- Fairbairn (1995) John Fairbairn. Go in ancient china. Go Base, 1995.
- Frank and Nowak (2004) Steven A Frank and Martin A Nowak. Problems of somatic mutation and cancer. Bioessays, 26(3):291–299, 2004.
- Gerber et al. (1999) Christian Gerber, Jörg Siekmann, and Gero Vierke. Holonic multi-agent systems. 1999.
- Gilpin (1975) Michael E Gilpin. Limit cycles in competition communities. The American Naturalist, 109(965):51–60, 1975.
- Gintis (2000) Herbert Gintis. Game theory evolving: A problem-centered introduction to modeling strategic behavior. Princeton university press, 2000.
- Greif (2006) A. Greif. Institutions and the Path to the Modern Economy: Lessons from Medieval Trade. Political Economy of Institutions and Decisions. Cambridge University Press, 2006. ISBN 9781139447065.
- Hardin (1968) Garrett Hardin. The tragedy of the commons. Science, 162(3859):1243–1248, 1968.
- Heckathorn (1989) Douglas D Heckathorn. Collective action and the second-order free-rider problem. Rationality and Society, 1(1):78–100, 1989.
- Henrich (2004a) Joseph Henrich. Cultural group selection, coevolutionary processes and large-scale cooperation. 53:3–35, 02 2004a.
- Henrich (2004b) Joseph Henrich. Demography and cultural evolution: how adaptive cultural processes can produce maladaptive losses—the tasmanian case. American Antiquity, 69(2):197–214, 2004b.
- Hernández-Orallo (2017) José Hernández-Orallo. The measure of all minds: evaluating natural and artificial intelligence. Cambridge University Press, 2017.
- Hessel et al. (2018) Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. arXiv preprint arXiv:1809.04474, 2018.
- Hughes et al. (2018) Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Information Processing Systems, pages 3330–3340, 2018.
- Humphrey (1976) Nicholas K Humphrey. The social function of intellect. In Growing points in ethology, pages 303–317. Cambridge University Press, 1976.
- Jaderberg et al. (2018) Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv preprint arXiv:1807.01281, 2018.
- Janssen et al. (2014) Marco Janssen, Madeline Tyson, and Allen Lee. The effect of constrained communication and limited information in governing a common resource. International Journal of the Commons, 8(2), 2014.
- Janssen et al. (2010) Marco A Janssen, Robert Holahan, Allen Lee, and Elinor Ostrom. Lab experiments for the study of social-ecological systems. Science, 328(5978):613–617, 2010.
- Kasparov (2018) G Kasparov. Chess, a drosophila of reasoning. Science (New York, NY), 362(6419):1087, 2018.
- Kasting and Siefert (2002) James F Kasting and Janet L Siefert. Life and the evolution of earth’s atmosphere. Science, 296(5570):1066–1068, 2002.
- Kavanagh et al. (2018) Patrick H Kavanagh, Bruno Vilela, Hannah J Haynie, Ty Tuff, Matheus Lima-Ribeiro, Russell D Gray, Carlos A Botero, and Michael C Gavin. Hindcasting global population densities reveals forces enabling the origin of agriculture. Nature Human Behaviour, page 1, 2018.
- Kempka et al. (2016) Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In Computational Intelligence and Games (CIG), 2016 IEEE Conference on, pages 1–8. IEEE, 2016.
- Lanctot et al. (2017) Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Perolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
- Legg and Hutter (2007) Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4):391–444, 2007.
- Leibo et al. (2017) Joel Z. Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS 2017), Sao Paulo, Brazil, 2017.
- Leibo et al. (2018) Joel Z Leibo, Cyprien de Masson d’Autume, Daniel Zoran, David Amos, Charles Beattie, Keith Anderson, Antonio García Castañeda, Manuel Sanchez, Simon Green, Audrunas Gruslys, et al. Psychlab: a psychology laboratory for deep reinforcement learning agents. arXiv preprint arXiv:1801.08116, 2018.
- Leibo et al. (2019) Joel Z Leibo, Julien Perolat, Edward Hughes, Steven Wheelwright, Adam H Marblestone, Edgar Duéñez-Guzmán, Peter Sunehag, Iain Dunning, and Thore Graepel. Malthusian reinforcement learning. In Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (AA-MAS 2019), Montreal, Canada, 2019.
- Mathew (2017) Sarah Mathew. How the second-order free rider problem is solved in a small-scale society. American Economic Review, 107(5):578–81, 2017.
- Maynard Smith and Szathmary (1997) John Maynard Smith and Eors Szathmary. The major transitions in evolution. Oxford University Press, 1997.
- Mirowski et al. (2016) Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673, 2016.
- Muthukrishna and Henrich (2016) Michael Muthukrishna and Joseph Henrich. Innovation in the collective brain. Phil. Trans. R. Soc. B, 371(1690):20150192, 2016.
- Narvekar (2017) Sanmit Narvekar. Curriculum learning in reinforcement learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 5195–5196, 2017. doi: 10.24963/ijcai.2017/757.
- North (1991) Douglass C North. Institutions. Journal of economic perspectives, 5(1):97–112, 1991.
- Nowak and Sigmund (1998) Martin A Nowak and Karl Sigmund. Evolution of indirect reciprocity by image scoring. Nature, 393(6685):573, 1998.
- Nowak et al. (2010) Martin A Nowak, Corina E Tarnita, and Edward O Wilson. The evolution of eusociality. Nature, 466(7310):1057, 2010.
- Okasha (2005) Samir Okasha. Multilevel selection and the major transitions in evolution. Philosophy of science, 72(5):1013–1025, 2005.
- Ostrom (1990) Elinor Ostrom. Governing the Commons: The Evolution of Institutions for Collective Action. Cambridge University Press, 1990.
- Ostrom (2000) Elinor Ostrom. Collective action and the evolution of social norms. Journal of economic perspectives, 14(3):137–158, 2000.
- Ostrom (2005) Elinor Ostrom. Understanding institutional diversity. Princeton University Press Princeton, 2005.
- Panchanathan and Boyd (2004) Karthik Panchanathan and Robert Boyd. Indirect reciprocity can stabilize cooperation without the second-order free rider problem. Nature, 432(7016):499–502, 2004.
- Perolat et al. (2017) Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation. In Advances in Neural Information Processing Systems, pages 3643–3652, 2017.
- Pinker (2011) Steven Pinker. The better angels of our nature: The decline of violence in history and its causes. Penguin UK, 2011.
- Rankin et al. (2007) Daniel J Rankin, Katja Bargum, and Hanna Kokko. The tragedy of the commons in evolutionary biology. Trends in ecology & evolution, 22(12):643–651, 2007.
- Rapoport (1974) Anatol Rapoport. Prisoner’s dilemma—recollections and observations. In Game Theory as a Theory of a Conflict Resolution, pages 17–34. Springer, 1974.
- Reader and Laland (2003) S.M. Reader and K.N. Laland. Animal Innovation. Oxford University Press, 2003. ISBN 9780198526223.
- Rockenbach and Milinski (2006) Bettina Rockenbach and Manfred Milinski. The efficient interaction of indirect reciprocity and costly punishment. Nature, 444(7120):718, 2006.
Samothrakis et al. (2013)
Spyridon Samothrakis, Simon Lucas, Thomas Philip Runarsson, and David Robles.
Coevolving gameplaying agents: Measuring performance and
IEEE Transactions on Evolutionary Computation, 17(2):213–226, 2013.
- Savinov et al. (2018) Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274, 2018.
- Schelling (1973) Thomas C Schelling. Hockey helmets, concealed weapons, and daylight saving: A study of binary choices with externalities. Journal of Conflict resolution, 17(3):381–428, 1973.
- Schluter (2000) Dolph Schluter. The ecology of adaptive radiation. Oxford University Press, 2000.
Silver et al. (2016)
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George
van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda
Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal
Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray
Kavukcuoglu, Thore Graepel, and Demis Hassabis.
Mastering the game of Go with deep neural networks and tree search.Nature, 529:484––489, 2016.
- Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Matthew Lai Lucas Baker, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354–359, 2017.
- Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
- Singh et al. (2000) Satinder P. Singh, Michael J. Kearns, and Yishay Mansour. Nash convergence of gradient dynamics in general-sum games. In Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, UAI ’00, pages 541–548, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1-55860-709-9.
- Smith and Price (1973) J Maynard Smith and George R Price. The logic of animal conflict. Nature, 246(5427):15, 1973.
- Such et al. (2017) Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567, 2017.
- Tesauro (1995) G. Tesauro. Temporal difference learning and TD-gammon. Commun. ACM, 38(3):58–68, March 1995.
- Trigos et al. (2018) Anna S Trigos, Richard B Pearson, Anthony T Papenfuss, and David L Goode. How the evolution of multicellularity set the stage for cancer. British Journal of Cancer, 118(2):145, 2018.
Tuyls and Nowé (2005)
K. Tuyls and A. Nowé.
Evolutionary game theory and multi-agent reinforcement learning.
The Knowledge Engineering Review, 20(1):63–90, 2005.
- Tuyls et al. (2018) Karl Tuyls, Julien Perolat, Marc Lanctot, Joel Z Leibo, and Thore Graepel. A generalised method for empirical game theoretic analysis. arXiv preprint arXiv:1803.06376, 2018.
- Vander Heiden et al. (2009) Matthew G Vander Heiden, Lewis C Cantley, and Craig B Thompson. Understanding the warburg effect: the metabolic requirements of cell proliferation. science, 324(5930):1029–1033, 2009.
- Walsh et al. (2002) William E Walsh, Rajarshi Das, Gerald Tesauro, and Jeffrey O Kephart. Analyzing complex strategic interactions in multi-agent systems. In AAAI-02 Workshop on Game-Theoretic and Decision-Theoretic Agents, pages 109–118, 2002.
- Wayne et al. (2018) Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska-Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, et al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint arXiv:1803.10760, 2018.
- Wellman (2006) Michael P Wellman. Methods for empirical game-theoretic analysis. In AAAI, pages 1552–1556, 2006.
- Wilson and Wilson (2007) David Sloan Wilson and Edward O Wilson. Rethinking the theoretical foundation of sociobiology. The Quarterly review of biology, 82(4):327–348, 2007.
- Wrangham (2018) Richard W Wrangham. Two types of aggression in human evolution. Proceedings of the National Academy of Sciences, 115(2):245–253, 2018.
- Yamagishi (1988) Toshio Yamagishi. Seriousness of social dilemmas and the provision of a sanctioning system. Social psychology quarterly, pages 32–42, 1988.
- Zaremba and Sutskever (2015) Wojciech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines. arXiv preprint arXiv:1505.00521, 2015.