Modelling the Safety and Surveillance of the AI Race

07/26/2019
by   The Anh Han, et al.
Teesside University
2

Innovation, creativity, and competition are some of the fundamental underlying forces driving the advances in Artificial Intelligence (AI). This race for technological supremacy creates a complex ecology of choices that may lead to negative consequences, in particular, when ethical and safety procedures are underestimated or even ignored. Here we resort to a novel game theoretical framework to describe the ongoing AI bidding war, also allowing for the identification of procedures on how to influence this race to achieve desirable outcomes. By exploring the similarities between the ongoing competition in AI and evolutionary systems, we show that the timelines in which AI supremacy can be achieved play a crucial role for the evolution of safety prone behaviour and whether influencing procedures are required. When this supremacy can be achieved in a short term (near AI), the significant advantage gained from winning a race leads to the dominance of those who completely ignore the safety precautions to gain extra speed, rendering of the presence of reciprocal behavior irrelevant. On the other hand, when such a supremacy is a distant future, reciprocating on others' safety behaviour provides in itself an efficient solution, even when monitoring of unsafe development is hard. Our results suggest under what conditions AI safety behaviour requires additional supporting procedures and provide a basic framework to model them.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 9

page 25

page 26

page 27

page 28

page 29

page 32

page 34

10/01/2020

Mediating Artificial Intelligence Developments through Negative and Positive Incentives

The field of Artificial Intelligence (AI) is going through a period of g...
12/30/2020

AI Development Race Can Be Mediated on Heterogeneous Networks

The field of Artificial Intelligence (AI) has been introducing a certain...
04/08/2021

Voluntary safety commitments provide an escape from over-regulation in AI development

With the introduction of Artificial Intelligence (AI) and related techno...
04/17/2021

Influence of Vaporfly shoe on sub-2 hour marathon and other top running performances

In 2019, Eliud Chipgoke ran a sub-two hour marathon in an unofficial rac...
06/10/2021

Hard Choices in Artificial Intelligence

As AI systems are integrated into high stakes social domains, researcher...
11/26/2020

Transdisciplinary AI Observatory – Retrospective Analyses and Future-Oriented Contradistinctions

In the last years, AI safety gained international recognition in the lig...
04/02/2019

Augmented Utilitarianism for AGI Safety

In the light of ongoing progresses of research on artificial intelligent...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Abstract

Innovation, creativity, and competition are some of the fundamental underlying forces driving the advances in Artificial Intelligence (AI). This race for technological supremacy creates a complex ecology of choices that may lead to negative consequences, in particular, when ethical and safety procedures are underestimated or even ignored. Here we resort to a novel game theoretical framework to describe the ongoing AI bidding war, also allowing for the identification of procedures on how to influence this race to achieve desirable outcomes. By exploring the similarities between the ongoing competition in AI and evolutionary systems, we show that the timelines in which AI supremacy can be achieved play a crucial role for the evolution of safety prone behaviour and whether influencing procedures are required. When this supremacy can be achieved in a short term (near AI), the significant advantage gained from winning a race leads to the dominance of those who completely ignore the safety precautions to gain extra speed, rendering of the presence of reciprocal behavior irrelevant. On the other hand, when such a supremacy is a distant future, reciprocating on others’ safety behaviour provides in itself an efficient solution, even when monitoring of unsafe development is hard. Our results suggest under what conditions AI safety behaviour requires additional supporting procedures and provide a basic framework to model them.

Keywords:

AI race modelling, emergence, cooperation, evolutionary game theory.

1 Introduction

Interest in Artificial intelligence (AI) has exploded in academia and businesses in the last few years. This excitement is, on one hand, due to a series of superhuman performances 31, 9, 30, 10 which have been exhibited. Although mostly successful in highly specialised tasks, exceeding human ability and precision, these AI success stories appear often in the imagination of the general public as Hollywood-like Artificial General Intelligence (AGI), able to perform a broad set of intellectual tasks while continuously improving itself. Large scale surveys show that AI researchers expect that AI systems will eventually reach and then exceed human-level performance in many of the surveyed tasks, although the timelines are quite diverse 32, 16. On the other hand, the excitement is promoted by business leaders as they anticipate important gains from turning their previously idle data into active assets within business plans 27. All these (un)announced business and political ambitions indicate that an AI race or bidding war has been triggered 1, 11, 2, where stake-holders in private and public sectors are competing to be the first to cross the finish line and hence the leader in the development and deployment of powerful, transformative AI 3, 6, 7, 11.

Irrespectively of the anticipated benefits, many actors have urged for due diligence as i) these AI systems can also be employed for more nefarious activities, e.g. espionage and cyberterrorism 34 and ii) when trying to be the first/best then some ethical consequences as well as safety procedures may be underestimated or even ignored 3, 11, notwithstanding the issue that certain claims about achieving AGI may be overly optimistic or just oversold. These concerns are highlighted by the many letters of scientists against the use of AI in military applications 24, 25 and the proclamations on ethical use of AI in the world 12, 33, 28.

While potential AI disaster scenarios are many 32, 3, 26, the uncertainties in accurately predicting these risks and outcomes are high 4. As insufficient data is available, the essential approach to clearly grasp what can be expected is to create models, i.e. dynamic descriptions of the key features (of parts thereof) of this race in order to understand what outcomes are possible under certain conditions and what crucial factors play an essential role. The Future of Life Institute (FLI) as well as other similar institutes have therefore launched open calls for projects to foster research on the topic of AI safety and the exploration of the AI race dynamics we are currently witnessing. This manuscript provides a baseline model established within one of the FLI awarded projects in 2018, discussing under what conditions unsafe versus safe AI developments may lead to disastrous outcomes, in races involving two or many more participants. This baseline model resorts to the framework of evolutionary game theory 18, 29 to study the dynamics and emergent behaviours within an AI development race.

Concretely, the model assumes that in order to achieve AI supremacy (AIS) in a domain , a number of development steps or rounds () are required. Distinct values of capture different regimes of AIS: in the limit of small , AIS can be expected to happen in the near future (near AIS regime) while when is large, AIS will only be achieved far away in time (distant AIS regime). Large-scale surveys and analysis of AI experts on their beliefs and predictions about progress in AI suggest that the perceived timeline for AIS is highly diverse across domains and regions 4, 16. Because this is a race, each participant acts by herself during each step in order to reach the target and differs in the speed () with which they can complete each of the subtasks at each round. A fast participant will, therefore, reap benefits () at each step, winning the ultimate prize () once it carries out the final step achieving AIS in the domain . If multiple participants reach the end of intermediate steps or the final target at the same time they share the benefits and . Yet, one can also assume that higher also implies that some ethical/safety procedures might be ignored. It takes time and effort to comply to precautionary requirements or acquire ethical approvals. Following a safe development process is thus not only more costly (), it also results in a slower development speed. One can therefore consider that participants in the AI race that act safely (SAFE) pay a cost , which is not paid by participants that ignore ethical/safety procedures (UNSAFE) and ii) the speed of development of UNSAFE participants is faster (), compared to the speed of SAFE participants being normalised to . So essentially a SAFE player needs rounds to complete the task, whereas an UNSAFE player will only need

. Yet, UNSAFE strategists may suffer a disaster, which is assumed here to increase with the number of times the safety requirements have been skirted. The probability that disaster occurs is denoted by

and assumed to increase linearly with the frequency the participant violates the safety precautions. For example, if a participant always plays SAFE then , while the participant that only follows it half of the time has a total probability of over all rounds. Moreover, if some sort of institutional or peer monitoring comes into place, we assume that with some probability those playing UNSAFE might be found out and disclosed concerning their unsafe development and their products will therefore not be used, leading to 0 benefit for the current round.

Let us consider a population of size in which players engage in a pairwise or N-player race, where they can choose to consistently follow safety precautions (denoted by AS, the SAFE players) or completely ignore them (denoted by AU, the UNSAFE players). Additionally, we assume that, upon realising that UNSAFE players ignore safety precautions to gain a greater development speed leading to the wining of the prize (and a larger share of the intermediate benefit in each round, , especially in the regime of hard monitoring or low ), SAFE players might adopt the same course of actions to avoid further disadvantage. It is indeed observed that competing countries or companies might engage in such a safety-cutting corner behaviour in deploying unsafe AI to avoid falling behind 2. We therefore consider, in line with previous literature of repeated games 5, 29, an alternative strategy (denoted by CS, the conditionally safe strategy), which plays SAFE in the first round and then adopts the move its co-player used in the previous round. This so-called direct reciprocity strategy has been shown to promote cooperation in the context of repeated social dilemmas, outperforming consistently defective individuals 5, 29. In the following, we will examine, across different regimes of the AIS, under which conditions (for instance, regarding the disaster probability), safety behaviour should to be promoted or externally enforced. Similarly, we shall address when one should omit the safety precautions for a larger social welfare and when the benefits gained in doing so exceeds the disaster risk. Moreover, given the first-mover advantage of UNSAFE players in the race to AI supremacy (i.e., acquire ), we will examine whether (and under what regime of the AIS) conditional behaviours can still act as a promoting mechanism to achieve safety when required, or otherwise other mechanisms are needed. For the sake of presentation, we start by describing the pairwise race model and go on to describe the general -player () race results afterwards.

2 Results

Figure 1: Different regimes of AIS: when is small (near AIS) vs when is large (distant AIS). Panels (a) and (b) show the frequency of each strategy in a population of AS, AU and CS (). In the near AIS regime, AU dominates the population, while AS and CS outperform AU in the distant AIS regime. This observation is valid for sufficiently small , see panel (c) (). For a high risk probability of disaster occurring due to ignoring safety precautions (high ), AU has a low frequency in both regimes. Parameters: , , , , , population size, .

We calculate the long-term frequency of each possible behavioural composition of the population, the so-called stationary distribution (cf. Methods). The stochastic evolutionary dynamics of the population occurs in the presence of errors, both in terms of errors of imitation and of behavioural mutations, the latter representing an open exploration of the possible strategies 18, 29. Figure 1 shows the frequencies of the three strategies AS, AU and CS across a spectrum of regimes of AIS: i) near AIS: in this regime, AIS will be achieved after a limited number of development steps, making the ultimate prize of winning the race highly significant (i.e. ) ; ii) distant AIS (very large ): in this regime, AIS will not be achieved in a foreseeable future, making the ultimate prize of winning the race insignificant compared to the cost and benefit at each step of the race (i.e. ). We observe that in the near AIS regime, AU dominates the population whenever the probability that an AI disaster occurs due to unsafe development () is not too high (Figure 1c; in panels a and b, ). In the distant AI regime, all strategies are present in the population, where AS and CS are slightly more frequent (Figure 1a-b). When an AI disaster is more likely to occur due to unsafe developments (i.e. large , see Figure 1c), AU has a low frequency in both regimes. Moreover, AU frequency increases much more dramatically from high risk to low risk in the near AIS regime, compared to the distant AIS one (cf. Figure S6 in SI for other values of ). It implies that more efforts and care are needed in the near AIS regime, since that can dramatically change the safety outcome of the race. Below we elaborate on each regime of AIS in greater detail.

Near AIS regime: speedy development overcomes the risk

First of all, we describe the conditions under which a population of individuals closely following safety precautions actually has a greater social welfare or average payoff than that of a population of players never following safety precautions, that is, when . In the near AIS regime, it is equivalent to (see SI for the proof)

(1)

That is, when the risk probability of AI disaster occurring due to the omission of safety precautions is large enough compared to the gain of a greater development speed by doing so, safety development is the preferred collective outcome for the population. On the other hand, when this risk is shallow compared to the gain of omitting safety precautions, UNSAFE is the more beneficial collective outcome. It would be detrimental however, to prevent this outcome from emerging (i.e. over-regulation of AI development).

Figure 2: Near AIG regime. (a) Frequency of AU as a function of the speed gained, , and the probability of AI disaster occurring, , when ignoring safety. In general, we observe that when the risk probability is small, AU is dominant. The larger is, AU dominates for a larger range. Region (II): The two solid lines inside the plots indicate the boundaries where safety development is the preferred collective outcome but unsafe development is selected by evolution. Regions (I) and (III) indicate where safe (resp., unsafe) development is both the preferred collective outcome and the one selected by evolution. Panels (b) () and (c) (): transition probabilities and stationary distribution in a population of AS, AU, and CS, with . AU dominates in panel (c), corresponding to region (II), while AS and CS dominate in panel (b), corresponding to region (I). We only show the stronger directions. Parameters: , , , , , , population size, .

On the other hand, we found that both the safety complying strategies, AS and CS, are preferred over AU by natural selection (i.e. see risk-dominance analysis in SI) when

(2)

Thus, the two boundary conditions in Equations 1 and 2 divide the parameter space - into three regions, see Figure 2a: (I) when : safety development is both the preferred collective outcome and selected by evolution (see Figure 2b for an example: for the condition becomes ); (II) when : although it is more desirable to ensure safety development as the collective outcome, natural selection/social learning would nevertheless drive the population to the state where safety precaution is mostly ignored (see Figure 2c for an example: for the condition becomes ); (III) when , unsafe development is both the preferred collective outcome and the one selected by evolution. Numerical results (cf. Methods) in Figure 2 confirm this division of the regions. These observations are also robust for other intensities of selection, see Figure S4 (SI).

In regions (I) and (III), the preferred collective outcomes are selected by evolution. In the latter, a significant speed gained by unsafe development actually compensates for the risk due to ignoring safety precautions: taking risks (AI innovation) is better off because of high gain. Region (II) is the most important one to study as additional mechanisms are needed to promote safety behaviour against the unsafe one.

Note that the boundaries established in Equations 1 and 2 are applicable for both CS and AS when playing against AU. Thus, similar results are obtained if we consider a population of just two strategies AS and AU (cf. Figure S5 in SI). Adding CS does not change the overall outcome and conditions for safety development to be selected.

In short, we have seen that in the near AIS regime, conditionally safe behaviours cannot overcome the speedy development advantage gained by completely ignoring the safety precautions. This points to the fact that external interference such as institution incentives need to be established, in order to effectively regulate safety behaviour in this regime. Moreover, sufficient care needs to be put in place to avoid over-regulation preventing a beneficial extra-speedy development (in region (III)). It is noteworthy that our this result is robust when we consider the AI race with among teams (for all , see SI). The main difference when increasing the group size is that the upper bound of region (II) would increase. That is, unsafe behaviour is selected by natural selection for a larger range of the parameter space -. The reason is, the larger the group size, the greater the chance that there is at least one AU player in the group (with other AS and CS players), who would then win the development race.

Distant AIS regime: conditional behaviour prevails even under weak monitoring

Figure 3: Distant AIS regime. (a) Frequency of AU as a function of the probability of unsafe development being found out, , and the probability of AI disaster occurring , when the number of development steps to reach AIS is very large (). AU has a low frequency whenever or are sufficiently high. (b-c): transition probabilities and stationary distribution (). Against AU, AS performs better than CS when is large, which is reversed when is small. Parameters: , , , , , population size, .

When AIS is unachievable in the short term, the effect of increasing (from low to high risk) on the frequency of safe and unsafe behaviours is less dramatic than in the near AIS regime, see Figure 3. In general, all strategies are present, where the frequency of AU decreases as a function of and . In contrast to the near AIS regime, the conditionally safe strategy, CS, contributes significantly to enhancing the safety behaviour outcome. Indeed, CS outperforms AS when the probability of uncovering an unsafe development in each round is small (which is is reversed for larger ) (see Figure 1a-b; also Figure S3 in SI). That is, when monitoring of unsafe development is highly efficient (i.e. large ), it is best to follow closely the safety precautions to avoid AI disaster by all means, even when facing unsafe opponents. However, when this monitoring is not efficient, acting conditionally provides the more efficient solution to prevent unsafe behaviour as it can avoid being disadvantageous after the first round. These observations can also be studied analytically (see SI). Namely, we derive conditions under which AS and CS are selected over AU by natural selection, as well as when safety behaviour is the preferred collective outcome than the unsafe one (cf. Figure S1). For a greater efficiency of monitoring (the larger ) or a lower speed gained by omitting safety precaution (the smaller ), we show that a lower threshold for the disaster risk is required for those conditions to hold. Moreover, this threshold for AS is higher than that for CS when is small, which is reversed when it is large. As shown in the SI, all these observations remain valid if, instead of pairwise interactions, we consider a general -team AI race.

3 Discussion

Our results suggest that it is significantly more challenging to achieve safety behaviour in the regime where AI supremacy is achievable within a limited number of development steps (near AIS) than when it is only feasible at a distant future (distant AIS). In the former regime, the extra development speed gained by ignoring safety precautions gives the unsafe players the first-mover advantage that could not be overcome by a conditionally safe strategy (CS). To the contrary, in the latter regime, such a conditional strategy provides an efficient pathway to achieve safety behavior, especially when the monitoring of unsafe development is difficult.

Our results thus point out that it is essential to provide in the near AIS regime the necessary supporting mechanisms (such as suitable rewards and sanctions) 32, 17 to control the speed of AI development of rogue teams, in order to drive the AI development dynamics towards more beneficial directions and outcomes. Without such mechanisms, the unconditional unsafe players (AU) would always win the race against any other strategies that play SAFE at any development round (such as CS players in our analysis, which only does so only in the first round), achieving a significant payoff advantage. On the other hand, in the distant AIS regime, because reciprocal behaviour by itself is sufficient to ensure high levels of beneficial safe behaviour, less effort would be needed to ensure sustainable AI systems. This observation is in line with the response to the risk of AGI by many researchers that ’no action needed’ because AGI development will take a long time or will not be possible at all 8, 13 (see also a survey of responses in ref. 32).

Moreover, our results imply that by advertising that AGI is about to arrive might lead to an acceleration of the AI race, and to a decrease of safety precautions. In other words, our results thus support the argument that the rhetoric and framing of the AI development race and how close it is to achieve the AGI might strongly influence the dynamics and outcome of the AI race 11, 6.

In the current models we assume that an AI disaster might occur only when a true AI or AGI has been achieved, i.e. after the development steps have been completed. However, it might be the case that some smaller scaled disasters might occur before that milestone, especially when it is not clear whether and when AGI will or has been be achieved, and there might even be false beliefs regarding its presence. What is more, parties may release over simplistic AI but deliberatively advertise more than it can achieve, thereby leading to unforeseen usage disasters. We will analyse these scenarios in future works.

Last but not least, it is noteworthy that despite focusing on AI development race in this paper, our results are generally applicable to any kind of racing situations such as technological innovation problems where there is a significant advantage to be achieved when reaching a target first.

4 Methods

AI race model definition.

The AI development race is modeled as a repeated two-player game, consisting of development rounds. In each round, the players can collect benefits from their intermediate AI products, depending on whether they choose to play SAFE or UNSAFE. Assuming a fixed benefit, , from the AI market, teams will share this benefit proportionally to their development speed. Moreover, we assume that with some probability those playing UNSAFE might be found out about their unsafe development and their products won’t be used, leading to 0 benefit. Thus, in each round of the race, we can write the payoff matrix as follows (with respect to the row player)

(3)

For instance, when two SAFE players interact, each needs to pay the cost and they share the benefit . When a SAFE player interacts with an UNSAFE one the SAFE player pays a cost and obtains the full benefit in case the UNSAFE co-player is found out (with probability ), and obtains a small part of the benefit otherwise (i.e. with probability ). When playing with a SAFE player, the UNSAFE does not have to pay any cost and obtains a larger share when not found out. Finally, when an UNSAFE player interacts with another UNSAFE, it obtains the shared benefit when both are not found out and the full benefit when it is not found out while the co-player is found out, and 0 otherwise. The payoff is thus: The payoff matrix defining averaged payoffs for the three strategies reads

(4)

Evolutionary Dynamics in Finite Populations.

We adopt here Evolutionary Game Theory (EGT) methods for finite populations to derive analytical results and numerical observations 23, 19, 22. In a repeated games, players’ average payoff over all the game rounds (see the payoff matrix in Equation 4) represents their fitness or social success, and evolutionary dynamics is shaped by social learning 18, 29, whereby the most successful players will tend to be imitated more often by the other players. In the current work, social learning is modeled using the so-called pairwise comparison rule 35, assuming that a player with fitness adopts the strategy of another player with fitness with probability given by the Fermi function, , where conveniently describes the selection intensity ( represents neutral drift while represents increasingly deterministic selection). For convenience of numerical computations, but without affecting analytical results, we assume here small mutation limit14, 19, 23

. As such, at most two strategies are present in the population simultaneously, and the behavioural dynamics can thus be described by a Markov Chain, where each state represents a homogeneous population and the transition probabilities between any two states are given by the fixation probability of a single mutant

14, 19, 23. The resulting Markov Chain has a stationary distribution, which describes the average time the population spends in an end state. In two-player game, the average payoffs in a population of A players and B players can be given as below (recall that is the population size), respectively,

(5)

The fixation probability that a single mutant A taking over a whole population with B players is as follows 35, 21, 23

(6)

where describes the probability to change the number of A players by one in a time step. Specifically, when , , representing the transition probability at neural limit.

Having obtained the fixation probabilities between any two states of a Markov chain, we can now describe its stationary distribution 14, 19. Namely, considering a set of strategies,

, their stationary distribution is given by the normalised eigenvector associated with the eigenvalue

of the transposed of a matrix , where and .

Risk-dominant conditions.

We can determine which selection direction is more probable: an A mutant fixating in a homogeneous population of individuals playing B or a B mutant fixating in a homogeneous population of individuals playing A. When the first is more likely than the latter, A is said to be risk-dominant against B 20, 15, which holds for any intensity of selection and in the limit of large when

(7)

5 Acknowledgements

This work is supported by Future of Life Institute (grant RFP2-154). L. M. P. acknowledges support from FCT/MEC NOVA LINCS PEst UID/CEC/04516/2019. F. C. S. acknowledges support from FCT Portugal (grants PTDC/EEI-SII/5081/2014, PTDC/MAT/STA/3358/2014).

References

  • 1 AI-Roadmap-Institute. Report from the ai race avoidance workshop, tokyo. 2017.
  • 2 Peter Apps. Are China, Russia winning the AI arms race?, January 2019. [Reuters; Online posted 15-January-2019].
  • 3 Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the precipice: a model of artificial intelligence development. AI & society, 31(2):201–206, 2016.
  • 4 Stuart Armstrong, Kaj Sotala, and Seán S Ó hÉigeartaigh. The errors, insights and lessons of famous ai predictions–and what they mean for the future. Journal of Experimental & Theoretical Artificial Intelligence, 26(3):317–342, 2014.
  • 5 Robert Axelrod. The Evolution of Cooperation. Basic Books, ISBN 0-465-02122-2, 1984.
  • 6 Seth D Baum. On the promotion of safe and socially beneficial artificial intelligence. AI & SOCIETY, 32(4):543–551, 2017.
  • 7 Nick Bostrom. Strategic implications of openness in ai development. Global Policy, 8(2):135–148, 2017.
  • 8 Rodney Brooks. I, rodney brooks, am a robot. IEEE Spectrum, 45(6):68–71, 2008.
  • 9 Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418–424, 2018.
  • 10 Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, page eaay2400, 2019.
  • 11 Stephen Cave and Seán ÓhÉigeartaigh. An ai race for strategic advantage: Rhetoric and risks. In AAAI/ACM Conference on Artificial Intelligence, Ethics and Society, pages 36–40, 2018.
  • 12 Montreal Declaration. The montreal declaration for the responsible development of artificial intelligence launched. https://www.canasean.com/the-montreal-declaration-for-the-responsible-development-of-artificial-intelligence-launched/, 2018.
  • 13 David Deutsch. The beginning of infinity: Explanations that transform the world. Penguin UK, 2011.
  • 14 D. Fudenberg and L. A. Imhof. Imitation processes with small mutations. Journal of Economic Theory, 131:251–262, 2005.
  • 15 Chaitanya S. Gokhale and Arne Traulsen. Evolutionary games in the multiverse. Proc. Natl. Acad. Sci. U.S.A., 107(12):5500–5504, March 2010.
  • 16 Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. When will AI exceed human performance? Evidence from AI experts. Journal of Artificial Intelligence Research, 62:729–754, 2018.
  • 17 The Anh Han, Luís Moniz Pereira, and Tom Lenaerts. Modelling and Influencing the AI Bidding War: A Research Agenda. In Proceedings of the AAAI/ACM conference AI, Ethics and Society, pages 5–11, 2019.
  • 18 J. Hofbauer and K. Sigmund. Evolutionary Games and Population Dynamics. Cambridge University Press, 1998.
  • 19 L. A. Imhof, D. Fudenberg, and Martin A. Nowak. Evolutionary cycles of cooperation and defection. Proc. Natl. Acad. Sci. U.S.A., 102:10797–10800, 2005.
  • 20 M. Kandori, G. J. Mailath, and R. Rob. Learning, mutation, and long run equilibria in games. Econometrica, 61:29–56, 1993.
  • 21 S. Karlin and H. E. Taylor. A First Course in Stochastic Processes. Academic Press, New York, 1975.
  • 22 M. A. Nowak. Evolutionary Dynamics: Exploring the Equations of Life. Harvard University Press, Cambridge, MA, 2006.
  • 23 M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg. Emergence of cooperation and evolutionary stability in finite populations. Nature, 428:646–650, 2004.
  • 24 Future of Life Institute. Autonomous Weapons: An Open Letter from AI & Robotics Researchers. Technical report, Future of Life Institute, Cambridge, MA, 2015.
  • 25 Future of Life Institute. Lethal autonomous weapons pledge. https://futureoflife.org/lethal-autonomous-weapons-pledge/, 2019.
  • 26 Dennis Pamlin and Stuart Armstrong. Global challenges: 12 risks that threaten human civilization. Global Challenges Foundation, Stockholm, 2015.
  • 27 PwC. Sizing the prize: What’s the real value of ai for your business and how can you capitalise? Technical report, PwC, London, United Kingdom, 2017.
  • 28 Stuart Russell, S Hauert, R Altman, and M Veloso. Ethics of artificial intelligence. Nature, 521(7553):415–416, 2015.
  • 29 Karl Sigmund. The Calculus of Selfishness. Princeton University Press, 2010.
  • 30 David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.

    Science, 362(6419):1140–1144, 2018.
  • 31 David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
  • 32 Kaj Sotala and Roman V Yampolskiy. Responses to catastrophic agi risk: a survey. Physica Scripta, 90(1):018001, 2014.
  • 33 Luc Steels and Ramon Lopez de Mantaras. The barcelona declaration for the proper development and usage of artificial intelligence in europe. AI Communications, (Preprint):1–10, 2018.
  • 34 Mariarosaria Taddeo and Luciano Floridi. Regulate artificial intelligence to avert cyber arms race. Nature, 556(7701):296–298, 2018.
  • 35 A. Traulsen, M. A. Nowak, and J. M. Pacheco. Stochastic dynamics of invasion and fixation. Phys. Rev. E, 74:11909, 2006.