Humankind faces many existential threats, but has limited resources to mitigate them. Choosing how and when to deploy those resources is, therefore, a fateful decision. Here, I analyze the priority for allocating resources to mitigate the risk of superintelligences.
Part I observes that a superintelligence unconnected to the outside world (de-efferented) carries no threat, and that any threat from a harmful superintelligence derives from the peripheral systems to which it is connected, e.g., nuclear weapons, biotechnology, etc. Because existentially-threatening peripheral systems already exist and are controlled by humans, the initial effects of a superintelligence would merely add to the existing human-derived risk. This additive risk can be quantified and, with specific assumptions, is shown to decrease with the square of the number of humans having the capability to collapse civilization.
Part II proposes that biotechnology ranks high in risk among peripheral systems because, according to all indications, many humans already have the technological capability to engineer harmful microbes having pandemic spread. Progress in biomedicine and computing will proliferate this threat. “Savant” software that is not generally superintelligent will underpin much of this progress, thereby becoming the software responsible for the highest and most imminent existential risk – ahead of hypothetical risk from superintelligences.
The analysis concludes that resources should be preferentially applied to mitigating the risk of peripheral systems and savant software. Concerns about superintelligence are at most secondary, and possibly superfluous.
Many scholars believe that a highly capable advanced intelligence – a “superintelligence” – poses a near-term existential threat to human civilization.111Herein, “existential threat” refers to the collapse of civilization or the extinction of humankind [Beard et al]. Although “collapse” has an accepted socio-political definition [Tainter], it here means a profound loss of technological capacity, e.g., to a level where interstellar communication is not possible [Sagan]. This essay demonstrates that these concerns, well-meaning as they may be, are harmful because, first, they distract attention from the true loci of danger and, second, because they underestimate the proximity of the threat from less capable “savant” systems already in existence.
As machine intelligences have attracted the most concern, our discussion will be restricted to such intelligences. Extrapolation to other substrates of intelligence is straightforward. Though interdisciplinary, this paper aims for an audience oriented more toward computers than biosciences, and thus simplifies many of the biotechnological concepts.
1 Superintelligences and Savants
A superintelligence is “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest” [Bostrom, p26] [Good].
By contrast, we may define a “savant” as any computing system that greatly exceeds the cognitive performance of humans in only one or a few domains of interest (Figure 1). Savants may range from low-complexity software to specialized artificial intelligence. For example, a pocket calculator and a protein-folding predictor both qualify as savant software, albeit of greatly differing complexities.222An alternative terminology calls savants “narrow superintelligence” and defines “general superintelligence” to be what the present work calls simply “superintelligence.” The alternative terminology seems undesirable because it is wordier and corrupts the Bostrom/Good definition of “superintelligence” given above.
2 Central Systems and Peripheral Systems
An intelligence, no matter how smart, how dull, how benevolent, or how evil, cannot affect human lives if it is walled off by itself and able neither to communicate with the outside world nor physically manipulate the outside world.333We ignore power consumption, waste production, and other sustaining functions normally unassociated with intelligence. The medical profession would apply the descriptors “locked-in” or “de-efferented” to such isolated intelligences [Plum & Posner].
Thus, if one is concerned about risk from intelligences, it is necessary to distinguish a “central system,” having some degree of intelligence, from the “peripheral systems” that the central system uses as effectors to alter the world, as in Figure 2. Because words can alter the physical world444Example: Thomas Paine’s pamphlet, Common Sense, helped instigate the American Revolution in 1776. and because markedly impaired word-production is part of the locked-in syndrome, words and communication are considered a peripheral system.
Readers may usefully conceptualize central systems as “agents” and peripheral systems as “tools.” However, the systems terms are preferred for their emphasis on information flow, and to distinguish from other meanings of “tool” [Dyson].
3 Misplaced Focus on Central Systems
In contemplating the dangers arising from machine intelligences, scholars to date have focused on characteristics of the intelligence itself, i.e., the central system. The clearest example is Professor Nick Bostrom’s 415-page book whose simple main title – Superintelligence – clearly signals its focus [Bostrom].
It is straightforward to demonstrate that existential catastrophe for humankind rests in the peripheral systems, not the central systems. An asteroid of the dinosaur-killing class is a peripheral system with a zero-intelligence central system. So, too, is the smallpox virus, which undid the Aztec civilization [Oldstone], and the Black Death, which caused demographic collapse across Europe [Kelly].555George R. Stewart’s unforgettable 1949 novel, Earth Abides, graphically shows how a pandemic of less than universal lethality may collapse civilization [Stewart]. It also exemplifies the “boring apocalypse” typology, in which an interplay of multiple critical systems enhance the deleterious effect of the primary insult [Liu et al].
It is also clear that central systems of modest capacity are extremely dangerous when teamed with powerful peripheral systems. The brain of Joseph Stalin was not superintelligent, but because it was coupled to a powerful far-reaching peripheral system – the state security apparatus of the Soviet Union – it could kill one million people in the Soviet Union during just two non-war years, 1937-1938 [Ellman]. Stalin’s brain could not have done so if it were in Switzerland. As further examples, the humans and computers that constitute the central system controlling nuclear weapons do not rise to superintelligence levels. And, of course, every war in human history has been prodded along by words (a peripheral system) emanating from less-than-superintelligent human brains.
Thus, when scholars interested in machine intelligence ask the question “What harms can a superintelligence cause?” they are neglecting a far more pressing question: “What harms can result from a central system that is composed of a human intelligence combined with a less-than-superintelligent machine?” (Figure 3). Or, more pointedly: What harms can result from intelligent human malevolence teamed with machine savants? The answer, as we have seen, is simply: “It depends on the peripheral system.”
This realization leads to a second conclusion that further diminishes the near-term relevance of machine superintelligence: For a given peripheral system, every threat that includes a superintelligent central system is duplicated or presaged by a similar threat in which the central system combines a machine savant with one or more humans (Figure 4).
As a concrete example, Bostrom [pp119-120] describes a malevolent superintelligence intending to destroy humankind by plague.666“Malevolent” and “intending” reflect a human victim’s assessment of the superintelligence and its actions. Although the actions may be driven by mechanisms as banal as a misaligned utility function – arising intentionally or unintentionally – combined with backward-chaining, they certainly have the appearance of malevolence and intention. Thus, this work uses “malevolent” synonymously with “harmful.” In his scenario, the superintelligence designs a microbe to accomplish this goal, then coopts and/or hoodwinks human accomplices to procure the necessary supplies and to synthesize the organism. Although this is a perfectly valid and internally consistent scenario, it would be misguided to worry about it today. The more proximate concern is that malevolent humans – which already exist in abundance – will use savant-level software to design an “end-times” microbe, and will use their own resources (personnel and lab control software) to synthesize it with existing techniques. Part II will demonstrate that the microbial threat is indeed proximate.
Thus, in gauging the existential risk from the biotechnology peripheral system, a superintelligence in the central system offers little additional risk because of the risk derived from human-plus-savant central-systems that occurs earlier in time (Figure 5). Any sensible mitigation strategy should, therefore, focus on the human-derived risk ahead of the speculative superintelligence-derived risk, assuming the human-derived risk is significant. Or, more colloquially, just as “Airway” traditionally comes before “Breathing” and “Circulation” in cardiopulmonary resuscitation, humans should get their own house in order before worrying about superintelligences.
Putting the focus back on peripheral systems, the instant a machine superintelligence takes shape, it will have at its disposal several peripheral systems by which to extinguish human civilization. The rise of a superintelligence, therefore, merely adds another entity to the list of sentient intelligences who are capable of pulling the end-times trigger, as section 5 and [Torres] explore in detail. The superintelligence may choose to invent new peripheral systems (last line of Figure 4), but that would be unnecessary for its human-killing goals if humankind had left pre-existing end-times peripheral systems unmitigated, ready for the just-awakened superintelligence to use effectively (the earlier lines of Figure 4).
As an example, over many decades the United States has labored to create precisely such mitigations for control of its nuclear weapons [Schlosser], with the aim of creating a “fail-safe” peripheral system that prevents any unauthorized human from initiating a nuclear strike. Inevitably flawed, but continually refined, these mitigations have so far worked. By contrast, many civilization-sustaining peripheral systems – producing and moving food, water, energy, and information – are connected to the global internet without similar mitigations, enabling central systems consisting of national leaders, human criminals, and even “script kiddies” [Putman] to unleash breakdowns of unplumbed depth. If these peripheral systems are so exploitable that 48 billion robocalls in America each year cannot be prevented [Palmer], then our destruction need not await a machine superintelligence.
A robust approach to mitigating existential risk assumes the worst of the central system and concentrates on limiting the destructive outcome of peripheral systems. That way, a Stalin-oriented superintelligence, newly awakened, finds itself among peripheral systems having a misuse potential more like Switzerland than the Soviet Union.
4 Mistaken Hopelessness
According to [Bostrom p116], the overwhelmingly superior intellectual capability of a superintelligence – perhaps as superior to humans as we are to beetles [Bostrom p112] – will enable it to foresee and evade all defenses humans contrive against it. One may believe, therefore, that all work to strengthen peripheral systems against misuse by a superintelligent central system would be wasted effort, because it is doomed to fail.
This is a dangerous belief because, first, it could lead to hopelessness and inaction at a critical point in human history, and second, because it is wrong.
It is wrong for the obvious reason that a superintelligence may never appear – for whatever reason – whereas malevolent central systems composed of humans plus savant software exist now. Moreover, extraordinarily capable savant software for biotechnology must come into existence before superintelligent software does, because such savant software is a mandatory component of a superintelligence that is definitionally required to excel in “all domains of interest.”777Attempting to neutralize savant-associated risk, by urgently creating a beneficent superintelligence to protect the human race, would be a literal deus ex machina, i.e., the creation of an infallible, immortal, all-knowing intelligence, ultimately with the ability to create souls and tend them [Bostrom pp122-123]. Although building God would solve many problems, as the pinnacle of hubris it cannot be recommended as a realistic solution to anything. Failing to limit the destructive potential of peripheral systems exposes humankind to very real existential threats.
Hopelessness is also wrong because mitigating the threat from peripheral systems may indeed succeed against a superintelligence, given that (a) a superintelligence is not required to be all-seeing, and (b) invulnerability need not develop simultaneously with superintelligence. For the latter case, consider a machine that becomes superintelligent in just minutes, and immediately devises an impregnable plan to foresee and resist attacks by humans. However, what if executing this plan requires some item not immediately available to the machine, e.g., seven billion doses of vaccine-resistant smallpox, or the DNA sequence of every human, or 5G transmitters spaced every 1000 feet worldwide? No matter the brilliance of the superintelligence, these physical items simply cannot come into being for some time. This creates a vulnerability window in which it remains possible for human preparation to frustrate the superintelligence’s plans. Limiting the tools available to the superintelligence is, therefore, a wise strategy.888The logical extreme of this reasoning is clearly true: the best defense against superintelligence is a material civilization incapable of building it.
This should not surprise: intellect is not all. Every year, tiny-brained snakes succeed in biting 5 million big-brained humans, and one hundred thousand humans die [Warrell].
5 Quantifying the Risk of Superintelligence
If a newly-arisen superintelligence is viewed as an increment to the number of sentient entities that can extinguish civilization via some peripheral system, then it becomes conceptually straightforward to quantify the existential risk attributable to superintelligence.
Consider some number,
, of independently-acting sentient entities who can end civilization. These entities correspond to central systems. Assume, further, that all of these entities have the same probability,, of ending civilization in any given year. Starting from Equation 5 in Appendix A, [Sotos] showed that the median lifetime in years of this civilization, i.e., the 50th-percentile “lethal duration” (), is:
So, for example, if molecular biologists, with each having one-in-a-million chance of collapsing civilization each year, then the predicted median lifespan for earth’s civilization is 231 years. If 30,000 molecular biologists exist, then the median lifespan is 23 years. (Herein, only molecular biologists who know how to collapse civilization “count.” Thus, the term is subtly different from the everyday meaning of “molecular biologist.” Even among this group, per year is probably too low.)
When a superintelligence appears, increases by one. We initially assume it has the same as the molecular biologists. Recalculating the median civilizational lifespan using , and comparing that to the lifespan with entities, yields a difference (loss) in civilizational lifespan, which is the existential risk attributable to the superintelligence:
In the 3000-molecular-biologist example, adding the 3001st entity, whether machine superintelligence or human, shortens the civilization’s median lifespan by 28 days. In the 30,000 example, the 30,001st entity shortens the civilization’s lifespan by almost nothing: less than 7 hours.
The term in the denominator of Equation 4
means there will be little risk effect – perhaps even trivial – from introducing a superintelligence into a world where a large number of humans can end civilization. Thus, the world’s first superintelligence, technologically momentous as it will be, could existentially be no more significant than adding a 3001st or 30,001st molecular biologist, ifis uniform.
The assumption of uniform is more general than it may seem. Appendix A shows that, so long as no entity has larger than about 0.1, simple arithmetic can recast any scenario having non-uniform values into an equivalent scenario having uniform values and the same , but a different . For example, an entity with thrice-normal would be recast to three entities having normal . Thus, even the creation of a superintelligence having a that is 100 times larger than the uniform of human molecular biologists would be risk-equivalent to the creation of molecular biologists 3001 through 3100 … hardly a newsworthy event. Going forward, references to “uniform ” will include well-behaved scenarios that have been transformed in this way.
We may now consider a machine superintelligence that is massively more murderous than humans. Scholars in the existential risk community in 2008 offered 5% odds that a machine superintelligence (S) would kill a billion humans before the year 2100, i.e., 0.05 over 92 years [Sandberg and Bostrom]. To be generous, we can setmuch higher – 0.05 per year – so that is 50,000 times greater than our baseline for molecular biologists. Hence, by Equation 16 in Appendix A, the appearance of this highly malevolent machine superintelligence is equivalent to bestowing on 50,000 humans the capability to end civilization, each with one-in-a-million annual odds. However, as will be shown later, a scenario with 50,000 such humans is both more immediate and more probable than the rise of software that wants to kill us all.
Extensions to Equation 4 would show a ceiling effect, per Figure 5. For example, if the existence of 3000 morally unconstrained molecular biologists yields a 99% chance of ending civilization in the next 20 years, then the addition of a murderous machine superintelligence can maximally add only a 1% risk in that time frame. No superintelligence can bring more risk than humans leave open.
Equation 1 encapsulates one more arithmetic insight. Assuming uniform , merging all the peripheral systems of Figure 4 into a single class, so that , demonstrates that the overall risk to humankind’s future is dominated by the peripheral system having the most entities that can trigger it (because ).
Recent and continuing progress in biotechnology arguably positions made-plagues as the current “most” system. Thus, if new savant-level software, far below the level of superintelligence, substantially increases (i.e., ), that would bring immediate, substantial existential risk to humankind and likely qualify as the most dangerous software development on earth. Part II, below, expands this point.
6 Biotechnology as a Dual-Use Peripheral System
The laudable goal of reducing human sickness animates the vast majority of biotechnological development. The field started in 1972, with the first controlled construction of a novel DNA molecule [Berg and Mertz] [Cohen]. Since then, biotechnology has revolutionized the treatment of heart attacks, stroke, cancer, auto-immune disease, and many other conditions. It enabled the rapid development of covid-19 vaccines in 2020 and is poised to eliminate one of humankind’s great scourges – malaria – by using a “gene drive” to cause extinction of the Anopheles mosquito that transmits the disease [Hammond & Galizi].
However, equally rational viewpoints would classify humankind’s biotechnology in 2021 as an existential threat to multiple Anopheles species and as a genocidal weapon against cancer cells. These alternate viewpoints perfectly illustrate the danger that is unalterably inherent in biotechnology: it is dual-use [Hoffman p 132], meaning it yields weapons as well as medicines. One organism’s biotech salvation is another’s biotech nightmare.
Today, great controversy surrounds the question of whether humans deliberately created the covid-19 virus, by adapting it from a natural coronavirus to become deadlier and more contagious [Anderson et al] [Wade]. The most important facet of the controversy is not the answer to the question, whatever it may be, but the absence of claims that humans in 2019 were incapable of engineering such a virus.
That lack of skepticism, alone, is a Rubicon for humankind’s future: it means that today there is no doubt that a cadre of humans already has the skills to create world-girding microbes of unnatural virulence.
Whether biotechnology becomes a nightmare or a salvation for Homo sapiens depends directly on choices that humans will make. Despite thousands of good choices, just one bad one could bring the nightmare.
Figure 4 shows the time horizons for several existential threats. For two of the eight threats that exist today – nuclear warfare and engineered plagues – the actions of one properly resourced individual human can trigger a global catastrophe. While the putative number of such humans is stable and small for the nuclear threat (n=2), the situation is entirely different for the biotechnology threat.
In biotechnology, we might assume that anyone having sophisticated knowledge of genetic techniques is able to create, or at least duplicate, an end-times plague. Of course, there is no clear definition of “sophisticated knowledge,” but as one proxy, from 2008 through 2015 approximately 180,000 individuals authored 5 or more articles in the scientific literature having the term “genetic techniques” as a major keyword [Sotos]. From Equation 1, if 180,000 persons each had an annual one-in-a-million probability of creating an end-times plague, our civilization’s median lifespan would be 3.8 years. That is nightmare enough, with no need to postulate script kiddies branching into biotechnology.
It may be argued, however, that scientific knowledge alone is insufficient to collapse civilization, and that significant capital resources will be required – to hire personnel and buy the equipment needed to develop and deploy a civilization-ending microbe. In 2020, the entire annual budget of the U.S. National Institutes of Health was $42 billion. Even if it took $50 billion to hire the scientists and buy the equipment, two dozen people and families on earth have such resources today; if it took $10 billion, then 228 people and families have the resources; if $1 billion, then 2755 people and families have the resources… including the Kardashians [Dolan et al].101010As far back as 2014 the author knew a PhD molecular geneticist who thought that $1 billion would enable someone with his level of training to kill 95% of humankind. From Equation 1, if 2755 entities each had an annual one-in-a-million probability of creating an end-times plague, our civilization’s median lifespan would be 251 years. If one of them had a 15% chance, then it would be 4.2 years.
Bad decisions could enormously increase proliferation. For example, the CEO of Twist Biosciences aims to use DNA molecules as an information-storage alternative to disk drives [Wired]. This would seem to require the ability to synthesize long strings of arbitrary DNA at very low cost. Her cost goal is less than 8 cents for a DNA string having the size of the entire human genome. Extrapolating, it would therefore cost 0.004 cents to manufacture the DNA for a smallpox virus (the sequence is available on the internet). Presumably, a trivial looping script could print essentially unlimited copies. If Twist applies for patents, their underlying technology would become public, allowing others to remove any safeguards that Twist might have built in.
This section has been necessarily speculative. No one knows (or is saying) exactly what needs to happen before the genie is freed completely from the bottle. Humans first created a synthetic life form in 2010 [Gibson et al]. In 2012 two research teams deliberately enhanced the contagiousness of an influenza virus [Herfst et al] [Imai et al]. In 2001 a team enhanced the lethality of a close relative of smallpox [Jackson et al]. In 2016 a team synthesized a different close relative of the smallpox virus from pieces of DNA they obtained by mail order [Kupferschmidt], leading them to conclude that “no viral pathogen is likely beyond the reach of synthetic biology” [Noyce]. Yet, we still inhabit biotechnology’s childhood, which was born less than one working lifetime ago.
8 Biotechnological Machine Savants
If a human chooses to design, build, and deploy a biotechnological weapon, how might machine savant software make that effort easier, thereby making it accessible to more people?
The answer is best explained if the weaponeer’s fundamental challenge is recast as a search problem. As an example, consider the SARS-CoV-2 virus, which, although not an end-times plague, has devastated the world. Conceptually, it is merely a very large number. Figure 6 represents the SARS-CoV-2 genome as a single hexadecimal number – it fits on one printed page. This number, when translated into RNA and inserted inside a human cell, becomes integrated into the cell’s metabolic processes and yields polymer and other molecules that assemble themselves into what we recognize as a virus (Figure 7).
Importantly, the nearly 30,000 base pairs of the SARS-CoV-2 genome differ in only 17 significant ways from the genome of a naturally occurring bat coronavirus [Anderson et al]. If aspiring but unsophisticated weaponeers started with the bat virus, and knew only that 17 changes were required to transform it to a weapon, they could adopt a brute force method to identify the weapon by searching through all variants of the bat virus having 17 differences. Because this “search space” would include an astronomical number of variants, the brute force search method is not practical.
In general, therefore, the challenge of designing a bioweapon reduces to the broader task of finding methods to narrow a search space – a general task that has long been at the leading edge of computer science [Barr & Feigenbaum].
With that recasting, we next examine a few intersections of software and biotechnology to see how software might accelerate the creation of bioweapons. The examples are chosen only for clarity, and are of course incomplete.
8.1 Replicating Human Expertise in Software
Today, if a human were to weaponize a virus – for example, the influenza virus – that human must possess a deep understanding of influenza virus biology. Otherwise, the virus’ genome is so large that making random, uninformed changes in it would be statistically unlikely to yield a virus with the highly refined (but awful) characteristics of an effective weapon.
Such expertise is scarce. Endowing a machine savant with the biological knowledge of a human influenza virus expert would be an obvious step in proliferating, and thereby accelerating, the discovery of both influenza treatments and weapons. Moreover, from an engineering standpoint, this would be far less challenging than developing a general artificial intelligence or a machine super-intelligence, given that software architectures to transfer expert knowledge have been available for decades [van Melle et al].
8.2 In Silico Modeling
In silico modeling of biological and biochemical systems has emerged as a powerful and widespread tool to discover new treatments for diseases. For example, early in the covid-19 pandemic, Oak Ridge National Laboratory unleashed the world’s most powerful computer for 24 hours to run models predicting the binding of one billion different chemical compounds to proteins in the SARS-CoV-2 virus [Parks & Smith] [Acharya].
In other words, the computer rapidly traversed an initial search space of one billion treatment candidates, eliminating those predicted to have low anti-virus efficacy by virtue of their poor binding to the virus. This left a much smaller search space of candidates to be tested in the slow, subsequent steps of clinical testing. No human could have performed this savant-like task with brain alone – the models are far too complicated [Naderi].
Similarly, and unsurprisingly, pharmaceutical companies have invested in computer systems that predict when a particular candidate drug molecule will cause unwanted, market-killing side effects [Wang]. Perversely, but in keeping with the dual-use nature of biotechnology, these systems also contain the knowledge needed to deliberately create molecules that cause side effects, i.e., weapons.
In general, computer models provide a means to perform experiments at a scale that humans cannot, thereby narrowing the search space.
8.3 Software-Controlled Laboratories
Biotechnological danger from completely unremarkable computer software is also considerable. Computer control of industrial-scale wet-ware laboratories (e.g., [Chory], [Strickland]) could leverage evolutionary selection to produce microbes having specific characteristics. Here, the difficult tasks are in the laboratory functions, not the computer control, but it is the automation that delivers the scale to rapidly reduce the search space by screening enormous numbers of candidate microbes.
One could therefore imagine a large computer-controlled laboratory aiming to evolve a variant of the Ebola virus that spreads through the air.111111This idea is from [Clancy p425], although airborne transmission is entertained with any new pathogenic virus. In the 1980s the Soviet Union tried to build robotic laboratories that would create a new viral weapon each month [Hoffman p107]. Today, they might succeed. Ominously, doing so is only a matter of will and resources, because this general evolutionary approach is well established: a 2018 Nobel Prize was awarded for developing “directed evolution of enzymes” with bacteria as factories [Arnold].121212Even earlier, in the 1940s, evolutionary techniques were used to increase penicillin yield from molds [Hobby]. Lab-on-a-chip technology, supplemented by the emerging organ-on-a-chip technology that could include human organoids [Azizipour et al] [Schutgens & Clevers] [Kim et al], will enable complex, highly parallelized, biological evolutionary selection functions, thereby bringing directed evolution to human-microbe interaction.
Here, a comparison with machine super-intelligence is instructive. One of the pre-eminent concerns about machine super-intelligence is the possibility that a moderately advanced digital intelligence could design a more intelligent digital successor that would quickly design a still more intelligent digital successor and so on, leading to an explosion of digital intelligence that could be malign [Bostrom pp75-94]. As yet, these concerns remain hypothetical, because the engineering basis for such an explosion remains unknown.
By contrast, directed-evolution-in-a-laboratory already provides an exact, actionable blueprint to create and discover biological malign entities while leveraging the short time scale of microbial generations (for SARS-CoV-2, on the order of 10 hours [Bar-On et al]). It would simply be a speedier reimplementation of nature’s proven technique that has produced all the scourges, known and unknown, that afflict all the living things on earth.
8.4 Software-Assisted Tools
Although scientific progress historically arises from the arrival of both new paradigms and new tools, in recent decades tools have become pre-eminent [Dyson], especially in biology [Jogalekar]. Without doubt, all parts of biology have benefitted – and will continue to benefit – from tools that incorporate computer software. It is therefore reasonable to believe that advances in software will spur further advances in molecular biology – and thus the potential for bioweaponry.
Synthetic biology, long considered an existential threat [Block], but also “a powerful tool to create therapeutics which can be rationally designed” [Leventhal et al], is a case in point. As a specific example, in discussing the prospects for designing synthetic “oncolytic” viruses (OVs) to treat cancerous glioma cells, Monie and colleagues highlight no fewer than 10 software-enabled capabilities within systems biology that enable rational synthetic biology:
Systems biology complements synthetic biology by enabling
complex design and analysis of genetic circuits. Tumor
microenvironment cell states can be defined by using
single-cell multiomics data sets such as RNA sequencing
and mass cytometry. These high-dimensional
data sets are amenable to machine learning and network-based
classification. Computation identifies biological
features unique to glioma cells, elucidates pathways
orthogonal to genetic circuit designs, and helps prioritize
OV-based approaches. After the synthetic OVs are
constructed and tested, further systems analyses ensure
that the OVs have predictable effects on the cells and the
greater tumor microenvironment, statistically accounting
for complex stochastic behavior. This knowledge is fed
back into the design of improved, next-generation OV genetic
circuits. [Monie et al]
Systems biology complements synthetic biology by enabling complex design and analysis of genetic circuits. Tumor microenvironment cell states can be defined by using single-cell multiomics data sets such as RNA sequencing and mass cytometry. These high-dimensional data sets are amenable to machine learning and network-based classification. Computation identifies biological features unique to glioma cells, elucidates pathways orthogonal to genetic circuit designs, and helps prioritize OV-based approaches. After the synthetic OVs are constructed and tested, further systems analyses ensure that the OVs have predictable effects on the cells and the greater tumor microenvironment, statistically accounting for complex stochastic behavior. This knowledge is fed back into the design of improved, next-generation OV genetic circuits. [Monie et al]
This is search-space-reduction of the highest order. If synthetic biology carries existential risk, then Monie et al show that even the comparatively low-performing software inside sequencing and cytometry machines contributes to that risk, as do domain-independent software techniques such as machine learning and statistics. The larger lesson is that progress and proliferation of software tools, teamed with human intelligence, leads to progress in molecular biology, which in turn enables smarter choices in search space to generate better anti-glioma bioweapons.
Or, in simpler terms, the rising software tide floats all boats, including the warships.
With biotechnology having passed the Rubicon stage where engineered pandemic organisms are accepted as possible, the central question has become: How easy and how fast will it become to design end-times infectious weapons?
Over the past decades, biotechnology and computer capabilities have risen together. The brief and limited examples above show that computer software brings a powerful capability into the central system for narrowing search spaces in the design of biological peripheral systems. Moreover, this capability has been, and will continue to be, achieved with software that functions far below the level of a machine superintelligence.
It is certain that innovation in biocomputation will continue, and will endow future biotechnological savants with improved power to reduce search spaces. Figure 8 shows how the scientific interest in biocomputation has exploded in less than one working lifetime, as indicated by the annual number of publications in the biomedical literature whose title contains the phrase “in silico.”
Publication counts do not, of course, predict changes in search-space-reducing capability. But, just as [Bostrom pp 43, 84] worries that a single breakthrough could deliver a superintelligence far sooner that expected, an unexpected breakthrough in biology (or computer science) could deliver a super-searching machine savant far sooner than the smooth curve of Figure 8 might suggest. The possibility of such discontinuities cannot be discounted, given molecular biology’s past history of wholly unexpected Nobel-honored breakthroughs, headed by the polymerase chain reaction, CRISPR/Cas9, phage display for producing monoclonal antibodies, and RNA interference.
9 Defensive Savants
Although vaccines against covid-19 were developed with unprecedented speed, and although physical distancing was a powerful defensive force when used, it should not be assumed that pandemic defense is a solved problem and that all microbes will be susceptible to such interventions. Had the SARS-CoV-2 virus spread in water, as cholera does; or had it spread in the air over intercontinental distances, as do the microbes that ride the winds from Africa to America [Griffin]; or had there been an environmental reservoir, as with anthrax; or had its clinical syndrome been a usually-mild initial illness followed by years of asymptomatic contagiousness before the fatal culmination, as with HIV; or had it been more difficult to develop a vaccine, as the HIV virus has demonstrated to human science every day for 40 years, the situation would have been incomparably more dire.
Because the threat from biotechnology depends on the gap between “offensive” and “defensive” biotechnological systems, the need to perpetually develop more capable defensive systems is apparent. Machine intelligence can certainly assist with defense, but defense has inherent characteristics that limit its ability to keep pace with offense:
Time scales in biology greatly favor offense over defense, because microbes can spread widely before countermeasures can be developed and fielded.
Defense requires deeper domain understanding than offense because it is harder to repair life than to break it. For example, safety constrains defense, but not offense.
Defense requires broader domain understanding because offense can choose any attack from a large target space, whereas defense must be ready to counter anything.
Humans are actively trying to overcome the body’s immune defenses, e.g., physicians altering a patient’s immune system so it will not destroy a gene therapy virus injected into the patient [Shirley et al] [Thacker et al].
It would seem, therefore, that even with the assistance of biotechnological savants, offense will be first and stronger, and defense later and weaker.
10 Biotechnology Summary
Four hundred and eight pages of small print constitute the 1995 book, Encyclopedia of Plague & Pestilence, which lists 474 such events in the preceding 500 years, including 165 in the 20th century [Kohn]. Clearly, epidemics are inherent to human existence, antibiotics notwithstanding.
Despite our species surviving all of those epidemics, it is reasonable to hypothesize that some nucleic acid sequence, i.e., a number perhaps not too different from the one shown in Figure 6, codes for an “end-times” microbe that would cause a pandemic so severe as to collapse human civilization on earth. Certainly it is not possible to reject the possibility.131313Any rejection attempt that cites some precedent in extant microbes may itself be rejected, as follows. Reproductive fitness has been the sole selection factor for natural microbial evolution on earth. A weaponeer would impose a different selection regime, as Soviet bio-engineers did in the 1980s when they developed a multiple sclerosis weapon that had two clinical phases [Hoffman pp109-110, 298-299]. Even in the natural world, the supposed inverse relationship between virulence and contagiousness has multiple counter-examples, including a single chytrid fungus responsible for the extinction of numerous amphibian species [Casadevall], and the delta variant of the SARS-CoV-2 virus, which is both more transmissible and more serious than earlier variants [Sheikh et al]. If one grants the possibility of such a microbe, then civilization survives only because of the enormously large search space of possible genomes for it. This is security by obscurity – a protection that fails as soon as the key is discovered.
If there is a genomic key-number that can end human civilization, software will aid in its discovery. Superintelligent software will not do this, because savant software, created and directed by smart but corrupt humans, will do it first. Machine superintelligence poses a small existential risk because human-controlled biotechnology poses a large risk, leaving little left over.
To facilitate mental manipulation, this appendix shows how to simplify mathematical models in which multiple people (“entities”) have different probabilities of ending civilization.
First we model the most basic scenario, in which independently-acting entities each have a uniform and unchanging annual probability, , of ending civilization. After years, the civilization will have a probability, , of still existing [Sotos]:
For the scenario in which entities have non-uniform values, we pick one probability, , as reference, and count the number of entities, , having that probability. We do the same for all the different probabilities, , , etc. We can then rewrite Equation 5 to:
We are interested in the median lifespan of the civilization, so will set henceforth. This entails that the number of years, , is the – the median predicted civilizational lifespan in years – as defined in [Sotos] and in Equation 1 above. So:
[Sotos] shows that for small (the error is about 5% for and decreases rapidly as gets smaller). Thus, if we assume that all , then:
With as a referent, we set and to yield:
In the situation where all are the same, all , , … will be zero, so:
Comparing Equations 16 and 17 achieves our goal of making plain the effect of non-uniform values. It shows that non-uniform values (Equation 16) equate to a scenario where is uniform (Equation 17), except that an adjustment in has been made. Moreover, the adjustment in is intuitively easy to understand. Having one entity with equates to having entities with (recall ).
For example, if, among entities, a single entity has a value that is twice the value of all the other entities (), then that entity is simply counted as two entities having , so that Equation 17 for uniform values is computed using instead of . Or, if three entities have their as six times the value of the other entities (), then each of those three entities is counted as six entities having the baseline , hence is increased by in Equation 17. Steps like these can be repeated as often as needed to make all uniform. The sole restriction is that no entity may have a value greater than about 0.1.
Acharya A. Supercomputer-Based Ensemble Docking Drug Discovery Pipelinewith Application to Covid-19. J Chem Inf Model.2020; 60: 5832-5852. doi: 10.3389/frai.2020.00065
Anderson KG, et al. The proximal origin of SARS-CoV-2. Nature Medicine. 2020; 26: 450-452. doi: 10.1038/s41591-020-0820-9
Arnold, Frances. Innovation by Evolution: Bringing New Chemistry to Life. Nobel Lecture, Dec. 8, 2018. https://www.nobelprize.org/uploads/2018/10/arnold-lecture.pdf
Azizipour N, et al. Evolution of Biochip Technology: A Review from Lab-on-a-Chip to Organ-on-a-Chip. Micromachines (Basel). 2020; 11: 599. doi: 10.3390/mi11060599
Bar-On YM, et al. SARS-CoV-2 (COVID-19) by the numbers. eLife. 2020; 9: e57309. doi: 10.7554/eLife.57309
Barr, Avron; Feigenbaum, Edward A. The Artificial Intelligence Handbook. Volume 1. Los Altos, CA: William Kaufman, 1981. Page 28.
Beard S, Rowe T, Fox, J. An analysis and evaluation of methods currently used to quantify the likelihood of existential hazards. Futures. 2020; 115 (102469): 102469-102469. doi: 10.1016/j.futures.2019.102469
Berg P, Mertz JE. Personal Reflections on the Origins and Emergence of Recombinant DNA Technology. Genetics. 2010; 184(1): 9-17. doi: 10.1534/genetics.109.112144
Block SM. “Living nightmares: biological threats enabled by molecular biology.” Pages 39-75 in: Drell DD, Sofaer AD, Wilson GD. The New Terror: Facing the Threat of Biological and Chemical Weapons. Stanford, CA: Hoover Institution Press, 1999.
Bostrom N. Superintelligence: Paths, Dangers, Strategies. 8th impression. Oxford: Oxford University Press, 2017. (Initial version published 2014.)
Casadevall A. The future of biological warfare. Microbial Biotechnology. 2012; 5(5): 584-587. doi: 10.1111/j.1751-7915.2012.00340.x
Chory EJ et al. Enabling high-throughput biology with flexible open-source automation.Mol Syst Biol. 2021; 17(3): e9942. doi: 10.15252/msb.20209942
Clancy T. Executive Orders. New York: Berkley Books, 1997.
Cohen SN. DNA cloning: A personal view after 40 years. Proc Nat Acad Sci USA. 2013; 110(39): 15521-15529. doi: 10.1073/pnas.1313397110
Dolan KA, Wang J, Peterson-Withorn C. Forbes World’s Billionaires List: the Richest in 2021. https://www.forbes.com/billionaires/
Dyson FJ. Is Science Mostly Driven by Ideas or by Tools? Science. 2012; 338: 1426-1427. doi: 10.1126/science.1232773
Ellman, Michael. Soviet repression statistics: some comments. Europe-Asia Studies. 2002; 54(7): 1151-1172.
Gibson DG, et al. Creation of a bacterial cell controlled by a chemically synthesized genome. Science. 2010; 329; 52-56. doi: 10.1126/science.1190719
Good IJ. Speculations concerning the first ultraintelligent machine. Advances in Computers. 1966; 6: 31-88. http://www.aeiveos.com/~bradbury/Authors/Computing/Good-IJ/SCtFUM.html
Greenberg A. The untold story of NotPetya, the most devastating cyberattack in history. Wired.com. Aug. 22, 2018. https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ = https://bit.ly/3wvQZ5B
Griffin DW. Atmospheric movement of microorganisms in clouds of desert dust and implications for human health. Clin Microbiol Rev. 2007; 20: 459-477. doi: 10.1128/CMR.00039-06
Hammond AM, Galizi R. Gene drives to fight malaria: curent state and future directions. Pathogens and Global Health. 2017; 111: 412-423.
Herfst S, et al. Airborne transmission of influenza A/H5N1 virus between ferrets. Science. 2012; 336:1534-1541. doi: 10.1126/science.1213362.
Hobby, Gladys L. Penicillin: Meeting the Challenge. New Haven: Yale, 1985. Pages 100-101, 158-159, 234.
Hoffman DE. The Dead Hand: the Untold Story of the Cold War Arms Race and its Dangerous Legacy. New York: Doubleday, 2009.
Imai M, et al. Experimental adaptation of an influenza H5 HA confers respiratory droplet transmission to a reassortant H5 HA/H1N1 virus in ferrets. Nature. 2012; 486: 420-428. doi: http://dx.doi.org/10.1038/nature10831
Jackson RJ, et al. Expression of mouse interleukin-4 by a recombinant ectromelia virus suppresses cytolytic lymphocyte responses and overcomes genetic resistance to mousepox. Journal of Virology. 2001; 75: 1205-1210. doi: 10.1128/JVI.75.3.1205-1210.2001
Jogalekar A. Chemistry and Biology: Kuhnian or Galisonian? The Curious Wavefunction blog. Dec. 20, 2012. https://bit.ly/3BkOoxG = https://blogs.scientificamerican.com/the-curious-wavefunction/chemistry-galisonian-rather-than-kuhnian/
Kelly J. The Great Mortality. New York: HarperCollins, 2005. Page 281.
Kim J, Koo B-K, Knoblich JA. Human organoids: model systems for human biology and medicine. Nature Reviews Molecular Cell Biology. 2020; 21: 571-584. doi: 10.1038/s41580-020-0259-3
Kohn GC. The Wordsworth Encyclopedia of Plague & Pestilence. New York: Facts on File, 1995.
Kupferschmidt K. How Canadian researchers reconstituted an extinct poxvirus for $100,000 using mail-order DNA. Sciencemag.org. July 6, 2017. doi: 10.1126/science.aan7069
Leventhal DS, et al. Immunotherapy with engineered bacteria by targeting the STING pathway for anti-tumor immunity. Nature Communications. 2020; 11: 2739.
Liu H-Y, Lauta KC, Maas MM. Governing boring apocalypses: a new typology of existential vulnerabilities and exposures for existential risk research. Futures. 2018; 102: 6-19. doi: 10.1016/j.futures.2018.04.009
Monie DD, et al. Synthetic and systems biology principles in the design of programmable oncolytic virus immunotherapies for glioblastoma. Neurosurg Focus. 2021; 50(2): E10. doi: 10.3171/2020.12.FOCUS20855
Naderi M, et al. Binding site matching in rational drug design: algorithms and applications. Brief Bioinform. 2019 Nov 27;20(6):2167-2184. doi: 10.1093/bib/bby078
Nakashima E, Torbati Y, Englund W. Ransomware attack leads to shutdown of major U.S. pipeline system. Washington Post. May 8, 2021. https://www.washingtonpost.com/business/2021/05/08/cyber-attack-colonial-pipeline/
Nassar, Sylvia. A Beautiful Mind: The Life of Mathematical Genius and Nobel Laureate John Nash. New York: Simon & Schuster Touchstone, 1998.
Noyce RS, Evans DH. Synthetic horsepox viruses and the continuing debate about dual use research. PLoS Pathog. 2018; 14: e1007025. doi: 10.1371/journal.ppat.1007025
Oldstone MBA. Viruses, Plagues, & History. Oxford: Oxford, 2010. Pages 61-63.
Palmer AW. On the Trail of the Robocall King. Wired. March 25, 2019. https://www.wired.com/story/on-the-trail-of-the-robocall-king/
Parks, Jerry M.; Smith, Jeremy C. “How to discover antiviral drugs quickly.” N Engl J Med. 2020; 382:2261-2264. doi: 10.1056/NEJMcibr2007042
Plum, Fred; Posner, Jerome B. The Diagnosis of Stupor and Coma. 3rd ed. Philadelphia: F.A. Davis, 1980. Page 9.
Putman P. Script kiddie: unskilled amateur or dangerous hackers? Accessed June 27, 2021. https://www.uscybersecurity.net/script-kiddie/
Sagan, Carl (ed.). Communication with Extraterrestrial Intelligence. Cambridge, MA: MIT Press, 1973.
Sandberg A, Bostrom N. “Global Catastrophic Risks Survey.” Technical Report #2008-1. Future of Humanity Institute, Oxford University. 2008. (Accessed June 27, 2021.) https://www.fhi.ox.ac.uk/reports/2008-1.pdf
Schlosser E. Command and Control: Nuclear Weapons, the Damascus Accident, and the Illusion of Safety. New York: Penguin, 2013.
Schrödinger E. Nature and the Greeks -and- Science and Humanism. Cambridge, UK: Cambridge University Press, 1996.
Schutgens F, Clevers H. Human Organoids: Tools for Understanding Biology and Treating Diseases. Annual Review of Pathology: Mechanisms of Disease. 2020; 15(1): 211-234. doi: 10.1146/annurev-pathmechdis-012419-032611
Sheikh A, et al. SARS-CoV-2 Delta VOC in Scotland: demographics, risk of hospitalization, and vaccine effectiveness. The Lancet. 2021: 397; 2461-2462
Shirley JM, de Jong YP, Terhorst C, Herzog RW. Immune Responses to Viral Gene Therapy Vectors.Molecular Therapy. 2020; 28(3): 709-722. doi: 10.1016/j.ymthe.2020.01.001
Sieburg HB, Baray C, Kunzelman KS. Testing HIV molecular biology in in silico physiologies. Proc Int Conf Intell Syst Mol Biol. 1993;1:354-361.
Sotos JG. Biotechnology and the lifetime of technical civilizations. International Journal of Astrobiology. 18 (5): 445-454. doi: 10.1017/S1473550418000447
Stewart, George R. Earth Abides. New York: Random House, 1949.
Strickland E. The robot revolution comes to synthetic biology. IEEE Spectrum. 2016; 53(12): 9-11. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7761863
Tainter, Joseph A. The Collapse of Complex Societies. Cambridge, UK: Cambridge University Press, 1988. Page 4.
Thacker EE, Timares L, Matthews QL. Strategies to overcome host immunity to adenovirus vectors in vaccine development. Expert Rev Vaccines. 2009 Jun; 8(6): 761-777. doi: 10.1586/erv.09.29
Torres P. Agential risks: a comprehensive introduction. Journal of Evolution and Technology. 2016; 26(2): 31-47. https://jetpress.org/v26.2/torres.htm
van Melle W, Shortliffe EH, Buchanan BG. EMYCIN: A knowledge engineer’s tool for constructing rule-based expert systems. Chapter 15 (pages 302-313) in:
Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. Reading, MA: Addison-Wesley, 1984. https://www.aaai.org/Papers/Buchanan/Buchanan17.pdf
Wade N. The origin of COVID: Did people or nature open Pandora’s box at Wuhan? Bulletin of the Atomic Scientists online. May 5, 2021. https://thebulletin.org/2021/05/the-origin-of-covid-did-people-or-nature-open-pandoras-box-at-wuhan/
Wang Y, et al. In silico ADME/T modelling for rational drug design. Q Rev Biophys. 2015 Nov;48(4):488-515. doi: 10.1017/S0033583515000190
Warrell DA. Snake bite. Lancet. 2010; 375: 77-88. doi: 10.1016/S0140-6736(09)61754-2
Wired Staff. WIRED25: Stories of People Who Are Racing to Save Us: Emily Leproust. Wired Magazine. November 2019.