The natural world yields many examples of adversarial advantages arising through evolution. These are testaments to biological arms races that have played out with complex dynamics, typically over macroscopic time scales. There is a wide range of advantages evolved by predators and prey including examples of animal coloration – camouflage that is static such as the yellow crab spider (Misumena vatia) and dynamic such as the cuttlefish (see Fig. 0(a),0(b)), bioluminescence (e.g. mid-water crustaceans and fireflies, Fig 0(e),0(f)), and mimicry (e.g. the Monarch and Viceroy butterflies, Fig 0(c),0(d)).
Contemporary human-made ecosystems also yield arms races that arise from adversarial competitions. In the U.S. automotive industry, firms engage in an arms race to develop innovative new products ahead of the competition. The relationship between innovations and firm performance has been analyzed from a coevolutionary perspective Talay et al. (See Appendix A.2). The circular economy and its related cradle-to-cradle idea describe manufacturer-customer relationships that cooperatively evolve while business ecosystems within the economy also evolve, though competitively Petrlic (2016). In an example from taxation, tax shelters prey upon unintended lapses of the tax code until defensive mediations address them. On occasion, however, these mediations either introduce new ambiguity or signal evaders to target a different weakness or inefficiency and the arms race continues Son of Boss (2018); Wright Jr (2013).
One of the contemporary crucibles of adversarial activity and arms races is cyberspace. It is a complex, human-made ecosystem that spans networks (including the Internet), edge devices, servers, clouds, supercomputers. Each sub-ecosystem has sets of resources that collectively are a source of contention. Each includes human actors in circumstances where their software is proxy for their part in cyber adversarial engagements. Actors can have conflicting objectives. From a security perspective one population of actors have benign intentions and the other has malicious intentions. Benign “cyber actors” are conforming within social and legal norms. They may control data which they need to keep confidential or private. They may operate networks, or websites in order to conduct their enterprise’s business – digital commerce. Different devices e.g. cellphones and laptops support everyday social and economic functions. Underpinning all these activities are computational hardware and software. Through attacks on these resources, malicious actors seek to disrupt, infiltrate and exfiltrate to steal, poison, extort so they can profit financially and advance their causes.
Focusing on network security, one pernicious kind of network attack is named Denial of Service (DOS). The goal of a DOS attack is to consume resources of a target so the target has none left to serve legitimate clients. At the extreme end of the range of volume are Distributed DOS (DDOS) attacks. These are extremely aggressive but more rare, likely due to the cost of setting them up. At a technical level, DDOS attacks are composed from a software toolkit. One set of software comprises multiple tactics that allow a network of compromised servers, i.e. a botnet, to be stealthily established. Another has tactics that enable, from a command and control location, a (human) controller to direct the bots to launch DOS attacks. These tactics can even react to defensive measures being deployed during an attack. The controller strategically selects the tactics and aims them at a target.
It is arguable that DDOS attacks are analogous to population members of a species, with tactical variation akin to biological variation. This population arguably undergoes evolutionary adaptation because ineffective attacks are not reused and effective ones undergo adaptation in order to circumvent defensive measures (i.e. reproductive selection and variation are present). With each new generation of attacks, the defensive measures themselves adapt or evolve then they face the next round of adaptation based on the toolkit. The attacker’s general goal is to minimize its use of resources while maximizing its denial of service and the defender’s general goal is to minimize the impact of the attack. In terms of dynamics, an escalating, adversarial arms race around engagements based on the conflicting objectives consequently emerges.
There is reliable and thorough documented evidence of DDOS attacks and counter measures that support an evolutionary interpretation, at a high level of abstraction Antonakakis et al. (2017). MIRAI Symantec Security Response (2016) is a notorious DDOS, receiving publicity for two high profile attacks: Krebs Brian Krebs (2016) and DYN Scott Hilton (2016). From its inception to the present, MIRAI has adapted its tactics and targets to thwart defensive counter-measures, see Fig. 2. Fig. 3 shows a Red Queen like DDOS dynamics – while evolution is churning beneath the surface, the number of attacks and their diversity of volume barely changes. Defensive gains are relative and transient.
DOS attacks sit amongst an alarming array of other cyber attack types, such as Advanced Persistent Threats (APT) and Ransomware (see Appendix A.3) that exhibit intelligent and agile attacks as well as attack versus defense arms race dynamics. Across the entire cyber ecosystem there is ceaseless evolution of kinds, arguably species, of attacks and co-evolving responses. Beyond their inconvenience, attacks risk lives, have financial costs, threaten businesses, impinge upon privacy and disrupt legitimate activity.
One way to improve defenses requires understanding better how an attacker behaves and how competing attacks and defenses react to one another. Feeding back information on what could happen is extremely valuable. It elucidates potential or past behaviors. For defensive design, knowing what could happen can help by identifying unforeseen or particularly critical attacks. Knowledge of arms race dynamics can inform the adversarially hardening of defensive designs ahead of time.
Understanding what could happen requires examining, actually playing out, or simulating, the dynamics of ongoing engagements between multiple parties with varying behaviors. Genetic programming (GP) can meet these requirements. It is ideally suited to represent the behavior of an adversary when it engages an opponent. Within a competitive evolutionary algorithm, GP representations can serve as the units of its selection and variation and GP operators can serve as the means of generating new solutions from existing ones.
Cyber security needs a way of exploring and executing attack versus defense engagements. GP is a paradigm that handles the performance selection and variation of units that behave! It can explore by directly manipulating executable functions without syntactic destruction or explicit program knowledge. It can express behavioral solutions drawn from a search space of hierarchical, varying length, executable structures and it has genetic operators that are able to blindly, at the representation level, generate novel syntactically correct solutions. Cyber security needs an expressively powerful way to describe abstracted attacks and defenses in a way that they are also able to actually compete against each other. GP can fulfill this because it can accommodate any abstraction that can be described as a set of functions and terminals or a computational language expressed by a grammar.
Combinations of adversarial evolutionary algorithms and GP matched with the rich problem domain of cyber security thus meld into an increasingly critical intersection with an agenda of compelling and still untapped scope, see Fig. 4. We name this topic Adversarial Genetic Programming for Cyber Security. It is not a new topic; it is the delineation of extant and current research approaches. We call attention to them because cyber security is urgent and escalating in challenge, and despite contributions to date, many novel approaches are required. At the 20 year mark of GP, when it is no longer necessary to show that GP works, it is timely to direct mature and nascent GP methods plus GP researchers in the direction of this critical domain with real world problems. Can search-based software engineering techniques gain traction on security exploits by considering adaptation at actual code level? Can GP enable the exploration of cyber space actors’ goals and resulting plans, thereby addressing their intentions and cognitive reasoning? These questions and ones in between represent an opportunity for GP research that matters.
Paper Roadmap Having motivated Adversarial Genetic Programming for Cyber Security, in Section 2, we first succinctly cover GP and evolutionary algorithms (EAs). To consider adversarial evolution, we then describe competitive coevolutionary algorithms to remind readers of their complexity. We end the section by showing how GP and a coevolutionary algorithm can be combined. In Section 3
we survey prior evolutionary computation research into Artificial Intelligence (AI) and games (relevant because games are also competitive contexts), and software engineering (related because tests have been evolved to compete with program solutions to spur correctness). We then survey research around and within our central topic with Table1 summarizing. Sections 2 and 3 inform the design motivations underlying a framework we present in Section 4. Named RIVALS, it combines competitive coevolutionary algorithms and GP for network security design. We describe three systems drawing on the framework and a component that extracts useful solutions to support the design decisions of a network defense designer. We draw to a close with a summary and a broad discussion of possible paths forward within this exciting, important paradigm.
2 Basis Algorithms for Adversarial Evolution
Variable length genotypes, such as the hierarchical tree structures introduced by Koza Koza (1992), define large search spaces and improve the flexibility and power of Evolutionary Algorithms conducted by executable structure search Kinnear et al. (1999); O’Reilly and Angeline (1997)
. GP is arguably defined by its distinctive structures that are variable length and executable plus its operators that preserve syntactic correctness while changing genotype size and structure. An Evolutionary Algorithm (EA) can evolve individual solutions in the form of executable structures, fixed length genotypes such as the bit strings used by Genetic Algorithms (GAs)Goldberg (1989) are inflexible.
Biological coevolution refers to the influences two or more species exert on each other’s evolution Rosin and Belew (1997). A seminal paper on reciprocal relationships between insects and plants coined the word “coevolution” Ehrlich and Raven (1964). Coevolution can occur in the context of cooperation, where the species experience mutual benefit and adaptation, or in the context of competition, where the species negatively interact due to constrained resources that are under shared contention or a predator-prey relationship.
EAs usually abstract the evolutionary process while ignoring coevolution. To evaluate the quality of an individual, they apply an a-priori defined fitness function and this function does not reflect the possible interactions between another individual or a dynamic environment. Optimality can be formulated by an absolute rather than relative objective. Coevolutionary algorithms extend EAs by basing the fitness of an individual on its interactions with other individuals or the environment. They mimic the coupled species-to-species interaction mechanisms of natural coevolution in order to solve a niche of search, optimization, design, and modeling problems Antonio and Coello (2018); Popovici et al. (2012b); Rosin and Belew (1997); Sims (1994).
Similar to the biology, coevolutionary algorithms are categorized as cooperative or competitive. Cooperative coevolutionary algorithms abstractly model the beneficial interaction between populations or in environments with time dependent factors Krawiec and Heywood (2016). They are frequently set up with multiple populations each solving a distinct sub-problem of a larger problem. We continue this section by describing competitive coevolutionary algorithms in more detail as it is central to adversarial behavior.
2.1 Competitive Coevolutionary Algorithms
A basic competitive coevolutionary algorithm evolves two coupled populations, each with conventional selection and representation-aligned variation (crossover and mutation) operators which suit the representation of the population’s genotype. One population comprises what is commonly called tests and the other solutions. We refer to individuals in a competitive coevolutionary algorithm generally as adversaries. In each generation, different competitions are formed by pairing up a test and a solution drawn from their respective populations. This couples the two population as they share a fitness evaluation component. We shall shortly describe different ways in which these competitions can be set up.
A test competes to demonstrate the solution as incorrect; this is its objective. The solution competes to solve the test correctly; this is its objective. In a security setting, it may be more apt to translate from test and solution to attack and defense as well as to refer to engagements rather than competitions. Fitness is calculated over all of an adversary’s engagements. The dynamic of the algorithm, driven by conflicting objectives and guided by performance-based selection and random variation can gradually produce better and more robust solutions (i.e defenses) Rosin and Belew (1997); Sims (1994). Generally, competitive coevolutionary algorithms suit domains in which there is no exogenous objective measure of performance but wherein performance is relative to others. These have been called interactive domains in Popovici et al. (2012b) and include games and software engineering. In most domains a competition is often computationally expensive because it involves simulation or a complex testbed. We next describe how competition structuring addresses this challenge.
For efficiency, the algorithm designer tries to minimize the number of competitions per generation while maximizing the accuracy of its fitness estimate of each adversary. The designer is able to control how competitions are structured and how many competitions are used to estimate the fitness of an adversary. Assuming one or both populations are of size, two extreme structures are: one-vs-one, each adversary competes only once against a member of the opposing population, see Fig. 4(a) and 4(d), and all-vs-all, each adversary competes against all members of the opposing population, see Fig. 4(b) and 4(e). One-vs-one has minimal fitness evaluations111computational cost is shown for two populations. () and strong competition bias. In contrast, all-vs-all has a quadratic number of fitness evaluations, yielding a high computational cost, but weaker competition bias Sims (1994). Other structures provide intermediate trade-offs between computation costs and competition bias, e.g. a tournament structure ranks individuals based on different rounds of peer competitions, see Fig. 4(c).
In Mitchell (2006), adversaries termed hosts and parasites are placed on a MM grid with a fixed neighborhood (size ) and one host and parasite per cell. The structure of the competitions is competition among all competitors in the neighborhood. Fitness evaluations are reduced to by this. An adversary has an outcome for each competition. We next discuss how these outcomes can be united to assign it a fitness score.
At each generation an adversary needs to be assigned a fitness score derived from some function of its performance outcomes in its competitions. For example, the function can average the performance outcomes or use the maximum, minimum, or median outcome Axelrod (1984). Other approaches have been defined depending on the specific problem domain, e.g. Fogel (2001); Angeline and Pollack (1993); Cardona et al. (2013); Lim et al. (2010); Luke and others (1998); Togelius et al. (2007). A more formal approach, using the solution and test perspective, describes fitness assignments as solution concepts Popovici et al. (2012b). Solution concepts include: best worst case, maximization of expected utility, Nash equilibrium, and Pareto optimality.
Coevolutionary algorithms are more complex and challenging to direct toward some expected outcome than a single population EA. We next explain the pathologies they exhibit and describe accepted remedies for them.
Pathologies and Remedies
The interactive aspect of fitness implies that a competitive coevolutionary algorithm lacks an exact fitness measurement. While EAs use a fitness function that allows an individual to be
compared and globally ranked, in coevolutionary algorithms two members of the same population, during selection, may be imprecisely compared when they did not compete against the same opponents. Adding more imprecision, regardless of the function that computes a fitness score, an adversary’s score may change if it later competes against different opponents. These properties imply that any ranking of individuals is a noisy estimate. This estimation can lead to competitive coevolutionary algorithms pathologies that
limit the effectiveness of their arms race modeling Popovici et al. (2012b). Furthermore, pathologies can arise from competitive imbalance, particularly when the behavior space of one population is not the same as that of the other. This asymmetry is considered and addressed in Olsson (2001).
disengagement –occurring when opponents are not challenging enough, eliminating an incentive to adapt,
or when opponents are too challenging and progress becomes
impossible Cartlidge and Bullock (2004),
cyclic dynamics – generally appearing in transitive domains (A beats B, B beats C and C beats A) as oscillation of the evaluation metric
generally appearing in transitive domains (A beats B, B beats C and C beats A) as oscillation of the evaluation metricHornby and Mirtich (1999), focusingor overspecialization – arising when an adversary evolves to beat only some of its opponents, while neglecting the others Bucci (2007), and coevolutionary forgetting –occurs when an adversary’s ability to beat an opponent evolves away Boyd (1989).
In order to remediate these pathologies, assorted methods have been proposed Bongard and Lipson (2005); Boyd (1989); Bucci (2007); Cartlidge and Bullock (2004); Ficici (2004); Hornby and Mirtich (1999); Krawiec and Heywood (2016); Williams and Mitchell (2005). These center on archives or memory mechanisms, maintaining suboptimal individuals in the population, using a neighborhood spatial structure, and/or monitoring the progress of the algorithm.
With the basics of GP and competitive coevolutionary algorithms introduced, we are now able to present a combined competitive coevolutionary and GP algorithm.
2.2 GP and Adversarial Evolution
Combining GP and competitive coevolutionary algorithms enables cyber security arms races to be replicated by evolving executable adversaries. An example of an alternating GP competitive coevolutionary algorithm Arcuri and Yao (2014) is shown in Algorithm 1. The adversaries (attacker/defender) are coupled and evolve in an alternating manner. First, the adversaries are initialized. Then, at each generation, a new attacker population, , is selected, mutated, recombined and evaluated against the current defense population, . Based on the evaluation the attackers are replaced. Then, control reverts to the defender population so it can evolve to conclude the generation. This algorithm allows more attack evolution than defensive evolution, reflecting what can happen in real cyber systems. A note of caution about this approach in the context of algorithms, rather than reality, is offered by Milano et al. (2019) where it is observed that moderate environmental variation across generations is sufficient to promote the evolution of robust solutions.
There are some open challenges when applying Adversarial Genetic Programming for Cyber Security. Some of them are inherent in the use of EAs and adversarial evolution, such as the subjective solution evaluation, i.e., individuals interact with different opponents and the fitness is based on these, so it only provides a relative ordering Popovici et al. (2012a); Popovici and Winston (2015); Popovici (2017). Other amplified challenges are the evaluation cost, since the individual’s fitness is calculated based on its interaction with other individuals which might require high computational costs Bari et al. (2018); Kim and Cho (2001); Liskowski and Krawiec (2016); Mitchell (2006); Liskowski and Krawiec (2017); Finally, combining GP and adversarial evolution generates complex models that complicate the algorithm operator and parameter selection.
This section has described algorithms that form the foundation of computational adversarial evolution. In the next section we describe prior Evolutionary Computation work that use GP, competitive coevolutionary algorithms or a combination of them in the domains of AI and Games, security, and Software Engineering.
3 Related Work
|Reference||Algorithm||# of Populations||Representation||Competition Structure||Fitness Scoring|
|Axelrod, et al. Axelrod (1984); Axelrod and others (1987)||Comp Coev GA||One||Bit strings||Tournament||Maximum expected utility|
|Crawford-Marks et al. Crawford-Marks et al. (2004)||Comp Coev GP||Two: players and smart-balls||Trees||One vs subset||Maximum expected utility|
|Harper Harper (2014)||Comp Coev GP||One||
Variable integer vector
|One vs a subset||Maximum expected utility|
|Keaveney & O’Riordan Keaveney and O’Riordan (2011)||Comp Coev GP||One||Trees||One vs subset||Maximum expected utility|
Lim et al. Lim et al. (2010)
|GP||One||Trees||Playing games||Pareto optimality|
|Luke et al. Luke and others (1998)||Comp Coev GP||One||Trees||One vs one||Maximum expected utility|
|Miles et al. Miles et al. (2007)||Comp Coev GA||One||Bit strings||One vs one||Maximum expected utility|
|Garcia et al. Garcia et al. (2017)||Comp Coev GP||Two: Defender & Attacker||Integers & BNF Grammar||NA||Multiple comparison|
|Hemberg et al. Hemberg et al. (2018)||Comp Coev GP||Two: Defender & Attacker||Integers & BNF Grammar||All vs All||Pareto optimality|
|Hingston & Preuss Hingston and Preuss (2011)||Comp Coev GA||Four: two of learners and two of tests||Pairs of real numbers||One vs all||Best-worst case|
|Kewley & Embrechts Kewley and Embrechts (2002)||Comp Coev GA||Eight: four friendly and four enemy forces||Vector of four parameters||One vs a subset||Maximum expected utility|
|McDonald & Upton McDonald and Upton (2005)||Comp Coev GA||Two: red and blue teams||Vector of parameters||One vs all||Force Exchange Ratio|
|Ostaszewski et al. Ostaszewski et al. (2007)||Comp Coev AIS||Two: detectors and anomalies||Vector of parameters||One vs subset||Maximum expected utility|
Rush et al. Rush et al. (2015)
|Comp Coev EA||Two: defenders and attackers||Not specified||One vs subset||Maximum expected utility|
|Service & Tauritz Service and Tauritz (2009)||Comp Coev GA||Two: system hardenings and system faults||Bit strings: unified power flow controller installation||One vs a subset||Nash equilibrium|
|Suarez-Tangil et al. Suarez-Tangil et al. (2009)||GP||One||Trees: intrusion detection rules||Risk assessment||Maximum expected utility|
Winterrose & Carter Winterrose and Carter (2014)
|GA||One: strategies||Bit strings: finite states machine||Fixed scenarios||Success against the defender|
|Arcuri & Yao Arcuri and Yao (2007)||Comp Coev GP||Two: programs and tests||Trees||One vs all||Nash equilibrium|
|Adamopoulos et al. Adamopoulos et al. (2004)||Comp Coev GA||Two: programs and tests||Sequences of mutant programs and tests||One vs all||Maximum expected utility|
|Oliveira et al. de Oliveira et al. (2013)||Comp Coev GA||Two: programs and tests||Sequences of mutant programs and tests||One vs all||Pareto optimality|
|Wilkerson & Tauritz Wilkerson and Tauritz (2010)||Comp Coev GP||Two: programs and tests||Trees and lists||One vs a subset||Maximum expected utility|
We now turn our attention to prior work relating to the theme of adversarial contexts. We organize by application domain, starting with domains which are competitive in nature but not cyber security: AI and games (3.1) and Software Engineering (3.2). Most examples take technical approaches that use competitive coevolutionary algorithms or GP. They provide essential illustrations of how the algorithms can be used as building blocks. Table 1 catalogs the related work by domain, algorithm, representation, competition structure, and fitness scoring. Fig. 6 illustrates how they intersect thematically.
3.1 AI and Games
We find related work in AI and games because many games are competitive and game playing is a fertile research ground for investigating adversarial learning. We embrace a broad definition of game – encompassing board games, e.g. Pollack and Blair (1998), video game playing, e.g. Keaveney and O’Riordan (2011); Sipper (2011) and social science games e.g. Axelrod (1984).
One of the first competitive coevolutionary algorithms was used to find efficient strategies for the Iterated Prisoner’s Dilemma Axelrod (1984); Axelrod and others (1987). This seminal project used a single population and it structured its strategy competition by running a tournament among the population members. A single population is typical for symmetric games, i.e. games where each player has the same set of moves (behavioral repertoire) and objectives. We find symmetric board game projects that use single population competitive coevolutionary algorithms to evolve Tic-Tac-Toe, Othello and Backgammon players. The Tic-Tac-Toe project represented players with GP Angeline and Pollack (1993) while both Othello and Backgammon projects used temporal difference learning, which is a gradient-based local search method Krawiec and Szubert (2011); Pollack and Blair (1998); Szubert et al. (2009, 2013); Smith (1994). A subsequent work used symmetric competitive coevolution in games and demonstrated impressive results through self-play Chellapilla and Fogel (1999). A noteworthy system at the intersection of competitive coevolutionary algorithms and GP used a single population of agents cooperating as a team to compete for the “RoboCup97” soccer competition Luke and others (1998). Soccer, being symmetric, would allow the system to potentially train against itself. In Sipper et al. (2007) the authors designed an evolutionary strategizing machine for game playing and other contexts. In Smith and Heywood (2017), the authors coevolved strategies to solve Rubik’s Cube where the test population evolved the Rubik’s Cube configurations while the learner population evolved GP individuals to solve the Cube configurations. An ablation study demonstrated the contribution of competitive coevolution over random replacement of Cube configurations.
Cyber security adversaries are arguably asymmetric. In asymmetric games where opponents have different objectives and move sets, we find multi-population competitive coevolutionary algorithms. These include a competitive coevolutionary algorithm with evolutionary strategies that evolves Pac-Man versus Ghost Team players Cardona et al. (2013) and a PushGP system coevolving player versus smart-ball for the game “Quidditch”. The latter is noteworthy for its intersection of a competitive coevolutionary algorithm and GP.
In some asymmetric games one adversary is external to the learning system and quite complex. This drives single or multi-population coevolutionary algorithm systems. A competitive coevolutionary algorithm and GP study compared using a single population to nine populations when evolving controllers for a car racing game Togelius et al. (2007). The multi-population approach generally produced better controllers. Similarly, real time strategy games (which provide challenging AI problems Lara-Cabrera et al. (2013); Nogueira-Collazo et al. (2016)) have applied a single-population competitive coevolutionary algorithm and GP to develop automated players that use a progressive refinement planning technique Keaveney and O’Riordan (2011). Other work Miles et al. (2007) coevolved real time strategy players by representing the AI agents with Influence Maps DeLoura (2001)
. A real-time Neuroevolution of Augmenting Topologies (rtNEAT) approach evolved neural networks in real time while a game was playedStanley et al. (2005). Behavior Trees (BT) were introduced to encode formal system specifications Alex (2007), but have also represented game AI behavior in commercial games Nicolau et al. (2017). GP evolved BT controllers for “Mario” Perez et al. (2011) and “DEFCON” Lim et al. (2010) games. Another study used Grammatical Evolution to generate Java programs to compete in “Robocode”. The players were represented by programs Harper (2014) and a spatial and temporal competitive coevolutionary algorithm and GP were used.
3.2 Software Engineering and Software Testing
Software testing called fuzzing arguably started with Hanford (1970) which used grammars to generate program inputs for tests. This transitioned to direct automated random testing. One relevant security example is Godefroid et al. (2005) that searched for bugs in a security protocol. Later, constraint based automatic test data generation was used to generate tests to verify if software variants were safe from malicious manipulation DeMilli and others (1991).
For similar purposes, the search based software engineering community uses evolutionary computation. Coevolution is used in SBSE with GA representations, see e.g. Adamopoulos et al. (2004); Anand et al. (2013); de Oliveira et al. (2013). The community has used competitive coevolutionary algorithms and GP to coevolve programs and unit tests from their specification, e.g. Arcuri and Yao (2007); Wilkerson and Tauritz (2010); Arcuri and Yao (2014). Another study, Barr et al. (2015), used coevolution to distinguish correct behavior from incorrect.
3.3 Cyber Security
We now turn to the domain of cyber security. There are a number of examples where, without any computational modeling or simulation, cyber security dynamics are described as evolutionary or as arms races. According to Willard (2015), cyber security is the task of minimizing an attack surface over time. The attack surface is the portion of a system which has vulnerabilities. Attackers attempt to influence the system’s nominal state and operation by varying their interactions with the attack surface by stealth and non-compliance (see Appendix A.4). Cyber security perimeter protection (e.g. a firewall) is a battle fought and lost. Attackers are able to now gain access and defenders must resort to strategies that assume the attacker is present but hidden. One tactic to detect their presence is deception. For this a “honeypot” that entraps an attacker by imitating its target is commonly deployed. This arms race escalates ad infinitum as attackers then anticipate what the defenders have anticipated (e.g. the honeypots) and so on.
Retrospective security event reports also document coevolution. For example, Gupta et al. (2009) is an empirical study of malware evolution. Arguments for employing nature-inspired technologies for cyber security that mention how biological and ecological systems use information to adapt in an unpredictable world include Crandall et al. (2009); Ford et al. (2006); Iliopoulos et al. (2011); Mazurczyk et al. (2015); Sagarin and Taylor (2012).
A selected set of cyber security research that takes technical approaches is described in the Security subsection of Table 1. This set include works that we would not consider Adversarial Genetic Programming for Cyber Security: one work uses a genetic algorithm and neither coevolution or GPWinterrose and Carter (2014) and six others use are adversarial in nature, i.e. they use coevolution, but not GPHingston and Preuss (2011); Kewley and Embrechts (2002); McDonald and Upton (2005); Ostaszewski et al. (2007); Rush et al. (2015); Service and Tauritz (2009). The remaining three projects fit within the topic of Adversarial GPSuarez-Tangil et al. (2009); Garcia et al. (2017); Hemberg et al. (2018). The research within this set, along with other related work, can also be distinguished by what specific application it addresses in the domain of cyber security:
- Moving target defense (MTD)
techniques seek to randomize components to reduce the likelihood of a successful attack, reduce the attack lifetime, and diversify systems to limit the damage Evans et al. (2011); Okhravi et al. (2014). A MTD study investigates an adaptable adversarial strategy based on Prisoners Dilemma in Winterrose and Carter (2014). Strategies are encoded as binary chromosomes representing finite state machines that evolve according to GA. The study has one adaptive defender population in GA and few fixed scenarios.
- Network Defense Investigation
is studied with the coevolutionary agent-based network defense lightweight event system (CANDLES) Rush et al. (2015) a framework designed to coevolve attacker and defender agent strategies with a custom, abstract computer network defense simulation. The RIVALS network security framework, elaborated in Section 4 supports three studies into respectively DDOS, deceptive and isolation defense.
- Self-Adapting Cyber Defenses
are for example the Helix self-regenerative architecture Le Goues et al. (2013). Helix shifts the attack surface by automatically repairing programs that fail test cases using Software Dynamic Translation, e.g. Strata Scott and Davidson (2001). Another example of automated fault analysis and filter generation within a system is named “FUZZBUSTER” Musliner et al. (2013); Musliner D. J. (2014). Like Helix, FUZZBUSTER is designed to automatically identify software vulnerabilities and create adaptations that protect those vulnerabilities before attackers can exploit them. Both FUZZBUSTER and Helix use a GP system called GenProg Weimer et al. (2010) for automatically fixing code.
- Physical Infrastructure defense
is studied in terms of how network components can be made resilient in Service and Tauritz (2009).
- Anomaly detection
attempts to discern network or other activity behavior that is out of the ordinary or that differs from normal. In the auto-immune system computational paradigm, normal has been characterized as “self” Forrest et al. (1996). Anomaly detection has been studied as a one class learning problem by Ostaszewski et al. (2007)
who use the artificial immune systems paradigm and coevolution for search. Attempts to build GP anomaly detectors have mostly assumed labeled data sets and they evolve a classifier that labels outputs as normal or not. They frequently encounter a class imbalance issue. This is explicitly addressed inSong et al. (2005). Later work adopted an explicitly Pareto archive formulation of competitive coevolutionLemczyk and Heywood (2007).
- Vulnerabilty Testing
Vulnerability testing involves testing a system with modifications to known exploits or attack vectors. Examples of developing mimicry attacks to test for vulnerability are found in Kayacık (2009); Kayacık et al. (2011). The approach used the alarm signal for coevolution of a GP exploit generator. Moreover, GP was limited to instructions that were in legitimate applications, forcing GP to search for exploits described in terms of instructions used by legitimate applications. Others have also used GP to coevolve port scans against the ’Snort’ intrusion detector with the objective of evolving port scans that demonstrate holes in the IDS. See for example LaRoche et al. (2009). Vulnerability testing for malware in mobile applications using coevolution appears in Bronfman-Nadas et al. (2018).
- Malware detection
- Intrusion detection
Most examples of intrusion detection are addressed as a multi-class classification problem where activity such as network traffic must be labeled as normal, denial of service, botnet or something else. Multi-class classification has been studied using GP without coevolution. For example, Suarez-Tangil et al. (2009) optimizes intrusion detection rules. Others study botnet detection with GP under streaming data label budgets and class imbalance Khanchi et al. (2018). With both intrusion detection and anomaly detection, one challenge is obtaining a dataset that truly reflects the network or any other cyber environment. For example, a widely used dataset known at KDD’99 has been criticized for its artificially generated normal data, which did not encompass the diversity in real normal behavior, see Lippmann et al. (2000); Mahoney and Chan (2003) for more details. It is also challenging to label and maintain datasets, see Garcia et al. (2014) as an example. The persistent nature of these challenges provides additional motivation for RIVALS.
- Battle management
has historically noteworthy work by Applegate et al. (1990) which focuses on security in battle management. It was novel in considering semi-autonomous control of several intelligent agents; plan adjustment based on developments during plan execution; and the influence of the presence of an adversary in devising plans. Similar but contemporary work on a computational military tactical planning system uses fuzzy sets and competitive coevolutionary algorithm with a battlefield tactics simulator for decision support Kewley and Embrechts (2002).
- Red teaming
is a technique utilized by the military operational analysis community to check defensive operational plans by pitting actors posing as attackers (a red team) against the defense (a blue team). This activity has evolved from human teams to teams of programmers overseeing automated attacks tactics and defensive measures. See DARPA (2016) and studies Abbass (2015); Nettles (2010) trying to automate the exercise to use less manpower, or use it more efficiently with agent-based models Beard (2011); McDonald and Upton (2005); Wood et al. (2000); Yuen (2015); Zeng et al. (2012); Hingston and Preuss (2011).
The Next Section
Prior work in AI and games, cyber security and software engineering domains is helpful in showing how competitive coevolutionary algorithms and GP can be used in adversarial contexts. It also provides a modest number of examples of GP combined with a competitive coevolutionary algorithm. We now proceed to present a detailed example of research within Adversarial Genetic Programming for Cyber Security. Named RIVALS, it is a software framework for studying network security. RIVALS addresses an aspect of central importance to the 2016 DARPA grand challenge in cyber security named “The World’s First All-Machine Hacking Tournament”DARPA (2016). This aspect that adaptation of both sides of a cyber security battle ground must be anticipated and that eventually posture reconfiguration needs to be fully automated.
Consistent with the Adversarial Genetic Programming for Cyber Security paradigm, our goal is to study the dynamics of cyber networks under attack by computationally modeling and simulating them. Ultimately we aim to provide defenders with information that allows them to anticipate attacks and design for resilience and resistance before their attack surfaces are attacked. We exploit competitive coevolutionary algorithms and GP for these purposes within a framework named RIVALS O’Reilly and Erik (2018); Pertierra (2018); Prado Sanchez (2018).
Conceptually, the framework consists of three elements: adversaries, engagements and their environments, and competitive coevolution. These fit together as two connected modules – one executing the coevolutionary algorithm and executing the engagements, and the other executing the engagements with the competitive coevolutionary algorithm directing attack and defense engagement pairings, see Fig. 7,.
We now proceed to describe each of these elements and RIVALS’ use cases.
4.1 Elements of Rivals
The system consists of two adversarial populations – attacks and defenses. The primary objective of an attack is to impair a mission by maximizing disruption of some resource. The primary objective of a defense is to complete a mission by minimizing disruptions. Both attack and defense can have secondary objectives based on the cost of their behavior. We design attack and defense behavior by defining grammars that can express the different variations of attacks and defenses. An attack or defense is a “sentence” in the grammar’s language. We implement a rewriting process (“generator”) to form sentences from the grammar O’Neill and Ryan (2003), see Fig. 8. The sentences (i.e. attacks and defenses) are functions that are executable in the layers of the engagement environment.
The grammar has Backus Naur Form (BNF) and is parsed to a context free grammar structure. The (rewrite) rules of the generator express how a sentence, i.e. attack or defense, can be composed by rewriting a start symbol. The adversaries are represented with variable length integer vectors. The generator decodes these vectors to control its rewriting. As a result, different vectors generate different attacks or defenses. For different use cases, it is only necessary to change the BNF grammar, engagement environment and fitness function of the adversaries. This modularity, and reusability of the parser and rewriter are efficient software engineering and problem solving advantages. The grammar additionally helps communicate to the designer how the system works and its assumptions, i.e. threat model. This enables conversations and validation at the domain level with experts and increases the confidence in solutions from the system.
Engagements and the Engagement Environment
An engagement is an attack on a defense. Engagements take place in an engagement environment which is initialized with a mission to complete and a set of resources such as network services that support the mission. The defense is first installed in the environment and then, while the mission runs, the attack is launched. The scenario (mission and resources) and attacks are then executed. Engagements have outcomes that match up to objectives; they are phrased in terms of mission completion (primary objective) and resource usage (second objective). Implementation-wise, the engagement environment component can support a problem-specific network testbed, simulator or model. Mod-sim is appropriate when testbeds incur long experimental cycle times or do not abstract away irrelevant detail.
RIVALS maintains two populations of competing attackers and defenders. It calculates the fitness of each population member (in both attack and defense populations) by assessing its ability to successfully engage one or more members from the adversarial population, given its objective(s). It also directs selection and variation. RIVALS Prado Sanchez (2018); Pertierra (2018) utilizes different coevolutionary algorithms to generate diverse behavior. The algorithms, for further diversity, use different “solution concepts”, i.e. measures of adversarial success. However, engagements are often computationally expensive and have to be pairwise sampled from two populations at each generation, a number of enhancements enables efficient use of a fixed budget of computation or time.
Solely emulating or simulating cyber arms races is not sufficient to practically inform the design of better, anticipatory defenses. In fact, competitive coevolution poses general challenges when used for design optimization. The following ones in RIVALS make it difficult to present a designer with clear information derived solely from multiple simulation runs Sanchez et al. (2018); Prado Sanchez (2018):
Attacks (and defenses) of different generations are not comparable because fitness is based solely on the composition of the defense (attack) population at each generation. So no clear champion emerges from running the algorithm.
From multiple runs, with one or more algorithms, it is unclear how to automatically select a “best” attack or design.
To this end, RIVALS provides an additional decision support component, named ESTABLO Prado Sanchez (2018); Sanchez et al. (2018), see Fig. 9. At the implementation level, the engagements and results of every run of any of the system’s coevolutionary algorithms are cached. Later, offline, ESTABLO filters these results and moves a subset to its compendium. To prepare for the decision support analysis, it then competes all the attacks in the compendium against all defenses and ranks them according to multiple criteria, e.g. maximum-expected utility, best-worst case. For the defensive designer, it also provides visualizations and comparisons of adversarial behaviors to inform the design process.
The RIVALS framework supports use cases – investigations into arms races of a particular cyber network context. They each interface with the framework through a set of domain specific information. For each use case, grammars define the behavioral space of the adversaries, objectives define the goals of the adversaries for scoring fitness and the threat environment describes its engagement environment. For each use case, different parameters to control the coevolutionary algorithm are also available. Fig. 10 depicts the high level decomposition of the framework and shows how a use case interfaces with it. Table 2 presents the domain specific information for three use cases. By name, these are:
The following sections elaborate on these use cases.
4.2 Diffracted– DOS Attacks on Peer-to-Peer Networks
A peer-to-peer (P2P) network is a robust and resilient means of securing mission reliability in the face of extreme distributed denial of service (DDOS) attacks. DIFFRACTED Garcia et al. (2017), assists in developing P2P network defense strategies against DDOS attacks. Attack completion and resource cost minimization serve as attacker objectives. Mission completion and resource cost minimization are the reciprocal defender objectives. DDOS attack strategies are modeled with a variety of behavioral languages.
A simple language e.g. allows a strategy to select one or more network servers to disable for some duration. Defenders choose one of three different network routing protocols: shortest path, flooding and a peer-to-peer ring overlay to try to maintain their performance. A more complex language allows a varying number of steps over which the attack is modulated in duration, strength and targets. It can even support an attack learning a parameterized condition that controls how it adapts during an attack, i.e. “online”, based on feedback it collects from the network on its impact. Defenders have simple languages related to parameterizations of P2P networks that influence the degree to which resilience is traded off with service costs. A more complex language allows the P2P network to adapt during an attack based on local or global observations of network conditions.
An example of attackers from ESTABLO on a mobile resource allocation defense used in DIFFRACTED Sanchez et al. (2018) is shown in Fig. 11. The mobile asset placement defense challenge is to optimize the strategic placement of assets in the network. While under the threat of node-level DDOS attack, the defense must enable a set of tasks. It does this by fielding feasible paths between the nodes that host the assets which support the tasks. A mobile asset is, for example, mobile personnel or a software application that can be served by any number of nodes. A task is, for example, the connection that allows personnel to use a software application.
4.3 Avail- Availability Attacks on Segmented Networks
Attackers often introduce malware into networks. Once an attacker has compromised a device on a network, they spread to connected devices, akin to contagion. AVAIL considers network segmentation, a widely recommended defensive strategy, deployed against the threat of serial network security attacks that delay the mission of the network’s operator Hemberg et al. (2018) in the context of malware spread.
Network segmentation divides the network topologically into enclaves that serve as isolation units to deter inter-enclave contagion. How much network segmentation is helpful is a tradeoff. On the one hand, a more segmented network provides lower mission efficiency because of increased overhead in inter-enclave communication. On the other hand, smaller enclaves contain compromise by limiting the spread rate, and their cleansing incurs fewer mission delays. Adding complexity, given some segmentation, a network operator can monitor threats and utilize cleansing policies to detect and dislodge attackers, with the caveat of cost versus efficacy.
AVAIL assumes an enterprise network in carrying out a mission, and that an adversary employs availability attacks against the network to disrupt it. Specifically, the attacker starts by using an exploit to compromise a vulnerable device on the network. This inflicts a mission delay when a mission critical device is infected. The attacker moves laterally to compromise additional devices to further delay the mission. Fig 12 shows AVAIL.
AVAIL employs a Monte Carlo simulation model as its engagement environment. Malware contagion of a specific spread rate is assumed. The defender decides placement of mission devices and tap sensitivities in the pre-determined enclave segmentation. The attacker decides the strength, duration and number of attacks in an attack plan targeting all enclaves. For a network with a set of four enclave topologies, the framework is able to generate strong availability attack patterns that were not identified a priori. It also identifies effective configurations that minimize mission delay when facing these attacks.
4.4 Dark-Horse– Internal Reconnaissance in Software Defined Networks
Once an adversary has compromised a network endpoint, they can perform network reconnaissance Sood and Enbody (2013). After reconnaissance provides a view of the network and an understanding of where vulnerable nodes are located, attackers are able to execute a plan of attack. One way to protect against reconnaissance is by obfuscating the network to delay the attacker. This approach is well suited to software defined networks (SDN) such as those deployed in cloud server settings because it requires programmability that they support Kirkpatrick (2013). The SDN controller knows which machines are actually on the network and can superficially alter (without function loss) the network view of each node, as well as place decoys (honeypots) on the network to mislead, trap and slow down reconnaissance. DARK-HORSE is shown in Fig. 13.
One such multi-component deceptive defense system Achleitner et al. (2016) foils scanning by generating “camouflaged” versions of the actual network and providing them to hosts when they renew their DHCP leases. We use this deception system and mininet Team (2018) within the framework as an engagement environment. This allows us to explore the dynamics between attacker and defender on a network where the deception and reconnaissance strategies can be adapted in response to each other Pertierra (2018).
A deception strategy is executed through a modified POX SDN controller. A reconnaissance strategy is executed by an NMAP scanLyon (2018). The attacker strategy includes choices of: which IP addresses to scan, how many IP addresses to scan, which subnets to scan, the percent of the subnets to scan, the scanning speed, and the type of scan. The defender strategy includes choices of: the number of subnets to set up, the number of honeypots, the distribution of the real hosts throughout the subnets, and the number of real hosts that exist on the network. Fitness is comprised of four components: how fast the defender detects that there is a scan taking place, the total time it takes to run the scan, the number of times that the defender detects the scanner, and the number of real hosts that the scanner discovers. Through experimentation and analysis, the framework is able to discover configurations that the defender can use to significantly increase its ability to detect scans. Similarly, there are specific reconnaissance configurations that have a better chance of being undetected.
4.5 Rivals Summary
We summarize RIVALS use cases with Table 2 and its architecture with Fig. 10. The RIVALS framework is one example of Adversarial Genetic Programming for Cyber Security, where the domain of network security is one of many possibilities in cyber security. Possible future steps are elaborated in the next section.
|Component||Robustness vs Denial||Deception vs Reconnaissance||Isolation vs Contagion|
|Threat Environment||P2P vs DDOS||Pox SDN vs Nmap||Enclave vs Malware|
|Evaluation||ALFA sim||Mininet||High level Sim|
|Objective||Mission disruption||Detection speed||Mission delay|
|Behavior: Attacker||Node/Edge impairment||Scanning parameters||Strength & Duration|
|Behavior: Defender||Network settings||Honeypots||Network taps & device placement|
|Text description||Section 4.2||Section 4.4||Section 4.3|
5 Taking Stock and Moving Forward
Considering the paper from bottom to top, we described three use cases: DIFFRACTED, AVAIL, and DARK-HORSE. For 3 specific network security contexts, they investigate unique threat environments, objectives and behaviors for their adversary species, and varying capacities to adapt during an attack on a mission. They draw upon the RIVALS framework in which adversarial arms races are the activity of interest. RIVALS is an example of Adversarial Genetic Programming for Cyber Security and is motivated by a call to arms to work on automated defensive reaction to waves of attacks. RIVALS pushes towards this goal by considering how both attack and defense adapt to each other. While we identified a breadth of applications to cyber security in the set of EAs we surveyed, we found just a modest number of extant adversarial GP that represented approaches for behavior investigation and two population dynamics, the latter combining competitive coevolution with GP. The body of prior work relevant to cyber security from AI and games and software engineering more generally informs the reader as to how competitive coevolutionary algorithms and EAs can be used as building blocks in different adversarial contexts. Finally, at the top, we argue that Adversarial Genetic Programming for Cyber Security is compellingly worthy of future community attention because of its potential to solve a critical and growing set of challenges in cyber security and because of how well GP and adversarial evolution match the technical nature of the challenges.
To end, we consider what future research questions and directions of study, stemming from the domain of cyber security, are prompted by Adversarial GP.
We forecast explorations driven by the wide array of cyber security scenarios will arise. There is a plethora of attack surfaces. We have mentioned some in Section 3.3, however, there are others such as networked cyber physical systems, ransomware, gadgets and malware. For example, how can GP help examine the adversarial cyber security of a power grid or self-driving cars? How could ransomware’s next mutation be predictable? Can GP leverage code repositories and gadgets to discover new malware? Social engineering is currently an Achilles heel for security measures because it preys on human nature and once privileged access has been obtained, it is much harder to discern an attacker. Insider threats similarly disguise attacks. How can GP inform insider threats and social engineering?
One challenge for the GP community is to become adequately conversant in cyber security topics (though no member needs to be conversant in all of them). Collaborations with cyber security researchers seem advisable, offering additive and synergistic benefits. It will be important to identify the most appropriate level of abstractions from which to design GP languages or function and terminal sets. It will be imperative to develop metrics of success and paths to technology transfer and solution deployment. At a practical level, undoubtedly new modeling and simulation modeling platforms will be needed. They will need to be general purpose (somewhat like RIVALS) and specific. Undoubtedly some will need to be scaled due to the computational cost of executing a mission.
We forecast the opportunity for extensions and innovations in GP and coevolutionary algorithms for cyber security. Different patterns and rates of evolution will be observed across different security scenarios. The opening to develop new algorithmic models of these phenomena is exciting. GP evolves behavior but many different expressions of behavior remain unexplored. Developing the new behavioral evolutionary capabilities that will be needed is both challenging and motivating.
We also forecast that research into Adversarial Genetic Programming for Cyber Security will push bridge building to other research areas. One impetus will be the quest to describe at a finer granularity what is happening during cyber evolution. For this, it may be advisable to consider evolution through the lens of individuals, such as how Artificial Life (ALife) Macal and North (2010) models individuals. What cyber security scenario should be studied with ALife? How can agents in a cyber security ALife system be represented by evolvable GP behaviors? What if a GP defender “died” after an attack and attackers starve as they fail to penetrate defenses? How would dynamics at this scale offer different insights? What would a simulation of the DDOS attack ecosystem reveal about the progression of botnet sizes if intra-species competition was examined? What can ALife studies into ecosystem dynamics around intra-species competition and cooperation reveal?
Arguably, adversarial GP is a form of agent-based modeling (ABM, Moran and Pollack (2017)) where agents are GP executable structures. Some ABM systems model more complex domains while the agents have simpler strategy spaces. Are there ABM investigations such as those of financial markets, crowd control, infectious diseases or traffic simulation that suggest innovative transfer to Adversarial GP and cyber security? Both ALife and ABM have examples that consider the spatial dimension of a domain. Competitive coevolutionary algorithms have spatial models. Interesting new models lie at these intersections.
Another area with potentially valuable interaction is Machine Learning (ML). Statistical machine learning relies on retrospective training data to learn a model capable of generalizing to new unseen data drawn from the same distribution as the retrospective data. In contrast, Adversarial Genetic Programming for Cyber Security
is not data driven and this conveys it a niche. However, what can be transferred from the studies that have started to bridge supervised learning and test-based co-optimizationPopovici (2017)? Adversarial classification Dalvi et al. (2004) as a game between classifier and adversary also appears in ML. As well, Generative Adversarial Networks (GANs) Goodfellow et al. (2014) derive generative models from samples of a distribution using adversarial neural network training. Coevolution has been applied to improve the robustness and scalability of these systems Schmiedlechner et al. (2018)
. Competitive coevolution has also been combined with multi-layer perceptron classifiers with floating point representationCastellani (2018).
Studies of adversarial coevolution in cyber security based on data that classify attacks using machine learning methods exist, e.g. see a pro-active defense for evolving cyber threats and a predictive defense against evolving adversaries in Colbaugh and Glass (2011, 2012, 2013)
. In adversarial Machine Learning the attacker manipulates data in order to defeat the Machine Learning modelLowd and Meek (2005); Huang et al. (2011); Biggio and Roli (2017). Can coevolution be used to anticipate continual adversarial evasion and guide model hardening?
Game theory considers equilibria and paths to equilibria. What is an equilibrium in cyber security? It could be the complete wipeout of one side of the adversarial equation. Or, it could be the point where an attacker chooses not to target a defense because it is less expensive to go elsewhere. It could be a Nash equilibrium where neither attacker or defender has a better tactical position to go to unless the other simultaneously changes also. One direction should investigate how these game theoretic notions can inform the highly empirical, algorithmic systems of Adversarial GP.
We believe the future directions are not enumerable because the likelihood of new adaptations by adversaries will never be zero. To date, defenses expose far more attack surface than they can protect and attackers only need to find one crack to penetrate. To this end, the emerging importance of Adversarial Genetic Programming for Cyber Security is not likely to fade. We trust this justifies our call to arms.
This was supported by the CSAIL CyberSecurity Initiative. This material is based upon work supported by DARPA. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements. Either expressed or implied of Applied Communication Services, or the US Government. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 799078.
- The art of red teaming. In Computational Red Teaming, pp. 1–45. Cited by: item Red teaming.
- Cyber deception: virtual networks to defend insider reconnaissance. In Proceedings of the 2016 International Workshop on Managing Insider Security Threats, pp. 57–68. Cited by: §4.4.
- How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In Genetic and Evolutionary Computation–GECCO 2004, pp. 1338–1349. Cited by: §3.2, Table 1.
- State of the internet quarterly security reports. External Links: Cited by: Figure 2.
- Akamai’s State of the Internet / Security Report - Q1 2017 report. Technical report Akamai Technologies, Inc.. External Links: Cited by: Figure 2.
- Akamai’s State of the Internet / Security Report - Q3 2017 report. Technical report Akamai Technologies, Inc.. External Links: Cited by: Figure 3.
- Behavior trees for next-gen game ai. In Game Developers Conference, Lyon, France, pp. 3–4. Cited by: §3.1.
- An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86 (8), pp. 1978–2001. Cited by: §3.2.
- Competitive environments evolve better solutions for complex tasks. In Proceedings of the Fifth International Conference (GA93), Genetic Algorithms, pp. 264–270. Cited by: §2.1, §3.1.
- Understanding the mirai botnet. In 26th USENIX Security Symposium (USENIX Security 17), pp. 1093–1110. Cited by: §1.
- Coevolutionary multi-objective evolutionary algorithms: a survey of the state-of-the-art. IEEE Transactions on Evolutionary Computation (), pp. 1–16. External Links: Cited by: §2.
- An architecture for adversarial planning. Systems, Man and Cybernetics, IEEE Transactions on 20 (1), pp. 186–194. Cited by: item Battle management.
- Coevolving programs and unit tests from their specification. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, pp. 397–400. Cited by: §3.2, Table 1.
- Co-evolutionary automatic programming for software development. Information Sciences 259, pp. 412–432. Cited by: §2.2, §3.2.
- The evolution of strategies in the iterated prisoner’s dilemma. The dynamics of norms, pp. 1–16. Cited by: §3.1, Table 1.
- The evolution of cooperation. 10, Basic, NY, New York. Cited by: §2.1, §3.1, §3.1, Table 1.
- Selection methods to relax strict acceptance condition in test-based coevolution. In 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. Cited by: §2.2.
- The oracle problem in software testing: a survey. Cited by: §3.2.
- Enhancing Automated Red Teaming with Monte Carlo Tree Search. Cited by: item Red teaming.
- Wild patterns: ten years after the rise of adversarial machine learning. arXiv preprint arXiv:1712.03141. Cited by: §5.
- Characterizing effects on the cyber adversary: a vocabulary for analysis and assessment. The MITRE Corporation, Bedford, MA. Cited by: §A.4.
- Nonlinear system identification using coevolution of models and tests. IEEE Transactions on Evolutionary Computation 9 (4), pp. 361–384. Cited by: §2.1.
- Mistakes allow evolutionary stability in the repeated prisoner’s dilemma game. Journal of theoretical Biology 136 (1), pp. 47–56. Cited by: item, §2.1.
- Akamai on the Record KrebsOnSecurity Attack. Note: https://krebsonsecurity.com/2016/11/akamai-on-the-record-krebsonsecurity-attack/(Accessed October 10,2018) Cited by: §1.
- An artificial arms race: could it improve mobile malware detectors?. In 2018 Network Traffic Measurement and Analysis Conference (TMA), pp. 1–8. Cited by: item Vulnerabilty Testing.
- Emergent geometric organization and informative dimensions in coevolutionary algorithms. Ph.D. Thesis, Brandeis University. Cited by: item, §2.1.
- Competitive coevolution in ms. pac-man. In 2013 IEEE Congress on Evolutionary Computation, pp. 1403–1410. Cited by: §2.1, §3.1.
- Combating coevolutionary disengagement by reducing parasite virulence. Evolutionary Computation 12 (2), pp. 193–222. Cited by: item, §2.1.
- Competitive co-evolution of multi-layer perceptron classifiers. Soft Computing 22 (10), pp. 3417–3432. Cited by: §5.
- Evolution, neural networks, games, and intelligence. Proceedings of the IEEE 87 (9), pp. 1471–1496. Cited by: §3.1.
- Moving target defense for adaptive adversaries. In Intelligence and Security Informatics (ISI), 2013 IEEE International Conference on, pp. 50–55. Cited by: §5.
- Proactive defense for evolving cyber threats. In Intelligence and Security Informatics (ISI), 2011 IEEE International Conference on, pp. 125–130. Cited by: §5.
- Predictive defense against evolving adversaries. In Intelligence and Security Informatics (ISI), 2012 IEEE International Conference on, pp. 18–23. Cited by: §5.
- The ecology of malware. In Proceedings of the 2008 workshop on New security paradigms, pp. 99–106. Cited by: §3.3.
- Virtual witches and warlocks: a quidditch simulator and quidditch-playing teams coevolved via genetic programming. In Late-Breaking Papers of GECCO-2004, the Genetic and Evolutionary Computation Conference. Published by the International Society for Genetic and Evolutionary Computation, Cited by: Table 1.
- Adversarial classification. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 99–108. Cited by: §5.
- The World’s First All-Machine Hacking Tournament. Note: http://archive.darpa.mil/cybergrandchallenge/(Accessed October 10,2018) Cited by: item Red teaming, §3.3.
- A coevolutionary algorithm to automatic test case selection and mutant in mutation testing. In 2013 IEEE Congress on Evolutionary Computation, pp. 829–836. Cited by: §3.2, Table 1.
- Game programming gems 2. Cengage learning. Cited by: §3.1.
- Constraint-based automatic test data generation. Software Engineering, IEEE Transactions on 17 (9), pp. 900–910. Cited by: §3.2.
- Butterflies and plants: a study in coevolution. Evolution 18 (4), pp. 586–608. Cited by: §2.
- Effectiveness of moving target defenses. In Moving Target Defense, pp. 29–48. Cited by: item Moving target defense (MTD).
- Solution concepts in coevolutionary algorithms. Ph.D. Thesis, Brandeis University. Cited by: §2.1.
- Note: Picture taken by Jay Cross - License: CC-BY-SA-2.0 External Links: Cited by: 0(e).
- Blondie24: playing at the edge of artificial intelligence. Morgan Kaufmann San Francisco. Cited by: §2.1.
- Predation and the cost of replication: new approaches to malware prevention?. computers & security 25 (4), pp. 257–264. Cited by: §3.3.
- A sense of self for unix processes. In Proceedings 1996 IEEE Symposium on Security and Privacy, pp. 120–128. Cited by: item Anomaly detection.
- Return-oriented programme evolution with roper: a proof of concept. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1447–1454. Cited by: item Malware detection.
- Investigating coevolutionary archive based genetic algorithms on cyber defense networks. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’17, New York, NY, USA, pp. 1455–1462. External Links: Cited by: §3.3, Table 1, 1st item, §4.2.
- An empirical comparison of botnet detection methods. computers & security 45, pp. 100–123. Cited by: item Intrusion detection.
- DART: directed automated random testing. In ACM Sigplan Notices, Vol. 40, pp. 213–223. Cited by: §3.2.
- Genetic algorithms in search, optimization and machine learning. 1st edition, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA. External Links: Cited by: §2.
- Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Cited by: §5.
- An empirical study of malware evolution. In Communication Systems and Networks and Workshops, 2009. COMSNETS 2009. First International, pp. 1–10. Cited by: §3.3.
- Automatic generation of test cases. IBM Systems Journal 9 (4), pp. 242–257. Cited by: §3.2.
- Evolving robocode tanks for evo robocode. Genetic Programming and Evolvable Machines 15 (4), pp. 403–431. Cited by: §3.1, Table 1.
- Adversarial co-evolution of attack and defense in a segmented computer network environment. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1648–1655. Cited by: §3.3, Table 1, Figure 12, 2nd item, §4.3.
- Red teaming with coevolution. In Evolutionary Computation (CEC), 2011 IEEE Congress on, pp. 1155–1163. External Links: Cited by: item Red teaming, §3.3, Table 1.
- Diffuse versus true coevolution in a physics-based world. In Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation-Volume 2, pp. 1305–1312. Cited by: item, §2.1.
- Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pp. 43–58. Cited by: §5.
- Darwin inside the machines: malware evolution and the consequences for computer security. arXiv preprint arXiv:1111.2503. Cited by: §3.3.
- Can a good offense be a good defense? Vulnerability testing of anomaly detectors through an artificial arms race. Applied Soft Computing 11 (7), pp. 4366–4383. Cited by: item Vulnerabilty Testing.
- CAN the best defense be a good offense? Evolving (MIMICRY) attacks for detector vulnerability testing under a ‘black-box’assumption. Ph.D. Thesis, Dalhousie University Halifax. Cited by: item Vulnerabilty Testing.
- Evolving coordination for real-time strategy games. IEEE Transactions on Computational Intelligence and AI in Games 3 (2), pp. 155–167. Cited by: §3.1, §3.1, Table 1.
- Computational military tactical planning system. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 32 (2), pp. 161–171. External Links: Cited by: item Battle management, §3.3, Table 1.
- On botnet detection with genetic programming under streaming data label budgets and class imbalance. Swarm and evolutionary computation 39, pp. 123–140. Cited by: item Intrusion detection.
- An efficient genetic algorithm with less fitness evaluation by clustering. Proceedings of the 2001 Congress on Evolutionary Computation, pp. 887–894. Cited by: §2.2.
- Advances in genetic programming. Vol. 3, MIT press. Cited by: §2.
- Software-defined networking. Communications of the ACM 56 (9). Cited by: §4.4.
- Genetic programming ii, automatic discovery of reusable subprograms. MIT Press, Cambridge, MA. Cited by: §2.
- Solving complex problems with coevolutionary algorithms. In Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 687–713. Cited by: §2.1, §2.
- Learning n-tuple networks for othello by coevolutionary gradient search. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, GECCO ’11, pp. 355–362. Cited by: §3.1.
- A review of computational intelligence in rts games. In 2013 IEEE Symposium on Foundations of Computational Intelligence (FOCI), pp. 114–121. Cited by: §3.1.
- Evolving tcp/ip packets: a case study of port scans. In 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, pp. 1–8. Cited by: item Vulnerabilty Testing.
- Moving target defenses in the helix self-regenerative architecture. In Moving Target Defense II, pp. 117–149. Cited by: item Self-Adapting Cyber Defenses.
- Training binary gp classifiers efficiently: a pareto-coevolutionary approach. In European Conference on Genetic Programming, pp. 229–240. Cited by: item Anomaly detection.
- Evolving behaviour trees for the commercial game DEFCON. In European Conference on the Applications of Evolutionary Computation, pp. 100–110. Cited by: §2.1, §3.1, Table 1.
- The 1999 darpa off-line intrusion detection evaluation. Computer networks 34 (4), pp. 579–595. Cited by: item Intrusion detection.
- Non-negative matrix factorization for unsupervised derivation of search objectives in genetic programming. In Proceedings of the 2016 on Genetic and Evolutionary Computation Conference, pp. 749–756. Cited by: §2.2.
- Online discovery of search objectives for test-based problems. Evolutionary computation 25 (3), pp. 375–406. Cited by: §2.2.
- Adversarial learning. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, pp. 641–647. Cited by: §5.
- Genetic programming produced competitive soccer softbot teams for robocup97. Genetic Programming 1998, pp. 214–222. Cited by: §2.1, §3.1, Table 1.
- Nmap network scanner. Note: https://nmap.org/[Online; accessed 6-July-2018] Cited by: §4.4.
- Tutorial on agent-based modelling and simulation. Journal of simulation 4 (3), pp. 151–162. Cited by: §5.
- An analysis of the 1999 darpa/lincoln laboratory evaluation data for network anomaly detection. In Recent Advances in Intrusion Detection, G. Vigna, C. Kruegel, and E. Jonsson (Eds.), Berlin, Heidelberg, pp. 220–237. External Links: Cited by: item Intrusion detection.
- Towards a systematic view on cybersecurity ecology. arXiv preprint arXiv:1505.04207. Cited by: §3.3.
- Investigating the dynamics of competition: coevolving red and blue simulation parameters. In Proceedings of the 37th conference on Winter simulation, pp. 1008–1012. Cited by: item Red teaming, §3.3, Table 1.
- Moderate environmental variation across generations promotes the evolution of robust solutions. Artificial life 24 (4), pp. 277–295. Cited by: §2.2.
- Co-evolving influence map tree based strategy game players. In Computational Intelligence and Games, 2007. CIG 2007. IEEE Symposium on, pp. 88–95. Cited by: §3.1, Table 1.
- Coevolutionary learning with spatially distributed populations. Computational intelligence: principles and practice. Cited by: §2.1, §2.2.
- Effects of cooperative and competitive coevolution on complexity in a linguistic prediction game. In Artificial Life Conference Proceedings 14, pp. 298–205. Cited by: §5.
- Automated fault analysis and filter generation for adaptive cybersecurity. In Proc. 6th Int’l Conf. on Adaptive and Self-Adaptive Systems and Applications, Cited by: item Self-Adapting Cyber Defenses.
- Meta-control for adaptive cybersecurity in fuzzbuster. In Self-Adaptive and Self-Organizing Systems (SASO), 2013 IEEE 7th International Conference on, pp. 219–226. Cited by: item Self-Adapting Cyber Defenses.
- The president has no clothes: the case for broader application of red teaming within homeland security. Technical report DTIC Document. Cited by: item Red teaming.
- Evolutionary behavior tree approaches for navigating platform games. IEEE Transactions on Computational Intelligence and AI in Games 9 (3), pp. 227–238. Cited by: §3.1.
- Competitive algorithms for coevolving both game content and ai. a case study:planet wars. IEEE Transactions on Computational Intelligence and AI in Games 8 (4), pp. 325–337. Cited by: §3.1.
- Grammatical evolution: evolutionary automatic programming in an arbitrary language. Vol. 4, Springer. Cited by: §4.1.
- Introduction to the special issue: trends in evolutionary methods for program induction. Evolutionary Computation 5 (2), pp. v–ix. Cited by: §2.
- An artificial coevolutionary framework for adversarial ai. In Adversary-aware Learning techniques and trends in Cybersecurity, AAAI Fall Symposium, Cited by: §4.
- Finding focus in the blur of moving-target techniques. Security Privacy, IEEE 12 (2), pp. 16–26. External Links: Cited by: item Moving target defense (MTD).
- Co-evolutionary search in asymmetric spaces. Information Sciences 133 (3-4), pp. 103–125. Cited by: §2.1.
- Coevolutionary-based mechanisms for network anomaly detection. Journal of Mathematical Modelling and Algorithms 6 (3), pp. 411–431. Cited by: item Anomaly detection, §3.3, Table 1.
- Evolving behaviour trees for the Mario AI competition using grammatical evolution. In European Conference on the Applications of Evolutionary Computation, pp. 123–132. Cited by: §3.1.
- Investigating coevolutionary algorithms for expensive fitness evaluations in cybersecurity. Master’s Thesis, Massachusetts Institute of Technology. Cited by: 3rd item, §4.1, §4.4, §4.
- Circular economy: a coevolutionary perspective on diversity. uwf UmweltWirtschaftsForum 24 (2), pp. 253–260. Cited by: §1.
- Co-evolution in the successful learning of backgammon strategy. Machine Learning 32 (3), pp. 225–240. Cited by: §3.1, §3.1.
- Coevolutionary principles. In Handbook of Natural Computing, pp. 987–1033. Cited by: §2.2.
- Coevolutionary principles. In Handbook of Natural Computing, G. Rozenberg, T. Bäck, and J. N. Kok (Eds.), pp. 987–1033. Cited by: §2.1, §2.1, §2.1, §2.
- A framework for co-optimization algorithm performance and its application to worst-case optimization. Theoretical Computer Science 567, pp. 46–73. Cited by: §2.2.
- Bridging supervised learning and test-based co-optimization. Journal of Machine Learning Research 18 (38), pp. 1–39. Cited by: §2.2, §5.
- Visualizing adversaries - transparent pooling approaches for decision support in cybersecurity. Master’s Thesis, Massachusetts Institute of Technology. Cited by: §4.1, §4.1, §4.1, §4.
- New methods for competitive coevolution. Evolutionary Computation 5 (1), pp. 1–29. Cited by: §2.1, §2, §2.
- Coevolutionary agent-based network defense lightweight event system (CANDLES). In Proceedings of the Companion Publication of the 2015 on Genetic and Evolutionary Computation Conference, pp. 859–866. Cited by: item Network Defense Investigation, §3.3, Table 1.
- Natural security: how biological systems use information to adapt in an unpredictable world. Security Informatics 1 (1), pp. 14. Cited by: §3.3.
- Competitive coevolutionary algorithm decision support. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 300–301. Cited by: §4.1, §4.1, §4.2.
- Towards distributed coevolutionary gans. arXiv preprint arXiv:1807.08194. Cited by: §5.
- Dyn Analysis Summary Of Friday October 21 Attack. Note: https://dyn.com/blog/dyn-analysis-summary-of-friday-october-21-attack/(Accessed October 10,2018) Cited by: §1.
- Strata: a software dynamic translation infrastructure. In In IEEE Workshop on Binary Translation, Cited by: item Self-Adapting Cyber Defenses.
- Increasing infrastructure resilience through competitive coevolution. New Mathematics and Natural Computation 5 (02), pp. 441–457. Cited by: item Physical Infrastructure defense, §3.3, Table 1.
- Evolving 3d morphology and behavior by competition. Artificial life 1 (4), pp. 353–372. Cited by: §2.1, §2.1, §2.
- Designing an evolutionary strategizing machine for game playing and beyond. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37 (4), pp. 583–593. Cited by: §3.1.
- Evolved to win. Lulu. com. Cited by: §3.1.
- Co-adaptive genetic algorithms: An example in othello strategy. In Proc. Florida Artificial Intelligence Research Symposium, 1994, Cited by: §3.1.
- Coevolving deep hierarchies of programs to solve complex tasks. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1009–1016. Cited by: §3.1.
- Son of boss — Wikipedia, the free encyclopedia. Note: https://en.wikipedia.org/wiki/Son_of_Boss(Accessed October 10,2018) Cited by: §1.
- Training genetic programming on half a million patterns: an example from anomaly detection. IEEE Transactions on Evolutionary Computation 9 (3), pp. 225–239. Cited by: item Anomaly detection.
- Targeted cyberattacks: a superset of advanced persistent threats. IEEE security & privacy 11 (1), pp. 54–61. Cited by: §4.4.
- Real-time neuroevolution in the nero video game. IEEE Transactions on Evolutionary Computation 9 (6), pp. 653–668. Cited by: §3.1.
- Automatic rule generation based on genetic programming for event correlation. In Computational Intelligence in Security for Information Systems, pp. 127–134. Cited by: item Intrusion detection, §3.3, Table 1.
- Mirai: what you need to know about the botnet behind recent major DDoS attacks. Note: https://www.symantec.com/connect/blogs/mirai-what-you-need-know-about-botnet-behind-recent-major-ddos-attacks(Accessed October 10,2018) Cited by: §1.
- Coevolutionary temporal difference learning for othello. In 2009 IEEE Symposium on Computational Intelligence and Games, pp. 104–111. Cited by: §3.1.
- On scalability, generalization, and hybridization of coevolutionary learning: a case study for othello. IEEE Transactions on Computational Intelligence and AI in Games 5 (3), pp. 214–226. Cited by: §3.1.
-  Coevolutionary dynamics of automotive competition: product innovation, change, and marketplace survival. Journal of Product Innovation Management 31 (1), pp. 61–78. Cited by: §A.2, §1.
- Mininet - realistic virtual sdn network emulator. Note: http://mininet.org/[Online; accessed 6-July-2018] Cited by: §4.4.
- Multi-population competitive co-evolution of car racing controllers. In 2007 IEEE Congress on Evolutionary Computation, pp. 4043–4050. Cited by: §2.1, §3.1.
- Automatic program repair with evolutionary computation. Communications of the ACM 53 (5), pp. 109–116. Cited by: item Self-Adapting Cyber Defenses.
-  ()(Website) Note: Picture taken by Nick Hobgood - License: CC BY-SA 3.0 External Links: Cited by: 0(b).
- Note: Picture taken by Olaf Leillinger - License: CC-BY-SA-2.0/DE and GNU FDL External Links: Cited by: 0(a).
- Note: License: CC BY-SA 3.0. Subject to disclaimers. External Links: Cited by: 0(d).
- Note: Created: 29 May 2007 By Kenneth Dwain Harrelson - License: CC BY-SA 3.0 External Links: Cited by: 0(c).
- Note: Picture taken by Chiswick Chap - License: CC BY-SA 4.0 External Links: Cited by: 0(f).
- Coevolutionary automated software correction. In Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, GECCO ’10, New York, NY, USA, pp. 1391–1392. Cited by: §3.2, Table 1.
- Understanding the co-evolution of cyber defenses and attacks to achieve enhanced cybersecurity. Warfare 14, pp. 17–31. Cited by: §3.3.
- Investigating the success of spatial coevolution. In Proceedings of the 7th annual conference on Genetic and evolutionary computation, pp. 523–530. Cited by: §2.1.
- Strategic evolution of adversaries against temporal platform diversity active cyber defenses. In Proceedings of the 2014 Symposium on Agent Directed Simulation, pp. 9. Cited by: item Moving target defense (MTD), §3.3, Table 1.
- Red teaming of advanced information assurance concepts. In DARPA Information Survivability Conference and Exposition, 2000. DISCEX’00. Proceedings, Vol. 2, pp. 112–118. Cited by: item Red teaming.
- Financial alchemy: how tax shelter promoters use financial products to bedevil the irs (and how the irs helps them). Ariz. St. LJ 45, pp. 611. Cited by: §1.
- Automatically evading classifiers. In Proceedings of the 2016 Network and Distributed Systems Symposium, Cited by: item Malware detection.
- Automated cyber red teaming. Technical report DTIC Document. Cited by: item Red teaming.
- Evolving optimal and diversified military operational plans for computational red teaming. Systems Journal, IEEE 6 (3), pp. 499–509. External Links: Cited by: item Red teaming.
a.1 Firefly squid bioluminescence defense
When firefly squid are seen from below by a predator, the bioluminescence helps to match the squid’s brightness and color to the sea surface above.
a.2 A coevolutionary perspective of the U.S. automotive industry
Talay, Calantone, and Voorhees presented an study that explicitly terms competitive interactions between firms “Red Queen competition”, in which gains from innovations are relative and impermanent Talay et al. .
a.3 Advanced Persistent Threats and Ransomware
Some DOS attacks includes Advanced Persistent Threats (APT) and Ransomware. The first ones have multiple stages starting at external reconnaissance then moving to intrusion (e.g. social engineering or use of zero day exploits), laterally moving malware, command and control direction to data exfiltration and, finally, self-erasure. Ransomware, which largely preys upon unpatched systems and which exploits anonymous payment channels enabled by Bitcoin, has also recently become more frequent.
a.4 Cyber security attack categorization
Examples of attacks, classifications and taxonomies can be found at https://cwe.mitre.org/index.html. One categorization is: A) Advanced Persistent Threats, “lurking” threats from resourceful persevering adversaries B) Denial of Service Attack, defense resource limitation and exposure, means of penetrating systems C) Identity theft, e.g. impersonating users. Attacks are also characterized by their stages on a timeline. Another characterization is based on the attacker identity, from individuals to organized criminals and nation states, and what resources they access, see Bodeau and Graubart (2013) for details.