Typically, AI researchers and roboticists try to realize intelligent behavior in machines by tuning parameters of a predefined structure (body plan and/or neural network architecture) using evolutionary or learning algorithms. Another but not unrelated longstanding property of these systems is their brittleness to slight aberrations, as highlighted by the growing deep learning literature on adversarial examples. Here we show robustness can be achieved by evolving the geometry of soft robots, their control systems, and how their material properties develop in response to one particular interoceptive stimulus (engineering stress) during their lifetimes. By doing so we realized robots that were equally fit but more robust to extreme material defects (such as might occur during fabrication or by damage thereafter) than robots that did not develop during their lifetimes, or developed in response to a different interoceptive stimulus (pressure). This suggests that the interplay between changes in the containing systems of agents (body plan and/or neural architecture) at different temporal scales (evolutionary and developmental) along different modalities (geometry, material properties, synaptic weights) and in response to different signals (interoceptive and external perception) all dictate those agents' abilities to evolve or learn capable and robust strategies.READ FULL TEXT VIEW PDF
A major characteristic of life is that three broad time scales are relevant to it: evolution, development and physiological functioning. Engineered systems, in marked contrast, often employ an evolutionary or learning algorithm to improve their behavior over time, but rarely employ morphological development; any changes to the physical layout are made in between evaluations (Sims, 1994; Lipson and Pollack, 2000; Cheney et al., 2013), if they are made at all.
Two notable exceptions are modular robots (Zykov et al., 2005), which may reconfigure their bodies by adding and removing discrete structures, and soft robots (Shepherd et al., 2011), which may continuously alter the local volumes of different parts of their bodies while behaving. Others (Bongard, 2011) have approximated topological change in rigid bodies by extending outward and angling downward appendages using a combination of linear and rotary actuators, thus simulating limb growth.
Several computational but embodied models of prenatal development have been reported in the literature (Dellaert and Beer, 1996; Eggenberger, 1997; Bongard and Pfeifer, 2001; Miller, 2004; Doursat, 2009). As implied, cellular growth therein occurred prior to any physiological functioning. Thus, these studies included change during only two of the three time scales relevant to life: evolutionary and behavioral change, but not postnatal developmental change.
The most common argument in favor of development is that some aspects of the environment are unpredictable, so it is advantageous to leave some decisions up to development rather than specifying them genetically. Although self evident, it remains to determine which mechanisms of development should be instantiated in robots to realize plastic, adaptive, and useful machines.
Naturally, the performance of an evolved system depends on its capacity for evolutionary improvement: its evolvability. Development can, under certain conditions, smooth the search space evolution that operates in, thus increasing evolvability. This process, known as the Baldwin effect (Baldwin, 1896; Dennett, 2003), starts with an advantageous characteristic acquired during the development of individuals, such as the callouses in Fig. 1. This can create a new gradient in the evolutionary search space, rewarding descendants that more rapidly manifest the trait during their lifetimes (Hinton and Nowlan, 1987; Kriegman et al., 2017a) and retain it through the remainder of their lifetime (Kriegman et al., 2017b). Assuming such mutations exist and can be naturally selected (Kriegman et al., 2017a), following the gradient requires incrementally reducing development in the manifold of the search space that can express variations on the trait (Waddington, 1942).
However, fitness landscapes that evolution climbs, and development sometimes smooths, tend not to remain static in realistic settings. On this vacillating landscape, when the best thing to do does not remain the same, a highly evolvable but non-robust system will need to keep starting over from scratch every time the conditions change. Computational and engineered systems provide countless examples of systems with nearly perfect performance in a controlled environment, such as a factory, but who turn out to be (often comically) brittle to slight changes in their internal structure, such as damage, or their external environment such as moving on to new terrain or transferal from simulation to reality (Carlson and Murphy, 2005; Koos et al., 2013; Goodfellow et al., 2013; Szegedy et al., 2013; Athalye and Sutskever, 2017; Nguyen et al., 2015).
Although generally absent from engineered systems (but see (Bongard et al., 2006; Cully et al., 2015)), the canonical form of robustness is seen to some extent in all organisms, and it comes from the act of development itself. For example, a plant that grows according to a fixed program will capture less light than a plant that grows toward sunlight. But there is another, more subtle form of robustness that we will refer to as ‘intrinsic robustness’ because it is a property of a system’s structure rather than of the process by which it may change.
Developmental change produces intrinsically robust systems because they evolved from designs that had to maintain adequate performance along additional dimensions of change (Bongard, 2011; Kriegman et al., 2017a). Through morphological development specifically, evolution is compelled to maintain designs that are capable across a series of body plans, with different sensor-motor contingencies; and the ability to tolerate such perturbations can become inherited to some extent in descendants’ behaviors (Bongard, 2011) and morphologies (Kriegman et al., 2017a), even when their developmental flexibility is reduced or completely removed by canalization or fabrication.
And yet, despite the ubiquity of morphological development in nature, and the adaptive advantages it seemingly confers, there are only a handful of cases reported in the literature in which a simulated robot’s mechanical structure was allowed to change while it was behaving (e.g. (Ventrella, 1998; Komosinski, 2003; Bongard, 2011; Kriegman et al., 2017a; Kriegman et al., 2017b)), all of which modeled morphological development as a genetically predetermined process: the environment could not influence the way in which development unfolded.
Assuming that an engineered system is capable of local morphological change in response to environmental signals, it is unclear how it should do so, beyond the examples of morphological plasticity observed in nature. Examples include Wolff’s law (Ruff et al., 2006)—bone grows in response to particular mechanical loading profiles—and Davis’ law—soft tissue increases in strength in response to intermittent mechanical demands. One can envisage other such laws that are not known to occur in biology but could be helpful in a specific artificial system, such as end effectors softening in response to pressure, which might enhance their ability to safely manipulate irregular or delicate objects (Brown et al., 2010). Indeed, the genesis of the work presented here is one such anecdotal example given in (Corucci et al., 2017), where a single robot, subjected to an abrupt doubling in gravity, stiffened its body in reaction to the increased pressure. However, whether or how it could provide a behavioral advantage, nor whether pressure is the best interoceptive signal to developmentally respond to, was not investigated.
As a step towards a more adequate picture, we introduce here a simple form of a developmental feedback mechanism: Genetic systems dictate how organisms develop in response to interoceptive stimuli, and development alters the kinds of interoceptive conditions the organism experiences. More specifically, at every time step, the proposed model of closed-loop development:
‘listens to’ load signatures generated from movement; and, in response,
modifies the robot’s rigidity,
which will change the way it distributes load and generates movement at the next time step.
Optimizing a system that may form a continuum of rigid and soft components—and in which this admixture may change over time—is extremely nonintuitive and underexplored. Thus, a study of the adaptive properties of such systems—and how they can best be optimized to render useful work—is initiated here.
We evolved locomotive machines constructed from voxels with heterogeneous stiffness. Like many organisms (Ruff et al., 2006), the robot’s material stiffness progressively changes in response to mechanical loading incurred as the robot behaves. This ontogenetic change occurs independently at each voxel according to an evolved local rule.
The soft-matter physics engine Voxelyze (Hiller and Lipson, 2014) is used to calculate the movement of robots resulting from their interaction with a virtual 3D terrestrial environment. Each robot is simulated for 25 times the length of an expansion/contraction cycle (a total of five seconds). The displacement between the starting coordinates and the agent’s final center of mass (in the plane) is recorded.111github.com/skriegman/2018-gecco contains the source code necessary for reproducing the results reported in this paper.
Materials are simulated to have increased mass relative to those used by (Hiller and Lipson, 2012; Cheney et al., 2013, 2014). Constructed from heavier materials, many previously mobile robots become crushed under their own weight and require stiffer material to support locomotion with the same geometry. However, we also restricted actuation amplitude as materials grow stiffer to better approximate the properties of real materials with different stiffnesses. This creates an interesting and realistic trade-off: the stiffest material can easily support any body plan but cannot move on its own (like a skeleton without muscle), whereas the softest material can readily elicit forward movement in smaller bodies but cannot support many larger and potentially faster-moving body plans, such as those with narrow supporting limbs. Thus, a robot must carefully balance support with actuation.
Following (Cheney et al., 2013), robot physiology is genetically encoded by a Compositional Pattern Producing Network (CPPN) (Stanley, 2007), a scale-free mapping that biases search toward symmetrical and regular patterns which are known to facilitate locomotion.
Each point on a 101010 lattice is queried by its cartesian coordinates in 3D space and its radial distance from the lattice center. An evolved CPPN takes these coordinates as input and returns a single value which is used to set some property of that point in the workspace. We used four independent CPPNs to separately encode: Geometry, Stiffness, Development, and Actuation.
The geometry of a robot is specified by a bitstring that indicates whether material is present (1) or absent (0) at each lattice point in the workspace, as dictated by . The robot’s geometric shape is taken to be the largest contiguous collection of present voxels.
Young’s modulus is often used as an approximate measure of material stiffness. Robots are typically constructed of materials such as metals and hard plastics that have moduli in the order of pascals (Pa), whereas soft robots (and natural organisms) are often composed of materials with moduli in the order of Pa (Rus and Tolley, 2015).
Here, voxels may have moduli in the range Pa. The robot’s congenital stiffness is set at each voxel by , but may be changed by development.
The robot’s stiffness distribution can change progressively during its lifetime in response to localized engineering stress or pressure . The rate of change is specified at the voxel by , with possible values in . We compare three developmental variants.
Robots are ‘controlled by’ volumetric actuation: a sinusoidal expansion/contraction of each voxel with a maximum amplitude of 50% volumetric change.222The scare quotes are intended to highlight the fact that actuation policies no more dictate the movement of a robot than its geometry. However, linear damping is implemented into the system such that the stiffest material does not actuate (Eq. 5). The phase difference of each voxel is determined by , which offsets its oscillation relative to a central pattern generator.
Prior to actuation, each voxel has a resting length of one centimeter. This length is periodically varying ( 5 Hz) by approximately 14.5% ( 0.145 cm), but damped by . The instantaneous length of the th voxel is thus:
We employed a standard evolutionary algorithm, Age-Fitness-Pareto Optimization(Schmidt and Lipson, 2011), which uses the concept of Pareto dominance and an objective of age (in addition to fitness) intended to promote diversity among candidate designs.
We performed twenty independent evolutionary trials with different random seeds (1-20); in each trial, a population of 24 robots was evolved for five thousand generations. Every generation, the population is first doubled by creating modified copies of each individual in the population. The age of each individual is then incremented by one. Next, an additional random individual (with age zero) is injected into the population (which now consists of 49 robots). Finally, selection reduces the population down to its original size (24 robots) according to the two objectives of fitness (maximized) and age (minimized).
Mutations add/remove/alter a particular node/link of a CPPN. They are applied by first selecting which networks to mutate, with the possibility to select all four, and then choosing which operations to apply to each.
We use bootstrapping to construct hypothesis tests. All -values are reported with Bonferroni correction for multiple (typically three) comparisons. We adopt the following convention: ‘’ for ‘’ for and ‘’ for .
We first investigated whether environment-mediated morphological development affected evolvability (Fig. 5A). At the termination of an evolutionary trial, we only consider the most fit individual—the run champion—from each of the twenty independent trials (Figs. 2, 3, 4
). After correcting for three comparisons, there was not enough evidence to reject the null hypothesis—that there is no difference between adaptive and nonadaptive robots, in terms of mean displacement—at the 0.05 level.
These results could be taken to suggest one of the following. Either the search space is sufficiently smooth prior to development (actuation and support are not as antagonistic as envisaged), or the proposed developmental mechanism is an insufficient smoothing mechanism (successful ways to change stiffness as a linear function of stimuli are sparse in the search space).
; lower bars indicate more canalization); (E) Variance, taken across the body, of relative lifetime change in stiffness, as a measure of heterogeneity/nonuniformity in developmental reactions (Eq.9); (F) Variance, taken across the body, of the coefficients/gain of developmental reactivity (Eq. 10).
We investigate morphological diversity next by employing the Hausdorff distance as a metric to compare the similarity of two robot geometries, and . For each voxel in , the closest voxel in is identified, according to euclidean distance . Similarly, for each voxel in , the closest voxel in is identified. The Hausdorff metric is the larger of these two distances. Formally,
Informally, two robots are close in the Hausdorff distance if every voxel of either robot is close to some voxel of the other robot.
We calculated the Hausdorff distance between each of the possible pairings of the 20 run champions (Fig. 5B). Because depends on the orientations of and , we rotate in the plane (0, 90, 180, and 270 degrees) and the plane (0 and 90 degrees), and select the rotation that creates the smallest .
We found the evolved body shapes of pressure-adaptive robots to be more diverse than those of stress-adaptive robots . We did not find a significant difference, at the 0.05 level, between adaptive and nonadaptive treatments using this particular measure of morphological diversity.
Across all three treatments, there appear on visual inspection to be three types of geometries (Figs. 2-4): a robot with wide posterior and anterior legs; a robot whose legs meet perpendicularly; and a robot that connects a (mainly cylindrical) leg perpendicularly to the center of a vertical plane. Depending on how one counts, the species can be seen in at most one nonadaptive robot (Fig. 2, run 19), two stress-adaptive robots (Fig. 3, runs 7 and 16), and six pressure-adaptive robots (Fig. 4, runs 2, 6, 7, 9, 12, and 16). Pressure-adaptive robots have more diversity by virtue of more robots.
To investigate the relative robustness (if any) across the three treatments, in the following experiment, development was manually removed from the stress- and pressure-adaptive run champions. We then tested the sensitivity of the resulting reduced robots to their evolved congenital stiffness distribution (Fig. 5C). To do so, we replaced the evolved network dictating material stiffness, with a random number generator that draws from the same range of possible stiffness ( Pa). That is to say, we ‘built’ the evolved run champions without any errors in the specifications of geometry and actuation, but completely ignored the evolved specifications of their material stiffness, replacing them instead with random noise. We then calculated the relative fitness
where is the fitness achieved using the evolved stiffness and is the fitness when tested with a random stiffness distribution. We repeated this process ten times for each run champion, each time drawing a new random stiffness distribution.
We found that, compared to nonadaptive robots, reduced stress-adaptive robots were more robust to this (extreme) discrepancy between training and testing stiffness distributions . These results are consistent with the found correlation between development and robustness (Miller, 2004; Bongard, 2011; Kriegman et al., 2017a). However, the results here indicate that this correlation is contingent on the kind of environmental signal the developing agent responds to: there was no difference between pressure-adaptive and nonadaptive robots in this regard, at the 0.05 level.
This implies that by behaving interoceptively with respect to engineering stress, robots evolved the ability to ameliorate large deviations from their expected material properties, but by behaving interoceptively with respect to pressure, robots did not evolve this character. Because development was manually removed beforehand, robustness in our case was not a matter of changing one’s body, as in the example of plant growth (Sultan, 2000); rather, it is an intrinsic property of structure (geometries and actuation patterns) educed from ancestors who changed in response to one particular internal state (stress), but not from those who responded to another (pressure).
The difference in robustness between nonadaptive robots and stress-adaptive, but not pressure-adaptive robots, could be due in part to the fact that there are simply more pressure-adaptive robots than stress-adaptive robots. While robots tend to be more fit than and robots , they also appear to exploit their material properties to a greater degree, and are thus more sensitive to changes in its constitution, compared to and robots .
The robot generates movement by pushing off its posterior leg, which must be rigid enough to support itself as well as propel forward the center portion of its anterior wall (e.g. run 12 in Fig. 4). The robot loses kinetic energy, which is stored as elastic strain energy in the spring-like voxels between the wall’s center and edge. The most strain is present in the dorsal portion of the anterior wall. The springs recoil, restoring kinetic energy and generating forward motion. If the posterior leg is too soft, or the dorsal anterior wall too rigid, the robot can suffer a large drop in performance.
Differences in geometry, however, shed no light on why (the reduced) stress-adaptive robots are more robust than nonadaptive robots: the level of significance
does not change after removing the only nonadaptive robot that could possibly be classified as(Fig. 2, run 19). Thus we continue our investigation by analyzing how stress and pressure might differentially affect the rate of developmental reactions.
One indication of canalization (Waddington, 1942; Kriegman et al., 2017a) in our system is given by the magnitude of in each voxel, as defined by Eqs. 2 and 3. However, this is but one of two necessary ingredients for a developmental reaction: it indicates bodywide responsiveness to potential stimuli, but ignores the actual stimulus.
Thus, as proxy for canalization, we measured the amount of morphological change in reaction to local stimulus, during evaluation. More precisely, we recorded the mean, across the body, of relative lifetime change in stiffness
where is the congenital stiffness, is the final stiffness, and contains the coordinates of each voxel present in the (bit array) geometry which has cardinality (total voxels). Less change—lower —indicates more canalization.
On average, voxels in stress-adaptive robots change their relative stiffness less than voxels in pressure-adaptive robots (Fig. 5D). In other words, developmental reactions are canalized to a greater extent in stress-adaptive robots. It follows, then, that the treatment with increased robustness was also the treatment with increased canalization.
To get a sense of the consistency of developmental reactions, as they occur across the body of evolved robots, we also recorded the spatial variance of this relative lifetime change
By this measure, stress-adaptive robots exhibit more uniform reactions than pressure-adaptive robots (Fig. 5E).
Taken together, then, we may say that the developmental reactions of stress-adaptive robots are more uniform in space (lower ; Fig. 5E), and more canalized in magnitude (lower ; Fig. 5D) than those of pressure-adaptive robots. Pressure-adaptive robots therefore experience larger and more localized changes in stiffness during their lifetime.
There are two possibilities that could explain this more localized change in stiffness in the pressure-adaptive robots. One possibility is that there is greater variance among the in the pressure-adaptive robots. The alternative is that there is greater variance in the application of pressure throughout the body. To test the first possibility, we first normalized in pressure- and stress-adaptive robots by the differing ranges of that evolved in the pressure-adaptive (-5.36 to 5.63) and stress-adaptive (-10.00 to 6.42) robots. Then we took the variance of across the body of each run champion, individually:
We found no evidence to support the hypothesis that in pressure-adaptive robots vary more (or less) in space than those in stress-adaptive robots (Fig. 5F).
Therefore, because there is no difference in the variation of (Fig. 5F), and because cannot change during operation (Eqs. 2 and 3), it follows that pressure was generally much more localized within the bodies of the pressure-adaptive robots than stress within the bodies of the stress-adaptive robots. In other words, the entire body plan encountered stress, but only a small portion of the body encountered appreciable pressure. (An example of this localized response to localized pressure can be seen in Fig. 1.) We hypothesize that this global spread of stress is the likely cause of increased robustness in the stress-adaptive robots (Fig. 5C).
Building systems that are robust in the face of changing environmental conditions is a grand challenge in robotics and AI. The brittleness of current systems is exemplified by the growing literature on adversarial examples (Szegedy et al., 2013; Nguyen et al., 2015; Athalye and Sutskever, 2017), and the fact that almost all practical robots are confined to the perfectly flat floors they clean, or the hermetic factories built around their work. Robustness is not unknown in human-engineered systems, but it is relatively rare; in nature it is everywhere, and one of the reasons is that in nature organisms develop: They constantly change not just their cognitive architectures but the morphologies that contain them and mediate with the external world.
It has been shown for rigid robots (Bongard, 2011) that morphological development can in some cases increase robustness since it exposes evolution to richer sensory information: the robot must maintain locomotion while changing its body. Soft robots have much greater potential in this domain: If soft, there are more ways that morphology can change, so by definition the increase in breadth in sensorimotor experiment induced by development will be even greater than that for developing yet rigid machines. Toward this goal, by allowing material stiffness to be plastic, we have here investigated a heretofore unexplored dimension of morphological change (stiffness) not available to rigid robots.
Advances in materials science and 3D printing promise new engineered systems—protean machines—that may continuously morph in response to changing environmental signals. Simply put, if a robot always changes its strategy along many morphological and neural modalities, it is more difficult to fool with a static adversarial example or a new task environment. Little to no analysis has been conducted, however, into how such systems should respond to environmental stimuli in order to adapt their functions in the face of changing environmental conditions.
In initiating such a study here, we have shown that it is not just a matter of reacting to any stimulus: different types of developmental feedback loops elicit different evolved properties. We observed that if one modality (stiffness) responds to one particular internal signal (engineering stress) but not another (pressure), robots evolved structure that intrinsically buffered large deviations from their expected material properties.
Pressure and stress bear distinct mechanical load signatures which in turn stimulated very different developmental reactions. Intriguingly, increased robustness was correlated with increased canalization: developmental reactions with stress were canalized to a greater degree than those with pressure. Although developmental reactions with pressure did not afford the evolution of robustness here, it did increase evolutionary divergence: pressure-adaptive robots evolved more diverse (congenital) shapes than stress-adaptive robots. Our work here suggests there may be other developmental feedback loops that could be made available to evolution that would lead to more diverse and robust robots.
For our purposes, ‘morphology’ is a robot body, but the concepts here could equally be applied to non-embodied systems, such as the architectures of deep neural networks (Miikkulainen et al., 2017; Zoph and Le, 2016). One could define internal neural processes such as node sharpening (French, 1994), Hebbian learning, or neurotransmitter diffusion (Husbands et al., 1998; Velez and Clune, 2017)
as interoceptive signals to which the neural network developmentally responds in a structural manner, such as adding or removing neurons. Meanwhile, at a faster time scale, synaptic weights might be tuned in response to exteroceptive signals such as gradients of a loss function. Finally, such a network could be placed inside a robot which itself is experiencing morphological change.
This work was supported by Army Research Office award W911NF-16-1-10304 and DARPA contract HR0011-18-2-0022. The computational resources provided by the UVM’s Vermont Advanced Computing Core (VACC) are gratefully acknowledged.