1 Introduction
Conceptual engineering design is an iterative process [5]. Under the paradigm commonly called ideation [3] a design problem is defined, the design space explored, candidate solutions evaluated, and finally design decisions are taken, which put constraints onto the next design iteration.
In a 2014 interview study by Bradner [3] on the realworld usage of automation in design optimization, “participants reported consulting Pareto plots iteratively in the conceptual design phase to rapidly identify and select interesting solutions”. This process of a posteriori articulation of preference [9] is described by the “design by shopping” paradigm [1, 23]. A Pareto front of optima is created by a multiobjective optimization algorithm, after which engineers choose a solution to their liking. That participants used optimization algorithms to develop preliminary solutions to solve a problem surprised the interviewers.
Design optimization has been applied to multimodal problems, using niching and crowding to enforce diversity in evolutionary optimization algorithms [21, 22]. For optimization algorithms to operate effectively in cases where evaluation of designs is computationally expensive, surrogate assistance is applied, using predictive models that replace most of the evaluations [10]. Recently introduced qualitydiversity (QD) algorithms, like NSLC [11] and MAPElites [13]
, are evolutionary algorithms capable of producing a large array of solutions constituting different behaviors or design features. Surrogate assistance was introduced for QD algorithms
[6] as well. It enables finding thousands of designs, using orders of magnitude fewer evaluations than running MAPElites without a surrogate. However, this large number of solutions can hinder the engineer’s ability to select interesting designs.As automated diversity gives too many solutions, their more concise presentation makes QD more useful to designers. In this paper we apply the design by shopping paradigm to QD, assisting design decisions by representing similar solutions succinctly with a representative solution using prototype theory. Therein, objects are part of the same class based on resemblance. Wittgenstein [26] questioned whether classes can even be rigidly limited, implying that there is such a thing as a distance to a class. Rosch later introduced prototype theory [18], stating that natural classes consist of a prototype, the best representative of its class, and nonprototypical examples, which can be ranked in terms of distance to the prototype. However, while feature diversity is enforced, surrogateassisted QD uses no metric for genotypic similarity in terms of the actual design space.
By applying prototype theory to the variety of designs produced by QD algorithms, computeraided ideation (CAI) is introduced (Fig. 1), allowing the same a posteriori articulation of preference as in the design by shopping paradigm. Although performance and diversity can both be formally described and optimized, design decisions are based on the intuition of the engineer, and cannot be automated. QD is used to discover a first set of optimal solutions. Then, by clustering similar solutions into classes and representative prototypes, the optimization process is guided by extracting seeds from the classes selected by the user, zooming in on a particular region in design space.
QD allows a paradigm shift in optimal engineering design, but integration of QD algorithms into the ideation process has yet to be studied extensively. In this work we introduce a CAI algorithm that takes advantage of recently introduced QD algorithms [11, 13]. Prototype Discovery using QualityDiversity (PRODUQD) [pr@”d2kt], which performs a representative selection of designs, enables engineers to make design decisions more easily and influence the search for optimal solutions. PRODUQD can find solutions similar to a selection of prototypes that perform similarly well as solutions that were found by searching the entire design space with QD. By integrating QD algorithms and ideation a new framework for design is created; a paradigm which uses optimization tools to empower human intuition rather than replace it.
2 Related Work
2.1 QualityDiversity and Surrogate Assistance
QD algorithms, like Novelty Search with Local Competition (NSLC) [11] and Multidimensional Archive of Phenotypic Elites (MAPElites) [13]
, use a lowdimensional behavior or feature characterization, such as neural network complexity or curvature of a design, to determine similarity between solutions
[16]. Solutions compete locally in feature space, superseding similar yet less fit solutions. In MAPElites, the feature space consists of a discrete grid of behavior or feature dimensions, called a feature map. Every bin in the map is either empty or holds a solution, called an elite, that is currently the best performing one in its niche. QD is able to produce many solutions with a diverse set of behaviors and is very similar to the idea of Zwicky’s morphological box [27] as it allows new creations by combining known solution configurations. QD algorithms perform many evaluations, making them unsuitable for design problems that need computationally expensive or real world evaluation.To decrease the number of expensive objective evaluations, approximative surrogate models replace the objective function close to optimal solutions using appropriate examples [10]. To sample the design space effectively and efficiently, Bayesian Optimization is used. Given a prior over the objective function, evidence from known samples is used to select the next best observation. This decision is based on an acquisition function that balances exploration of the design space, sampling from unknown regions, and exploitation, choosing samples that are likely to perform well. This way, the surrogate model becomes more accurate in optimal regions during sampling. The most common surrogates used are Gaussian Process (GP) regression models [17].
Surrogate assistance has been applied to QD with SurrogateAssisted Illumination (SAIL) [6]. In SAIL the GP model is pretrained with solutions evenly sampled in the parameter space with a Sobol sequence [14]
. The sequence allows iteratively finer sampling, approximating a uniform distribution. Then, using the upper confidence bound (UCB) acquisition function, an
acquisition map is created, containing a diverse set of candidate training samples. UCB, described by the function , is a balance between exploitation (, the mean prediction of the model), and exploration (, the model’s uncertainty), parameterized by .The acquisition map is first seeded with the previously acquired samples, assigning them to empty bins or replacing less performant solutions. MAPElites is then used in conjunction with the GP model to fill and optimize the acquisition map, using UCB as a fitness function and combining existing solutions from bins, illuminating the surrogate model through the ”lens” of feature map. A candidate sample is created for every bin in the map. Then, the acquisition map is sampled using a Sobol sequence and selected solutions are evaluated using the expensive evaluation function. After gathering a given number of samples, the acquisition function is adapted by removing the model’s uncertainty and the final prediction map, seeded with the set of known samples, is illuminated, producing a discrete map of diverse yet highperforming solutions.
2.2 Dimensionality Reduction
Clustering depends on a notion of distance between points. The curse of dimensionality dictates that the relative difference of the distances of the closest and farthest data points goes to zero as the dimensionality increases
[2]. Clustering methods using a distance metric break down and cannot differentiate between points belonging to the same or to other clusters [25]. Dimensionality reduction (DR) methods are applied to deal with this problem. Data is often located at or close to a manifold of lower dimension than the original space. DR transforms the data into a lowerdimensional space, enabling the clustering method to better distinguish clusters [25]. Common DR methods are Principal Component Analysis (PCA)
[15], kernel PCA (kPCA) [20], Isomap [24][8] and tdistributed Stochastic Neighbourhood Embedding (tSNE) [12].tSNE is commonly used for visualization and has been shown to be capable of retaining the local structure of the data, as well as revealing clusters at several scales. It does so by finding a lowerdimensional distribution of points that is similar to the original highdimensional distribution . The similarity of datapoint to datapoint
is the conditional probability (
for and for , Eq. 1), that would pickas its neighbor if neighbors were picked in proportion to their probability density under a Gaussian distribution centered at
. The Studentt distribution is used to measure similarities between lowdimensional points in order to allow dissimilar objects to be modeled far apart in the map (Eq. 1).(1) 
The local scale is adapted to the density of the data (smaller in denser parts). is set such that perplexity of the conditional distribution equals a predefined value. The perplexity of a distribution defines how many neighbors for each data point have a significant and can be calculated using the Shannon entropy of the distribution around (Eq. 2).
(2) 
(3) 
Using the bisection method, are changed such that approximates the preset value (commonly 550). The similarity of to and to is absorbed with the joint probability . A lowdimensional map is learned that reflects all similarities as well as possible. Locations
are determined by iteratively minimizing the KullbackLeibler divergence of the distribution
from the distribution (Eq. 3) with gradient descent.3 Prototype Discovery using QualityDiversity
PRODUQD is an example of a CAI algorithm (Fig. 2). Initially, the design space is explored with a QD algorithm. SAIL [6] is used to explore the design space, creating highperforming examples of designs with varying features. These features can be directly extracted from design metrics, for instance the amount of head space in a car. SAIL produces a prediction map that contains a set of diverse highperforming solutions.
A similarity space is constructed using tSNE. In this space, similar solutions are clustered into classes. Since no prior knowledge on the structure of optimality in design space is available and due to the stochastic nature of QD, the number and density of clusters is unknown. To group the designs into clusters we use the wellknown density based clustering algorithm DBSCAN [4]^{1}^{1}1DBSCAN’s parameterization is automated using the L Method [19]..
A prototype is then extracted for every class. According to prototype theory [18], prototypes are those members of a class, “with the most attributes in common with other members of the class and least attributes in common with other classes”. The most representative solution of a class is the member of a cluster that has the minimum distance to other members. The medoid is taken rather than a mean of the parameters, as this mean could lie in nonoptimal or even invalid regions of the design space.
The prototypes are presented to the user, offering them a concise overview of the diversity in the generated designs. After the user selects one or more prototypes, the affiliated class members are used as seeds for the next SAIL iteration, serving as initial solutions in the acquisition and prediction maps. Initializing SAIL with individuals from the chosen class forces SAIL to start its search within the class boundaries. Using a subset of untested solutions of a particular class stands in contrast to SAIL, which focuses on searching the entire design space, seeding both maps with actual samples. Within each SAIL iteration, the GP surrogate is retrained whenever new solutions are evaluated.
A precise definition of PRODUQD can be found in Alg. 1, including the use of the selected seeds in SAIL. This ideation process explores the design space while taking into account online design decisions.
4 Evaluation
PRODUQD is a tool for CAI which allows the optimization and exploration of QD to be focused to produce designs which resemble userchosen prototypes. We show that PRODUQD creates solutions of comparable performance to SAIL, produces models with the same level of accuracy, while directing the search towards design regions chosen by the user.
4.0.1 2D Domain
PRODUQD and SAIL are applied to a classic design problem, similar to [6]
, but with a different representation. A highperforming 2D airfoil is optimized using free form deformation, with 10 degrees of freedom (Fig.
3). The base shape, an RAE2822 airfoil, is evaluated in XFOIL^{2}^{2}2web.mit.edu/drela/Public/web/xfoil/, at an angle of attack of 2.7 at Mach 0.5 and Reynolds number 10.The objective is to find diverse deformations, minimizing the drag coefficient while keeping a similar lift force and area, described by and Eq. 45. The feature map, consisting of the x and y coordinates of the highest point on the foil ( and , see Fig. 3), is divided into a 25x25 grid.
(4) 
(5) 
4.0.2 3D Domain
To showcase CAI on a more complex domain, the side mirror of the DrivAer [7] car model is optimized with a 51 parameter free form deformation (Fig. 3 (right)). The objective is to find many diverse solutions while minimizing the drag force (in Newtons) of the mirror. The numerical solver OpenFOAM^{3}^{3}3openfoam.org, simulation at 11 m/s. is used to determine flow characteristics and calculate the drag force. The feature map, consisting of the curvature of the edge of the reflective part of the mirror and the length of the mirror in flow direction, is divided into a 16x16 grid.
4.0.3 Choice of Dimensionality Reduction Technique
Various DR methods are analyzed as to whether they improve the clustering behavior of DBSCAN compared to applying clustering on the original dimensions. , a measure of the discordance between pairs of point distances and is robust w.r.t. differences in dimensionality [25], is used as a metric. It indicates whether members of the same cluster are closer together than members of different clusters. A low value () indicates a high quality of clustering. PCA, kPCA, Isomap, tSNE^{4}^{4}4Perplexity is set to 50, but at most half the number of samples. and an Autoencoder are compared using DBSCAN on the latent spaces. tSNE has been heavily tested for a dimensionality reduction to two dimensions. To allow a fair comparison, the same reduction was performed with the alternative algorithms.
Original  PCA  kPCA  Isomap  tSNE  Autoencoder  
avg. score  0.36  0.33  0.22  0.30  0.05  0.454 (0.17) 
avg. number of clusters  4  5  7  4  10  4 (1.23) 
Quality of DR methods. Variance of the Autoencoder in parentheses.
SAIL is performed 30 times on the 2D airfoil domain. For every run of SAIL, the dimensionality of the resulting predicted optima is reduced with the various methods and the optima are clustered with DBSCAN. Table 1 shows that tSNE allows DBSCAN to perform about an order of magnitude better than using other methods. Although tSNE is not a convex method, it shows no variance, indicating that the method is quite robust. The number of clusters found is about twice as high as using other methods, and since the cluster separation is of higher quality, tSNE is selected as a DR method in the rest of this evaluation.
4.0.4 Quantitative Analysis
To show PRODUQD’s ability to produce designs based on a chosen prototype, it is replicated five times, selecting a different class in each run. In every iteration of PRODUQD 10 iterations of SAIL are run to acquire 100 new airfoils. The first iteration starts with an initial set of 50 samples from a Sobol sequence. Then, the five classes containing the largest number of optima are selected, and the algorithm is continued in separate runs for each class. After each iteration, the we select the prototype that is closest to the one that was selected in the first iteration. PRODUQD runs are compared to the original SAIL algorithm, using the same number of samples, a total of 500.
The similarity of designs to prototypes of optima found in four separate runs, selecting a different prototype in each one, are shown in Fig. 4 (left). The usage of seeds does not always prevent the ideation algorithm of finding optima outside of the selection, but PRODUQD produces solutions that are more similar to the selected prototype than SAIL. The parameter spread in solutions found with PRODUQD is lower than with SAIL. Yet the true fitness scores and surrogate model prediction errors of both SAIL and PRODUQD are very similar.
Fig. 5 shows the similarity space of three consecutive iterations. The effect of selection, zooming in on a particular region, can be seen by the fact that later iterations cover a larger part of similarity space, close to the prototype that was selected. Some designs still end up close to nonselected classes (in gray), which cannot be fully prevented without using constraints. PRODUQD is able to successfully illuminate local structure of the objective function around a prototype. It finds optima within a selected class of similar fitness to optima found in SAIL using no selection, and is able to represent the solutions in a class in a more concise way, using prototypes as representatives, shown by the decreased variance within classes (Fig. 4).
The performance of PRODUQD’s designs is comparable to SAIL while directing the search towards design regions chosen by the user.
4.0.5 Qualitative Analysis
A twodimensional feature map, consisting of the curvature and the length of the mirror in flow direction (Fig. 3), is illuminated from an initial set of 100 car mirror designs from a Sobol sequence. After acquiring 200 new samples with SAIL, a prediction map is produced and from this set of solutions the two prototypes having the greatest distance to each other are selected and PRODUQD is continued in two separate instances, sampling another 100 examples. Then the newly discovered prototype that is closest to the one first selected is used to perform two more iterations, resulting in a surrogate model trained with 600 samples. The two resulting runs are shown in Fig. 6. Every branch in the phylogenetic tree of designs represents a selected prototype and every layer contains the prototypes found in an iteration. prototypes were found on average in each iteration. The surrogate model gives an accurate prediction of the drag force of all classes.
5 Conclusion
Qualitydiversity algorithms can produce a large array of solutions, possibly impeding an engineer’s capabilities of making a design decision. We introduce computeraided ideation, using QD in conjunction with a state of the art dimensionality reduction and a standard clustering technique, grouping similar solutions into classes and their representative prototypes. These prototypes can be selected to constrain QD in a next iteration of design space exploration by seeding it with the selected class. A posteriori articulation of preference allows automated design exploration under the design by shopping paradigm. Decisions can be based on an engineer’s personal experience and intuition or other “softer” design criteria that can not be easily formalized. PRODUQD, an example of such a CAI algorithm, allows an engineer to partially unfold a phylogenetic tree of designs by selecting prototypical solutions.
The similarity space can be used continuously as it is decoupled from the feature map. This allows the diversity metric, the feature characterization, to change between iterations. The order in which the feature dimensions are chosen can be customized depending on the design process. For example, the engineer can start searching the design space in terms of diversity of design and later on switch to functional features. In future work, changes in the feature map and their effects on PRODUQD will be analyzed. Although seeding the map proves to be sufficient to guide QD towards the selected prototype, it is not sufficient to guarantee that QD only produces solutions within its class. Constraints could limit the search operation. Adding the distance to the selected prototype to the acquisition function could bias sampling to take place within the class. Finally, although the median solution might be most similar to all solutions within a class, one indeed might choose the fittest solution of a class as its representative.
CAI externalizes the creative design process, building up a design vocabulary by concisely describing many possible optimal designs with representative prototypes. Engineers can cooperate using this vocabulary to make design decisions, whereby ideation allows them to understand the design space not only in general, but around selected prototypes. CAI, a new engineering design paradigm, automates humanlike search whilst putting the human back into the loop.
Acknowledgments
This work received funding from the German Federal Ministry of Education and Research (BMBF) under the Forschung an Fachhochschulen mit Unternehmen programme (grant agreement number 03FH012PX5 project ”Aeromat”). The authors would like to thank Adam Gaier for their feedback.
References
 [1] Balling, R.: Design by Shopping: a New Paradigm? In: Third World Congress of Structural and Multidisciplinary Optimization, pp. 295–297. ISSMO, New York (1999)
 [2] Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is ”Nearest Neighbor” Meaningful? In: International Conference on Database Theory, pp. 217–235. Springer, Berlin, Heidelberg (1999)
 [3] Bradner, E., Iorio, F., Davis, M.: Parameters Tell the Design Story: Ideation and Abstraction in Design Optimization. In: Symposium on Simulation for Architecture & Urban Design, pp. 172–197. SCSI, San Diego (2014)
 [4] Ester, M., Kriegel, H. P., Sander, J., Xu, X.: A DensityBased Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In: 2nd International Conference on Knowledge Discovery and Data Mining, pp. 226–231. AAAI Press, Portland (1996)
 [5] Flager, F., Haymaker, J.: A Comparison of Multidisciplinary Design, Analysis and Optimization Processes in the Building Construction and Aerospace Industries. 24th W78 Conference on Bringing ITC Knowledge to Work, pp. 625–630. Elsevier, Maribor (2007)

[6]
Gaier, A., Asteroth, A., Mouret, J.: DataEfficient Exploration, Optimization, and Modeling of Diverse Designs through SurrogateAssisted Illumination. In: Genetic and Evolutionary Computation Conference, pp. 99–106. ACM, Berlin (2017)
 [7] Heft, Angelina I., Thomas Indinger, Nikolaus A. Adams.: Introduction of a New Realistic Generic Car Model for Aerodynamic Investigations. SAE 2012 World Congress, Technical report. SAE, Detroit (2012)
 [8] Hinton, Geoffrey E., Ruslan R. Salakhutdinov.: Reducing the Dimensionality of Data with Neural Networks. Science 313.5786, pp. 504–507 (2006) .
 [9] Hwang, C.L., Masud, A.S.M.: Multiple Objective Decision Making  Methods and Applications: A StateoftheArt Survey (Vol. 164). Springer Science & Business Media, New York (1979)
 [10] Jin, Y.: Surrogateassisted Evolutionary Computation: Recent Advances and Future Challenges. Swarm and Evolutionary Computation 1 (2), pp. 61–70 (2011)
 [11] Lehman, J., Stanley, K. O.: Evolving a Diversity of Virtual Creatures Through Novelty Search and Local Competition. Genetic and Evolutionary Computation Conference, pp. 211–218. ACM, Dublin (2011)

[12]
Maaten, L. van der, Hinton, G.: Visualizing Data Using TSNE. Journal of Machine Learning Research 9 (Nov), pp. 2579–2605 (2008)
 [13] Mouret, J.B., Clune, J.: Illuminating Search Spaces by Mapping Elites. arXiv:1504.04909 (2015)
 [14] Niederreiter, H.: Lowdiscrepancy and Lowdispersion Sequences. Journal of Number Theory 30, pp. 51–70 (1988)
 [15] Pearson, K.: On Lines and Planes of Closest Fit to Systems of Points in Space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 2, pp. 559–572 (1901)
 [16] Pugh, J. K., Soros, L. B., Stanley, K. O.: Quality Diversity: A New Frontier for Evolutionary Computation. Frontiers in Robotics and AI 3, pp. 1–17 (2016)
 [17] Rasmussen, C. E.: Gaussian Processes for Machine Learning. MIT press, Cambridge (2006)
 [18] Rosch, E.: Cognitive Reference Points. Cognitive Psychology 7.4, pp. 532547 (1975)

[19]
Salvador, S., Chan, P.: Determining the Number of Clusters/Segments in Hierarchical Clustering/Segmentation Algorithms. In: 16th IEEE International Conference on Tools with Artificial Intelligence, pp. 576–584. IEEE, Boston (2003)
 [20] Schölkopf, B., Smola, A., Müller, K.R.: Kernel Principal Component Analysis. In: International Conference on Artificial Neural Networks, pp. 583–588. Springer, Lausanne (1997)
 [21] Shir, O.M., Bäck, T.: Niching in Evolution Strategies. In: 7th Annual Conference on Genetic and Evolutionary Computation, pp. 915–916. ACM, Washington (2005)
 [22] Singh, G., Deb, K.: Comparison of Multimodal Optimization Algorithms Based on Evolutionary Algorithms. In: 8th Annual Conference on Genetic and Evolutionary Computation, pp. 1305–1312. ACM, Seattle (2006)
 [23] Stump, G.: Design Space Visualization and Its Application To a Design By Shopping Paradigm. In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 795–804. ASME, Chicago (2003)
 [24] Tenenbaum, J. B., de Silva, V., Langford, J. C.: A Global Framework for Nonlinear Dimensionality reduction. Science 290.5500, pp. 2319–2323 (2000)

[25]
Tomašev, N., Radovanović, M.: Clustering Evaluation in Highdimensional Data. Unsupervised Learning Algorithms, pp. 71–107 (2016)
 [26] Wittgenstein, L.: Philosophische Untersuchungen, Basil Blackwell, Oxford (1953)
 [27] Zwicky, F.: Discovery, Invention, Research Through the Morphological Approach. Macmillan, New York (1969)