 # How to Read Many-Objective Solution Sets in Parallel Coordinates

Rapid development of evolutionary algorithms in handling many-objective optimization problems requires viable methods of visualizing a high-dimensional solution set. Parallel coordinates which scale well to high-dimensional data are such a method, and have been frequently used in evolutionary many-objective optimization. However, the parallel coordinates plot is not as straightforward as the classic scatter plot to present the information contained in a solution set. In this paper, we make some observations of the parallel coordinates plot, in terms of comparing the quality of solution sets, understanding the shape and distribution of a solution set, and reflecting the relation between objectives. We hope that these observations could provide some guidelines as to the proper use of parallel coordinates in evolutionary many-objective optimization.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

The classic scatter plot is a basic tool in viewing solution vectors in multiobjective optimization. It allows us to observe/perceive the quality of a set of solutions, the shape and distribution of a set of solutions, the relation between objectives (e.g., the extent of their conflict), etc. Unfortunately, the scatter plot may only be done readily in a 2D or 3D Cartesian coordinate space

111The scatter plot matrix  is scalable to any dimension, but it only reflects the relation between two objectives.. It could be difficult for people to comprehend the scatter plot in a higher-dimensional space.

An alternative to view data with four or more dimensions is using parallel coordinates [2, 3, 4] (also called value paths ). Parallel coordinates display multidimensional data (a set of vectors) in a two-dimensional graph, with each dimension of the original data being translated onto a vertical axis in the graph, and a vector is represented as a polyline with vertices on the axes. As a visualization tool, parallel coordinates have received modest attention in the early stage of evolutionary multiobjective optimization (EMO) [6, 7]. As many-objective optimization (i.e., the number of objectives to be optimized being larger than three [8, 9]) becomes a new research topic in the EMO area, there has been increasing interest in presenting solution vectors in the high-dimensional space. Parallel coordinates which are scalable to objective dimensionality naturally become a good alternative to do so . Now the parallel coordinates plot has been dominantly used in many-objective optimization despite the emergence of various visualization techniques [11, 5, 12, 13]. This includes it being used to investigate the search behavior of algorithms [14, 15, 16], to examine preference-based search [17, 18, 19], to compare different solution sets [20, 21, 22], to verify performance metrics [23, 24, 25], and furthermore to help design new many-objective optimizers [26, 27].

Despite the popularity, the parallel coordinates plot is not as straightforward as the scatter plot in presenting the information contained in a solution set. Due to mapping multi-dimensional data onto a lower 2D space, the loss of information is inevitable. This could naturally lead to questions raised. Specifically, in the context of multiobjective optimization, one may ask

• Can the parallel coordinates plot indicate the quality of a solution set, e.g., its convergence, extensity, uniformity and coverage?

• Can the parallel coordinates plot imply the shape and distribution of a solution set? In other words, what can we see from the pattern of solution lines in parallel coordinates?

• How much information can the parallel coordinates plot tell in terms of the relation among objectives? To be specific, does the order of objectives displayed in parallel coordinates matter?

In this paper, we make some observations on the above questions, attempting to provide some guidelines as to the use of parallel coordinates in evolutionary multiobjective optimization. The rest of the paper is organized as follows. Section II briefly introduces parallel coordinates. Sections III–V are devoted to answering those three questions, respectively. Section VI describe how to draw a parallel coordinates plot. Section VI concludes the paper and presents some possible future research lines.

## 2 Parallel Coordinates

To show a set of points in an -dimensional space, parallel coordinates map them onto a 2D graph, with parallel axes being plotted, typically vertical and equally spaced. A point in an -dimensional space is represented as a polyline with vertices on these parallel axes, and the position of the vertex on the -th axis corresponds to the value of the point on the -th dimension. Parallel coordinates are simple to construct and scale well with the dimensionality of data. Adding more dimensions only involves adding more axes. Figure 1 presents an example of the parallel coordinates plot, where three 4D points are mapped to three polylines, respectively. Figure 1: The parallel coordinates plot of three four-dimensional points a (15,31,20,50), b (10,18,2,30) and c (20,5,32,20).

## 3 Quality Measuring

Given dimensionality reduction in the mapping of parallel coordinates, some loss of information is expected. In this section, we will see what and how much information parallel coordinates can preserve and reflect in terms of the quality of a solution set in multiobjective optimization.

Often, the quality of a solution set in multiobjective optimization can be reflected via four measures: convergence, coverage, uniformity, and extensity. Convergence of a solution set measures the closeness of the set to the Pareto front; coverage considers the region of the set covering in comparison with the whole Pareto front; uniformity quantifies the distance between neighboring points in the set; and extensity refers to the range of the set. In general, there is no clear conceptual difference of these quality measures between many-objective optimization and multi-objective optimization with two or three objectives. However, many-objective optimization typically poses bigger challenge for evolutionary algorithms to achieve a good balance among these aspects.

A straightforward feature that parallel coordinates can tell is the the extensity of a solution set. This feature can make parallel coordinates replace extensity metrics in EMO, e.g., maximum spread . In the following, we will discuss if parallel coordinates can reflect other aspects of a solution set’s quality, i.e., the convergence, coverage and uniformity.

### 3.1 Convergence

In multiobjective optimization, Pareto dominance is a fundamental criterion to compare solutions in terms of convergence. Parallel coordinates can clearly reflect the Pareto dominance relation between two solutions (such as polyline being dominated by polyline in Figure 1, assuming a minimization problem scenario) if the solution polylines are not overcrowded. It is worth mentioning that one can remove dominated solutions in parallel coordinates if they are only interested in nondominated ones. This may make the plot clearer especially when comparing the quality of solution sets.

In addition to reflecting the Pareto dominance relation, the range that a solution set in parallel coordinates can largely imply its convergence. Figure 2 is such an example, where the parallel coordinates representation of two solution sets obtained by one run222The setting of the population size and maximum evaluations was 100 and 30,000, respectively. This setting was used in all conducted experiments in this paper, unless explicitly mentioned otherwise. In addition, the grid division in GrEA was set to 8. of two EMO algorithms, NSGA-II  and GrEA , on the 10-objective DTLZ2 problem  is shown. As can be seen, NSGA-II fails to converge, with its solution set ranging from 0 to around 3.5 in contrast to the problem’s Pareto front ranging from 0 to 1. GrEA has a good convergence on this problem and its solution set has the same range as the Pareto front. These observations can be confirmed by the results of the convergence metric GD  shown in the figure. GD is a modified version of the original GD , which makes it compatible with Pareto dominance. Figure 2: The solution set obtained by NSGA-II and GrEA on the 10-objective DTLZ2, and their evaluation results on the convergence measure GD+ (the smaller the better). Figure 3: The solution set obtained by NSGA-II and GrEA on the 10-objective WFG7, and their evaluation results on the convergence metric GD+ (the smaller the better).

However, we may not be able to accurately know the convergence of solution sets by their range shown in parallel coordinates. That is, even if two solution sets are located in the same range, they can perform considerably differently in terms of convergence. Figure 3 gives such an example, where solution sets obtained by one run of NSGA-II and GrEA on the 10-objective WFG7 problem  are shown. As seen, both algorithms virtually reach the range of the Pareto front (from to where is the objective index of the problem), but they have different GD results. NSGA-II is returned a significantly higher (worse) GD value than GrEA. This occurrence can be caused by two possibilities. One is that the solution set of NSGA-II is not actually close to the Pareto front. The other is that most of solutions in the set converge already while a small portion of set is far away (but still in the range of the Pareto front).

### 3.2 Coverage

In parallel coordinates, it is straightforward to see which region a solution set does not reach on any objective333Note that for real-world problems whose Pareto front is unknown, we cannot tell if a solution set reaches the optimal region of objectives or not.. For example, in Figure 4 the solution set obtained by the AR method  concentrates in one tiny area and the set by IBEA  fails to cover the first six objectives on the 10-objective DTLZ2. Moreover, we can conjecture some distribution features of solution sets from their parallel coordinates representation. Take the solution sets of Figure 5 as an example; their parallel coordinates representation is shown in Figure 6. From Figure 6, we can know that the solution sets of IBEA and SMS-EMOA  fail to cover the region between 0 and 0.2 on all three objectives. Also, most of the solutions obtained by IBEA are located in the boundary of the Pareto front as there are very few lines distributed around the middle section on all three objectives in the figure.

However, there do exist some cases that different solution sets have the same parallel coordinates plots. We can easily construct such an example. In Figure 7, solution set B has a better coverage than set A (the four solutions in set A being duplicate), but the two sets have the same pattern in parallel coordinates (Figure 8). Note that if we change the order of some objectives (e.g., and ), the parallel coordinates of the two solution sets in this example would be different.

One important fact that we would like to note is that as parallel coordinates map an -dimensional graph onto a 2D graph they cannot fully reflect the coverage of solution sets. A set of solutions (represented by polylines) may have a good coverage over the range of the Pareto front in the 2D graph, but they may only cover part of the Pareto front in the original -dimensional space. An interesting example is shown in Figure 9. In that figure, NSGA-II appears to have a better coverage than GrEA according to the parallel coordinates plots, but GrEA has a better coverage evaluation result, measured by the coverage metric Diversity Comparison Indicator (DCI) .

### 3.3 Uniformity

In parallel coordinates, it is not easy to see how evenly a set of solutions are distributed. However, a set of uniformly-distributed polylines in parallel coordinates often implies a uniformly-distributed solution set. As shown in Figures

10 and 11, MOEA/D  has a perfectly-distributed solution set and its corresponding polylines in parallel coordinates are distributed uniformly and regularly. This is in contrast to the solution set of NSGA-II which are distributed rather irregularly in both Cartesian and parallel coordinates plots. Note that a set of irregularly-distributed polylines may not represent a badly-distributed solution set, as uniformly-distributed solutions can have distinct values on different objectives. To show this, we select two EMO algorithms, MOEA/D  and BCE-MOEA/D , both of which are able to obtain a uniformly-distributed solution set on DTLZ2 (see Figure 12). In MOEA/D, the population distribution is maintained by a set of systematically-generated, uniformly-distributed weight vectors (within a simplex), and thus ideally its solutions only take several equivalent values on all the objectives. In contrast, in BCE-MOEA/D the population distribution is maintained by a niching-based criterion, and thus its solutions can spread over the whole range for each objective. Figure 13 gives the solution sets obtained by MOEA/D and BCE-MOEA/D on the 10-objective DTLZ2444Here the number of DTLZ2’s decision variables is set to ( is the number of objectives) to ensure that all solutions produced by algorithms are Pareto optimal; thus the uniformity measure cannot be affected by the difference of solution sets’ convergence.. As seen, on the uniformity metric Spacing (SP)555In this paper, the SP metric has been slightly modified to make it compatible with Pareto dominance. That is, if two solution sets are comparable in terms of Pareto dominance, then the SP value of the dominating set is and the SP value of the dominated set is ., BCE-MOEA/D even performs better than MOEA/D, but we cannot see this from their parallel coordinates representation in the figure. This phenomenon may happen frequently when comparing decomposition-based algorithms having a set of systematically-generated weight vectors (such as MOEA/D and NSGA-III ) with algorithms that do not use such decomposition techniques (such as SPEA2  and Two_arch2 ). So care needs to be taken when making a conclusion about the distribution uniformity of solution sets from parallel coordinates. Figure 10: The solution set obtained by NSGA-II and MOEA/D on the 3-objective DTLZ1, shown in Cartesian coordinates. Figure 11: The corresponding parallel coordinates of the solution sets in Figure 10. Figure 12: The solution set obtained by MOEA/D and BCE-MOEA/D on the 3-objective DTLZ2, shown in Cartesian coordinates.

Finally, it is worth mentioning that parallel coordinates plots can be easily cluttered with multiple lines overlaid. This may completely prevent solution sets’ distribution from being observed. Figures 14 and 15 show such an example, with two solution sets obtained by NSGA-II and SPEA2 on the 10-objective ML-DMP problem [45, 46]. The -objective ML-DMP minimizes the distance of two-dimensional points to a set of straight lines, each of which passes through one edge of a given regular polygon with vertexes. One interesting characteristic of ML-DMP is that the points in the regular polygon and their objective images are similar in the sense of Euclidean geometry. In other words, the ratio of the distance between any two points in the polygon to the distance between their corresponding objective vectors is a constant. This allows a straightforward understanding of the distribution of the objective vector set via observing the solution set in the 2D decision space. As can be seen in Figure 14, SPEA2 has a far better distribution uniformity than NSGA-II, but we cannot see the difference between their parallel coordinates representation in Figure 15. Figure 13: The solution set obtained by MOEA/D and BCE-MOEA/D on the 3-objective DTLZ2, and their evaluation results on the uniformity metric SP (the smaller the better).

## 4 Solution Set Distributions Figure 14: The solution sets (in the decision space) obtained by NSGA-II and SPEA2 on the 10-objective ML-DMP where the search space is precisely the optimal polygon. Figure 15: Parallel coordinates of the solution sets (in the objective space) in Figure 14.

In parallel coordinates, it is straightforward to know the conflict between objectives. The number of intersection lines between adjacent objectives reflects their conflicting degree. If there is no intersection of any pair of lines between adjacent objectives, then these two objectives are completely non-conflicting (i.e., harmonious ), such as objectives versus and objectives versus in Figure 16. If there are many lines intersecting, then the two objectives are heavily conflicting, such as objectives versus and objectives versus in Figure 16. If any pair of lines intersect, then the two objectives are completely conflicting to each other.

An interesting phenomenon in the parallel coordinates plot is that if all lines between two adjacent objectives intersect at one point, then these two objectives are negatively linearly dependent. Figure 17 is such an example. The four-objective ML-DMP problem minimizes the distance of points to four lines passing through the four edges of the given rectangle. From this definition, we can see that the two pairs of objectives, versus and versus , are negatively linearly dependent for the solutions in the rectangle (). Therefore, each of the objective pairs intersects at one point, as shown in Figure 17(b).

This property is the known duality between the parallel coordinates representation and the Cartesian coordinate representation of data [2, 3]: points in Cartesian coordinates map into lines in parallel coordinates, while lines in Cartesian coordinates map into points in parallel coordinates. Consider a line in the Cartesian coordinate plane of and , and consider two points lying on this line, say () and () (shown in Figure 18(a)). Figure 18(b) shows the corresponding parallel coordinates representation of the two points. For simplicity, let the distance between the vertical axes and be ; then it is easy to know that the two lines intersect at a point given by in parallel coordinates. This point depends only on and , the parameters of the original line in the Cartesian plane. This indicates that the parallel coordinates representation of any point on passes through the point . Figure 16: The solution set obtained by SPEA2+SDE  on the 5-objective DTLZ5(I,M) , where I=3. Figure 17: The solution set of SPEA2+SDE on the 4-objective ML-DMP. Figure 18: An example of a line ℓ:f2=kf1+b in the Cartesian coordinate plane corresponding to a point ρ:((1−k)−1,b(1−k)−1) in the parallel coordinates plane.

From the coordinates of the point , we can see the relation between the position of and the slope of the line . If , the intersection occurs between the two parallel coordinates axes. Especially, when the intersection is precisely midway, as in the example of Figure 17(b). If or , then the intersection point is on the left side or right side of the two coordinate axes, respectively. When or , the intersection point is on the left axis or right axis, respectively. Finally, when , the lines are parallel between the two axes in parallel coordinates. The above properties can help us understand the relation between objectives. For example, from the parallel coordinates plot in Figure 16 we know that and , where and . Figure 19: The solution set obtained by MOEA/D on the 3-objective DTLZ1.

Finally, note that since the horizontal position of the intersection point in parallel coordinates depends only on the slope , when we can see many lines between two objectives in the parallel coordinates plot intersecting at same horizontal position (but at different vertical positions), this means that many lines connecting two points in the Cartesian coordinate space have the same slope with respect to these two objectives. This occurs often when points in Cartesian coordinates are absolutely uniformly-distributed on the plane of these two objectives. The solution set in Figure 19 has such a pattern (see the midway of two adjacent objectives in Figure 19(b)). More interesting correspondence between the patterns of lines in parallel coordinates and the relation of objectives in the solution set can be found in .

## 5 Objective Order in Parallel Coordinates

In parallel coordinates, each axis has at most two neighboring axes (one on the left and one on the right). Different order of objective axes presents different information with respect to the relation between objectives. Take Figure 20 as an example. In Figure 20(a) where the order of objectives is , the conflict between any two adjacent objectives is rather weak. In contrast, in Figure 20(b) where the order of objectives is , the conflict between any two adjacent objectives is quite intense.

In a solution set with objectives, its parallel coordinates representation can only show relationships at a time. This can be a very small portion compared to the total relationships existing in objectives. Therefore, a good objective axis arrangement providing the user as much (clear) information as possible is of importance. As shown in Figures 21 and 22

, after swapping some objectives, we can see interesting patterns (linearly dependent) between some pairs of objectives. However, determining a good order of axes is nontrivial. Researchers in the data visualization area mainly focus on clutter reduction techniques

[50, 51]. In the multiobjective optimization domain, an attempt has been made to place the most harmonious objectives in a row . Figure 20: The solution set of SPEA2+SDE on the 5-objective ML-DMP, shown by different order of objectives in parallel coordinates. Figure 21: The solution set of SPEA2+SDE on the 5-objective DTLZ5(I,M) where I=3, shown by different order of objectives in parallel coordinates. Figure 22: The solution set of SPEA2+SDE on the 4-objective ML-DMP, shown by different order of objectives in parallel coordinates.

## 6 How to Draw A Parallel Coordinate Plot

In this section, we give procedures of how to plot a solution set in parallel coordinates by several commonly-used graphing tools: MS Excel, MATLAB, Latex, and Origin. Tables 14 provide the steps/codes by MS Excel, MATLAB, Latex, and Origin, respectively. For the command-line format tools (i.e., MATLAB and Latex), we use the example in Figure 1. Figure 23 also presents the graphs drawn by the four tools to that example. Finally, we would like to note that in this paper all of the parallel coordinates graphs of data examples were drawn by Origin.

## 7 Conclusions

Parallel coordinates have drawn increasing attention in many-objective optimization, but mapping a many-objective solution set onto a 2D parallel coordinates plane may not be straightforward to reveal the information contained in the set. This paper has made some observations on the use of parallel coordinates to present a solution set in many-objective optimization. In particular,

• the parallel coordinates representation of a solution set can partly reflect its convergence, coverage and uniformity. This suggests that parallel coordinates can be an assistant tool (but not entirely replacing quality metrics) in assessing a many-objective solution set.

• although the clarity can be affected by overlapping polygonal lines, parallel coordinates transform certain geometrical features of a many-objective solution set into easily seen 2D patterns.

• the order of objective axes matters in parallel coordinates. To better present the relationship between objectives, it may need to be rearranged according to features of the solution set considered. Figure 23: An example of parallel coordinates plots in MS Excel, MATLAB, Latex, and Origin.

Our subsequent study is towards overcoming/alleviating the difficulties of interpreting parallel coordinates plots presented in this paper. Particularly, how to arrange the order of objectives will be the focus of our future work as it had presented its usefulness in the paper. In this regard, a straightforward thought is to place the most conflicting objectives or the most harmonious objectives together so that people could see some meaningful patterns (such as the examples in Figures 21 and 22). Another thought is to consider the coverage of lines between objectives in a parallel coordinates plot; people may acquire more information from less coverage of the lines, for example, after exchanging the order of objectives and in Figure 8.

## Acknowledgement

This work was supported in part by the Engineering and Physical Sciences Research Council (EPSRC) of U.K. under Grants EP/K001523/1 and EP/J017515/1, and National Natural Science Foundation of China (NSFC) under Grants 61329302 and 61403326. X. Yao was also supported by a Royal Society Wolfson Research Merit Award.

## References

•  J. W. Emerson, W. A. Green, B. Schloerke, J. Crowley, D. Cook, H. Hofmann, and H. Wickham, “The generalized pairs plot,” Journal of Computational and Graphical Statistics, vol. 22, no. 1, pp. 79–91, 2013.
•  A. Inselberg, “The plane with parallel coordinates,” The visual computer, vol. 1, no. 2, pp. 69–91, 1985.
•  E. J. Wegman, “Hyperdimensional data analysis using parallel coordinates,” Journal of the American Statistical Association, vol. 85, no. 411, pp. 664–675, 1990.
•  A. Inselberg, Parallel coordinates.   Springer, 2009.
•  K. Miettinen, “Survey of methods to visualize alternatives in multiple criteria decision making problems,” OR spectrum, vol. 36, no. 1, pp. 3–37, 2014.
•  C. M. Fonseca and P. J. Fleming, “Multiobjective optimization and multiple constraint handling with evolutionary algorithms. I. A unified formulation,” IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 28, no. 1, pp. 26–37, 1998.
•  K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms.   New York: John Wiley, 2001.
•  H. Ishibuchi, N. Tsukamoto, and Y. Nojima, “Evolutionary many-objective optimization: A short review,” in

Proceedings of the IEEE Congress on Evolutionary Computation (CEC)

, 2008, pp. 2419–2426.
•  R. C. Purshouse and P. J. Fleming, “Evolutionary many-objective optimisation: An exploratory analysis,” in IEEE Congress on Evolutionary Computation (CEC), vol. 3, 2003, pp. 2066–2073.
•  P. Fleming, R. Purshouse, and R. Lygoe, “Many-objective optimization: An engineering design perspective,” in Evolutionary Multi-Criterion Optimization (EMO), 2005, pp. 14–32.
•  D. J. Walker, R. M. Everson, and J. E. Fieldsend, “Visualising mutually non-dominating solution sets in many-objective optimisation,” IEEE Transactions on Evolutionary Computation, vol. 17, no. 2, pp. 165–184, 2013.
•  T. Tusar and B. Filipic, “Visualization of Pareto front approximations in evolutionary multiobjective optimization: A critical review and the prosection method,” IEEE Transactions on Evolutionary Computation, vol. 19, no. 2, pp. 225–245, 2015.
•  Z. He and G. G. Yen, “Visualization and performance metric in many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 20, no. 3, pp. 386–402, 2016.
•  T. Wagner, N. Beume, and B. Naujoks, “Pareto-, aggregation-, and indicator-based methods in many-objective optimization,” in Evolutionary Multi-Criterion Optimization (EMO), 2007, pp. 742–756.
•  H. K. Singh, A. Isaacs, and T. Ray, “A Pareto corner search evolutionary algorithm and dimensionality reduction in many-objective optimization problems,” IEEE Transactions on Evolutionary Computation, vol. 15, no. 4, pp. 539–556, 2011.
•  M. Li, S. Yang, X. Liu, and R. Shen, “A comparative study on evolutionary algorithms for many-objective optimization,” in Evolutionary Multi-Criterion Optimization (EMO), 2013, pp. 261–275.
•  U. K. Wickramasinghe and X. Li, “Using a distance metric to guide PSO algorithms for many-objective optimization,” in Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation (GECCO), 2009, pp. 667–674.
•  R. J. Lygoe, M. Cary, and P. J. Fleming, “A many-objective optimisation decision-making process applied to automotive diesel engine calibration,” in Asia-Pacific Conference on Simulated Evolution and Learning.   Springer, 2010, pp. 638–646.
•  R. Wang, R. C. Purshouse, and P. J. Fleming, ““whatever works best for you” - A new method for a priori and progressive multi-objective optimisation,” in International Conference on Evolutionary Multi-Criterion Optimization, 2013, pp. 337–351.
•  Y. Yuan, H. Xu, B. Wang, and X. Yao, “A new dominance relation based evolutionary algorithm for many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 20, no. 1, pp. 16–37, 2016.
•  R. Cheng, Y. Jin, M. Olhofer, and B. Sendhoff, “A reference vector guided evolutionary algorithm for many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 20, no. 5, pp. 773–791, 2016.
•  Y. Xiang, Z. Yuren, M. Li, and Z. Chen, “A vector angle based evolutionary algorithm for unconstrained many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 21, no. 1, pp. 131–152, 2017.
•  H. Ishibuchi, H. Masuda, Y. Tanigaki, and Y. Nojima, “Difficulties in specifying reference points to calculate the inverted generational distance for many-objective optimization problems,” in IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM), 2014, pp. 170–177.
•  M. Li, S. Yang, and X. Liu, “A performance comparison indicator for Pareto front approximations in many-objective optimization,” in Proceedings of the 17th Confernece on Genetic and Evolutionary Computation (GECCO), 2015, pp. 703–710.
•  H. Wang, Y. Jin, and X. Yao, “Diversity assessment in many-objective optimization,” IEEE Transactions on Cybernetics, 2016, published online.
• 

W. Hu and G. G. Yen, “Adaptive multiobjective particle swarm optimization based on parallel cell coordinate system,”

IEEE Transactions on Evolutionary Computation, vol. 19, no. 1, pp. 1–18, 2015.
•  R. Hernández Gómez, C. A. Coello Coello, and E. Alba Torres, “A multi-objective evolutionary algorithm based on parallel coordinates,” in Proceedings of the 2016 on Genetic and Evolutionary Computation Conference.   ACM, 2016, pp. 565–572.
•  E. Zitzler, K. Deb, and L. Thiele, “Comparison of multiobjective evolutionary algorithms: Empirical results,” Evolutionary Computation, vol. 8, no. 2, pp. 173–195, 2000.
• 

K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: NSGA-II,”

IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182–197, 2002.
•  S. Yang, M. Li, X. Liu, and J. Zheng, “A grid-based evolutionary algorithm for many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 17, no. 5, pp. 721–736, 2013.
•  K. Deb, L. Thiele, M. Laumanns, and E. Zitzler, “Scalable test problems for evolutionary multiobjective optimization,” in Evolutionary Multiobjective Optimization. Theoretical Advances and Applications, A. Abraham, L. Jain, and R. Goldberg, Eds.   Berlin, Germany: Springer, 2005, pp. 105–145.
•  H. Ishibuchi, H. Masuda, Y. Tanigaki, and Y. Nojima, “Modified distance calculation in generational distance and inverted generational distance,” in Evolutionary Multi-Criterion Optimization (EMO), 2015, pp. 110–125.
•  D. A. Van Veldhuizen and G. B. Lamont, “Evolutionary computation and convergence to a Pareto front,” in

Late Breaking Papers at the Genetic Programming Conference

, 1998, pp. 221–228.
•  S. Huband, P. Hingston, L. Barone, and L. While, “A review of multiobjective test problems and a scalable test problem toolkit,” IEEE Transactions on Evolutionary Computation, vol. 10, no. 5, pp. 477–506, 2006.
•  P. J. Bentley and J. P. Wakefield, “Finding acceptable solutions in the Pareto-optimal range using multiobjective genetic algorithms,” in Soft Computing in Engineering Design and Manufacturing, 1997, ch. 5, pp. 231–240.
•  E. Zitzler and S. Künzli, “Indicator-based selection in multiobjective search,” in Proceedings of the International Conference on Parallel Problem Solving from Nature (PPSN), 2004, pp. 832–842.
•  N. Beume, B. Naujoks, and M. Emmerich, “SMS-EMOA: Multiobjective selection based on dominated hypervolume,” European Journal of Operational Research, vol. 181, no. 3, pp. 1653–1669, 2007.
•  M. Li, S. Yang, and X. Liu, “Diversity comparison of Pareto front approximations in many-objective optimization,” IEEE Transactions on Cybernetics, vol. 44, no. 12, pp. 2568–2584, 2014.
•  Q. Zhang and H. Li, “MOEA/D: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on Evolutionary Computation, vol. 11, no. 6, pp. 712–731, 2007.
•  M. Li, S. Yang, and X. Liu, “Pareto or non-Pareto: Bi-criterion evolution in multi-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 20, no. 5, pp. 645–665, 2016.
•  J. R. Schott, “Fault tolerant design using single and multicriteria genetic algorithm optimization,” Master’s thesis, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, 1995.
•  K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: Solving problems with box constraints,” IEEE Transactions on Evolutionary Computation, vol. 18, no. 4, pp. 577–601, 2014.
•  E. Zitzler, M. Laumanns, and L. Thiele, “SPEA2: Improving the strength Pareto evolutionary algorithm for multiobjective optimization,” in Evolutionary Methods for Design, Optimisation and Control, 2002, pp. 95–100.
•  H. Wang, L. Jiao, and X. Yao, “TwoArch2: An improved two-archive algorithm for many-objective optimization,” IEEE Transactions on Evolutionary Computation, vol. 19, no. 4, pp. 524–541, 2015.
•  M. Li, S. Yang, and X. Liu, “A test problem for visual investigation of high-dimensional multi-objective search,” in IEEE Congress on Evolutionary Computation (CEC), 2014, pp. 2140–2147.
•  M. Li, C. Grosan, S. Yang, X. Liu, and X. Yao, “Multi-line distance minimization: A visualized many-objective test problem suite,” IEEE Transactions on Evolutionary Computation, 2017, published online.
•  R. C. Purshouse and P. J. Fleming, “Conflict, harmony, and independence: Relationships in evolutionary multi-criterion optimisation,” in International Conference on Evolutionary Multi-Criterion Optimization, 2003, pp. 16–30.
• 

M. Li, S. Yang, and X. Liu, “Shift-based density estimation for Pareto-based algorithms in many-objective optimization,”

IEEE Transactions on Evolutionary Computation, vol. 18, no. 3, pp. 348–365, 2014.
•  K. Deb and D. K. Saxena, “Searching for Pareto-optimal solutions through dimensionality reduction for certain large-dimensional multi-objective optimization problems,” in IEEE Congress on Evolutionary Computation (CEC), 2006, pp. 3353–3360.
•  G. Ellis and A. Dix, “A taxonomy of clutter reduction for information visualisation,” IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 6, pp. 1216–1223, 2007.
•  L. F. Lu, M. L. Huang, and J. Zhang, “Two axes re-ordering methods in parallel coordinates plots,” Journal of Visual Languages & Computing, vol. 33, pp. 3–12, 2016.
•  A. R. de Freitas, P. J. Fleming, and F. G. Guimarães, “Aggregation Trees for visualization and dimension reduction in many-objective optimization,” Information Sciences, vol. 298, pp. 288–314, 2015.