In vivo and in silico experiments are standard tools for reducing lag between design and industrialization phases (Montomoli et al. 2015). These experiments make the most of cutting-edge instruments and High Performance Computing (HPC) resources to perform Uncertainty Quantification (UQ) (Sacks et al. 1989) studies dedicated for instance to parametric studies and design optimization. A system is hereafter described in terms of input and output spaces of potentially large dimension, related by a numerical solver for in silico experiments. Generally speaking, UQ stands in: (i) describing the uncertainties in the input space, (ii) propagating these uncertainties onto the output space where the Quantity of Interest (QoI) is defined, (iii)estimating and ranking the contribution of each source of uncertainty on the QoI. These contributions are estimated with sensitivity indices such as Sobol’ indices (Saltelli et al. 2007)
that are based on variance decomposition. The uncertainty in the input space can then be reduced using optimization or data assimilation algorithms(Asch et al. 2016) when observations are available in order to reduce the uncertainty in the output. The ultimate goal is to assess a risk, for instance of failure or of exceeding a threshold.
In spite of a large literature on UQ, the community has yet to propose efficient ways to visualize uncertainties. Indeed, to the authors’ knowledge, there is no chapter dedicated to visualization in UQ reference books (Saltelli et al. 2007; Sullivan 2015; Ghanem et al. 2017). This remains to be investigated, especially for Computational Fluid Dynamics (CFD) applications (Moreland 2016) that involve complex fields of large dimensions. Classical ways of visualizing standard statistics can lead to misinterpretation (Anscombe 1973). The challenge for state-of-the-art visualization solutions stands in the dimension of the data. Assuming that the dimension of the data is limited (a set of scalars), for instance when dealing with input data , canonical subplots of subspaces are adapted. Parallel coordinate plots (Inselberg 1985) or Kiviat plot (or spider plot) (Hackstadt et al. 1994) that are represented in LABEL:fig:sketch_Kiviat-parallel offer an interesting alternative and share the same idea of dedicating one input variable (noted ) per axis; Kiviat (right panel) plot being the equivalent to parallel coordinates (left panel) plot in polar coordinates.
When the dimension of the data increases, for instance when dealing with functional output fields discretized in both space and time on fine meshes, advanced strategies should be proposed. Different strategies are found in the literature to visualize statistics on the QoI (Potter et al. 2012; Brodlie et al. 2012; Bonneau et al. 2014). Beyond deterministic simulation, moving on to ensemble-based approaches, the dimension of the data further increases. A first approach is to look at each realization individually. For scalar QoI, a simple answer is found with the Hypothetical Outcome Plots (HOPs) technique (Ehlschlaeger et al. 1997) generalized by (Hullman et al. 2015) for a set of scalars as presented in Fig. 2. HOPs consists in animating a succession of possible outcomes sampled from the data. It was proven to enable the exploration and the understanding of the dataset general characteristic, even for people lacking statistical background (Belia et al. 2005) with the possibility to visually grasp correlations in the outputs.
Statistical moments and Probability Density Function (PDF) plot are common tool for visualizing uncertainties in the context of ensemble simulation as illustrated inFig. 3. They are useful for risk analysis as the probability of exceeding a threshold is directly observable. Figure 3(a) displays the PDF for a scalar QoI (noted ) where the mean, the mode and extreme probabilities can be observed. Figure 3(b) displays the principal mode of the PDF (solid black line) for a functional QoI discretized in the
direction. The PDF standard deviation (added/removed to/from the mean) is plotted in dashed lines. These curves are computed for eachindependently and they do not represent a possible solution of the numerical solver. The box plot in Figure 3(c) provides similar information while diminishing the false illusion of median/mean curves. It should be noted that users usually have a better understanding of frequency (Gigerenzer and Hoffrage 1995)
than of PDFs and that there is a general misinterpretation of confidence intervals(Belia et al. 2005).
. It allows to depict some statistics (for instance median or outliers) taking into account the functional QoI as a whole and working in a reduced space spanned by the most significant directions of the output space. Within this reduced space, metrics for functional outputs are computed such as the distance to the median (blue curve) so that abnormal or outlier outputs (red and green curves) can be detected and quantiles can be estimated (blue envelope). Each curve represents either a realization within the data set or an additional realization sampled in the reduced space; thus functional characteristics such as spatial or temporal correlation are preserved. From(Popelin and Iooss 2013; Ribés et al. 2015)
, the HDR method is more robust to outlier detection than other methods such as functional boxplot(Sun and Genton 2011).
This article proposes a solution to visualize high input and output dimensions. This solution relies on HDR metrics and a 3-dimensional version of Kiviat plot. For the 3D-Kiviat, each layer stands for a hypothetical outcome (a realization) coloured by a scalar value related to the QoI. This value is either the QoI at a given point and time or some distance computed with the HDR metrics. This solution can be augmented with data sonification to traduce the HDR metrics.
The paper is tailored as follows: Section 2 presents existing material and methods for our study. Section 2.1 presents the dataset used for illustration purpose and Section 2.2 presents the HDR metrics. Section 3 presents our innovative technique to visualize output uncertainties based on f-HOPS with sonification for HDR metrics. Section 4 extends this advanced solution to both input and output uncertainties visualization building on 3D-Kiviat representation augmented with HDR metrics and sonification. Conclusion and discussion are finally given in Section 5.
2 Materials and methods
Visualization solutions proposed in this paper are applied on two datasets described in Table 1 in order to state the applicability to various fields of applications. The first dataset has no input-output relation and only features a temporal output. The second dataset features an input-output relation with spatially varying output. The datasets are as follows:
The El Niño dataset is a well-known functional dataset (Hyndman and Shang 2009)
. It consists in a time series of monthly averaged Sea Surface Temperature (SST) in degrees Celsius spatially averaged over the Pacific Ocean region (0-10°S and 90-80°W) from January 1950 to December 2007. The QoI is a vector of size 12 and the data set gathers 58 realizations. Data originate from NOAA ERSSTv5’s database available at
The Hydrodynamics dataset gathers water levels (in m) computed with the 1-dimensional Shallow Water Equation MASCARET solver (opentelemac.org) for a 50 km reach of the Garonne river in South-West of France (Roy et al. 2017). Uncertain inputs relate to 4 scalars: the friction coefficients of the river bed defined over three homogeneous spatial areas, and the upstream boundary condition described by a constant scalar value for the inflow in stationary flow. The QoI is a vector of size 463 (number of computation nodes for the 1D mesh) and an ensemble of 200 realizations is considered here.
|Dataset||Scalar input||Functional output||Sample size|
2.2 Highest Density Region
The dataset output is considered as a matrix where each line corresponds to a realization. This matrix is decomposed by principal component analysis (PCA) (Anindya Chatterjee 2000). The modes are ordered by decreasing importance in terms of contribution to the variance and only a finite number of modes are kept. In this reduced space, the functional dataset of large dimensions is conveniently represented by a limited number of scalars mapped onto most significant directions that maximizes the variance of the QoI. Within this reduced space, the median realization corresponds to the HDR location. The distance to this point is computed in the modal space; the farther a point is from the HDR, the less probable is the realization.
A multivariate Kernel Density Estimation (KDE)(Wand and Jones 1995) technique is used to estimate the Probability Density Function (PDF) of this multivariate space. The density estimator is given by
with the bandwidth for the ith component and the kernel chosen as a modal probability density centred and symmetric function. is the Gaussian kernel and are optimized given the data. From this KDE, the HDR reads
with such that . The region has a probability of containing points that verify the highest density estimate. Thus the median curve is defined as the inverse transform—from the reduced space into the original space—of . Also 50% and 90% highest density regions are computed—with respectively and .
Fig. 5 illustrates the HDR boxplot for the El Niño dataset in the reduced space (left panel) when only two modes are retained ensuring that at least 80% of the QoI variance is conserved and in the output space (right) panel. Each realization is characterized with respect to the HDR metric. In the modal space, each dot represents a realization within the dataset and the contouring represents the 50% and 90% quantiles. In the QoI physical space, cyan curves represent the realizations from the dataset, the outliers are coloured-dashed curves, the thick black curve is the median and the grey shaded areas represent 50% and 90% quantiles envelopes. It should be noted that additional realizations with chosen characteristics on the outputs could be drawn by sampling the input for specific HDR criteria.
3 Uncertainty Visualization of Functional Output Data
3.1 Dynamic Visualization of Functional Outputs’ Statistics
The HOPs dynamic visualization can be applied to functional outputs (Kale et al. 2018) as in Fig. 6 animating successive realizations of the data (for the El Niño dataset), it is noted f-HOPs. When combined to HDR criteria, each realization is discriminated taking into account the functional characteristics of the output and outliers are easily detected. This solution is complementary to the classical PDF plot shown in Fig. 7 that displays the probability of monthly sea surface temperature. The median and standard deviation curves are shown but these statistics are computed point by point independently and the outliers cannot be represented. Moreover, f-HOPs does not allow to exhibit multiple modes statistics that are shown by the PDF at a given location. Figure 8 shows both the functional PDF and a PDF at a precise location of the Hydrodynamics dataset. It can be noted that there are multiple modes.
3.2 Sonification of Functional Outputs’ Statistics
When the number of realizations in the dataset is limited, a static or an animated visualization, using f-HOPs and HDR metrics, allow to depict the most significant characteristics of the ensemble members. When this number increases, the animation time increases and the analysis becomes harder. Sonification comes as an alternative for meaningful analysis of the dataset: it conveniently allows to draw the attention on specific realization discriminated by the HDR metric. We propose to compute the -norm between each realization and a reference realization within the reduced modal space. The reference realization is here chosen as the median and is mapped into a base sound. The frequency associated with each realization increases, and the sound becomes higher as the distance between each realization and the reference increases. Evidently, the reference can differ from the median. (Alexander et al. 2014) presents that sonification allows for well-informed understanding of a large dataset and that practitioners usually develop a physics-dependent vocabulary that is adapted to specific need as in (Hughes 2003) where sound is used to discriminate the types of gravitational waves. In this context, sonification serves data exploration. It can also be an alert system: f-HOPs augmented with HDR metric sonification allows to get a fairly monotonous sound for realizations that are close to the median, while outliers are clearly spotted as proposed in S1 for the El Niño dataset.
4 Uncertainty Visualization of Large Number of Input Variables
In the previous section, the focus was made on visualizing the QoI, not taking into account the relation between inputs and outputs. As visualizing the input alone, especially in case of small input dimension, is classical, the dual representation of input and output data remains challenging especially for large dimension functional outputs. The response surface plot is adapted when the input space is 2D or 3D. When the input space dimensions further increases, other solutions should be preferred such as parallel coordinates plot Inselberg (1985) or 3D-Kiviat proposed by (Hackstadt et al. 1994).
In this paper, we propose to adapt the 3D-Kiviat plot to the visualization of both input and QoI spaces. Each plane of the Kiviat represents a realization within the dataset with as many directions as the input dimensions as shown in Fig. 9 for the hydrodynamics dataset. The input variables here correspond to the friction coefficients (, , ) and the constant inflow ; the output variable is the water level at Marmande, it is colour-coded onto the Kiviat plane. For the 3D-Kiviat, planes are stacked into a 3D object with respect to the QoI (scalar or functional) related value that is colour-coded. It should be noted that each plane is filled with only one colour to preserve readability. The benefit of 3D-Kiviat stands in the choice of both the stacking and the colouring strategies as shown in Fig. 10 for the hydrodynamics dataset. Additionally, the 3D-Kiviat can be augmented with sound—as described in Section 3.2.
When representing functional output data, different stacking and colouring strategies allow to highlight different information in the dataset. Four choices of stacking and colouring are illustrated in Fig. 10; the stacking and colouring choices are indicated in the legend, they are achieved with respect to the QoI at a given location and time (noted ), with respect to the HDR metric (noted ) or with respect to one of the input variables. In Fig. 10(a), stacking is done with respect to the QoI at a given location and time while the colouring is done with respect to the difference to the median realization computed with the HDR metric. This allows to get a sense of the spatial PDF represented in Fig. 8 augmented with the input parameter mapping. Another possibility in Fig. 10(b) consists in stacking with respect to the HDR metric and colouring with respect to the QoI value. In Fig. 10(c), stacking is done with respect to one of the input variable (in the present case ) and the colouring is done with respect to the HDR metrics. Finally, Fig. 10(d) displays both stacking and colouring with respect to the QoI value. It should be noted that an animated version of Fig. 10(a) is provided in supplementary material Section 5 with sounding for the HDR metric that represents the distance to the median realization and allows for a convenient analysis of the data set especially when its dimension increases.
From Fig. 10(a, c, d), the impact of on the water level is easily readable; water level increases with . High water level values are also obtained for low values while other parameters seem to have no significant impact on the QoI. and have barely any impact on the QoI. The manipulation of the animated 3D-Kiviat (Section 5) is even more adapted to data analysis. The coloured HDR in Fig. 10(a, c) indicates how each realization differs from the median realization. It appears that stacking for colouring with respect to QoI or HDR serves different purposes. Ordering by QoI allows to discriminate which input lead to specific QoI value while ordering by HDR illustrated the dispersion of the dataset with respect to a reference realization. Sounding is a supplementary way to emphasis the information, especially for large datasets.
3D-Kiviat comes as a complementary tool to classical sensitivity analysis criteria such as Sobol’ indices (Saltelli et al. 2007). When computed on a large dataset (100 000 members) for the hydrodynamics example and averaged over space, these indices show that most of the variance of the water level is explained by with . A small part of the variance is explained by as and even smaller by and with and .
While the Sobol’ indices quantify the importance of each parameter on the QoI’s variance, they do not indicate the nature of these contributions. Indeed, it is stated that has a significant impact but from Fig. 10(a, c, d), we can also add that the contribution is monotonous. Additionally, the weak impact of and is confirmed by the lack of a pattern in the 3D-Kiviat along these axes. Finally, while the Sobol index is weak, the 3D-Kiviat indicates that small values lead to high-water level values.
It appears that 3D-Kiviat plot becomes difficult to manipulate when the dimension of the input parameter increases (). To overcome this caveat, we have designed a method to generate a surface mesh of the Kiviat plot as shown in Fig. 11. This allows the use of regular CAD viewers and thus facilitates the manipulation of the 3D object as well as the comprehension of the data structure. The construction is based on a vertex-vertex representation using quadrilateral elements. An animated version of Fig. 11 is provided in Section 5 with stacking with respect to the QoI value, colouring with respect to the HDR metric, then colouring with respect to the input parameters in Section 5.
The analysis from 3D-Kiviat is similar to that drawn from the parallel coordinates plot in Figure 12. The latter consists of parallel axes, with the number of input parameters for the hydrodynamics dataset. The last axis is dedicated to the output value, here water level at Marmande. Each grey line corresponds to one realization in the dataset; the red lines are discriminated for high-water level values resulting from high and small , independently of and . It should be noted that parallel coordinate plot is not adapted to simultaneous representation of QoI and HDR metrics contrary to 3D-Kiviat. Finally, the linear relationship between water level and that was clear on the 3D-Kiviat in not readable on the parallel coordinate plot that is more adapted to clustering with the possibility of advanced strategies proposed by (Raidou et al. 2016).
The specific case of a 2D input space is treated with a Tree plot solution where Kiviat coloured planes are replaced by coloured segments that are stacked and coloured regarding to QoI related value. The vertical stacking and colouring are achieved with respect to the QoI value. The HDR metric is encoded as an azimutal component: the angle is null if the realization corresponds to the median and the angle increases as the realization differs to the median. Figure 13 displays a tree plot for the hydrodynamics dataset where and are not accounted for (since they were previously shown to have barely any impact on the QoI) and the QoI is the water level at Marmande. Here again, stacking, colouring, angle and eventually sounding strategies can be adapted to convey information on a dataset and efficiently enhance meaningful information.
5 Conclusion and Discussion
This work proposes a systematic way to look at uncertainties when dealing with high-dimensional environment. As uncertainty quantification related analysis are moving toward high dimensionality, we believe that our tool could help these analyses.
In this work, a new method to visualize input and output uncertainties was presented. The concept of Hypothetical Outcomes Plot is taken a step further and applied to functional data (f-HOPs) considering the HDR boxplot information that estimates a distance to a reference realization (here the median). HDR metrics allow to reduce the output space to a limited number of principal components and compute, in this reduced space, a distance between dataset realizations taking into account temporal or spatially discretization. This metric allows to discriminate the realizations statistically instead of only regarding the QoI value at a specific location and time. f-HOPs provides an animated version of members within a dataset that can be augmented with data sonification that is efficient for outlier detection. Both input and output uncertainties are then visualized with a 3-dimensional version of the Kiviat where each realization is represented by a plane surface. Realizations are stacked along the vertical axis and colour-coded. The 3D-Kiviat can be augmented with data sonification. Stacking, colouring and sounding strategies are chosen with respect to the QoI value or the HDR metric to highlight some information within the dataset.
These visualization solutions were applied to two functional datasets: (i) El Niño and (ii) Hydrodynamics. The first one is a commonly used dataset that only provides temporal outputs, the second dataset presents an input-output relation with a large dimension output space and the input space larger than the classically represented 3D space. For both datasets, f-HOPs and 3D-Kiviat have proven to be efficient at representing the nature of the dataset and highlighting important impact factors. Moreover, visualization of the 4-dimensional input space was eased by the Kiviat solution that was conveniently adapted to CAD objects for larger dimensions.
Perspective for this work stands in the adaptation of stacking, colouring and sounding to specific purposes. For instance, for extreme events detection, the HDR metrics can be computed with respect to the mean or the median, but it could also be computed with respect to high quantiles. The reference can also represent additional information to the dataset. For instance, in the context of data assimilation, it can stand for observation data that are usually distributed in space and time, in order to compute the innovation vector or to achieve observation quality control steps.
Our strategy finally can be applied to operational context with ensemble integration. Straightforward applications are for instance real-time weather and flood forecasting as well as structure failure risk assessment and outliers detection are at stake. In this context, our strategy would also be applied to larger dimension input and output spaces. The stacking strategy allows to create a 3D object that can easily be manipulated, even printed for deep analysis. Sounding is also an efficient way to draw the attention to an operator while the ensemble forecast is issued in case threshold exceed occurs. Finally, the complete solution with stacking, colouring and sounding offers social and human perspectives for statistical analysis of datasets by people with disabilities for which written information is not handy.
Sounding Functional HOPs on the El Niño dataset. (movie file with audio)
Sounding 3-dimensional Kiviat on the Hydrodynamics dataset. (movie file with audio)
Mesh animation on the Hydrodynamics dataset, colouring with respect to HDR metric. (movie file)
Mesh animation on the Hydrodynamics dataset, colouring with respect to input parameters. (movie file)
open-source (MIT) Python script used to do HDR boxplot, Kiviat and data sonification. (python file)
- Alexander et al. (2014) Robert L Alexander, Sile O’Modhrain, D Aaron Roberts, Jason A Gilbert, and Thomas H Zurbuchen. The bird’s ear view of space physics: Audification as a tool for the spectral analysis of time series data. Journal of Geophysical Research: Space Physics, 119(7):5259–5271, jul 2014. doi: 10.1002/2014JA020025.
- Anindya Chatterjee (2000) Anindya Chatterjee. An introduction to the proper orthogonal decomposition. Current Science, 78(7), 2000.
- Anscombe (1973) F J Anscombe. Graphs in Statistical Analysis. The American Statistician, 27(1):17–21, 1973. doi: 10.2307/2682899.
- Asch et al. (2016) Mark Asch, Marc Bocquet, and Maëlle Nodet. Data assimilation: methods, algorithms, and applications. Fundamentals of Algorithms. SIAM, 2016. URL https://hal.inria.fr/hal-01402885.
Belia et al. (2005)
Sarah Belia, Fiona Fidler, Jennifer Williams, and Geoff Cumming.
Researchers Misunderstand Confidence Intervals and Standard Error Bars.Psychological Methods, 10(4):389–396, 2005. doi: 10.1037/1082-989X.10.4.389.
- Bonneau et al. (2014) Georges Pierre Bonneau, Hans Christian Hege, Chris R. Johnson, Manuel M. Oliveira, Kristin Potter, Penny Rheingans, and Thomas Schultz. Overview and state-of-the-art of uncertainty visualization. Mathematics and Visualization, 37:3–27, 2014. doi: 10.1007/978-1-4471-6497-5˙1.
Brodlie et al. (2012)
Ken Brodlie, Rodolfo Allendes Osorio, and Adriano Lopes.
A Review of Uncertainty in Data Visualization.Expanding the Frontiers of Visual Analytics and Visualization, pages 81–109, 2012. doi: 10.1007/978-1-4471-2804-5˙6.
- Ehlschlaeger et al. (1997) Charles R Ehlschlaeger, Ashton M Shortridge, and Michael F Goodchild. Visualizing spatial data uncertainty animation using. Computers & Geosciences, 23(4):387–395, 1997.
- Ghanem et al. (2017) Roger Ghanem, David Higdon, and Houman Owhadi, editors. Handbook of Uncertainty Quantification. 2017. ISBN 978-2-7598-1269-1.
- Gigerenzer and Hoffrage (1995) Gerd Gigerenzer and Ulrich Hoffrage. How to Improve Bayesian Reasoning Without Instruction : Frequency Formats. Psychological Review, 102(4):684–704, 1995. doi: 10.1037/0033-295X.102.4.684.
- Hackstadt et al. (1994) S T Hackstadt, A D Malony, and B Mohr. Scalable performance visualization for data-parallel programs. Scalable High-Performance Computing Conf., pages 342–349, 1994. doi: 10.1109/SHPCC.1994.296663.
- Hughes (2003) Scott A. Hughes. Listening to the universe with gravitational-wave astronomy. Annals of Physics, 303(1):142–178, 2003. doi: 10.1016/S0003-4916(02)00025-8.
- Hullman et al. (2015) Jessica Hullman, Paul Resnick, and Eytan Adar. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS ONE, pages 1–25, 2015. doi: 10.1371/journal.pone.0142444.
- Hyndman and Shang (2009) Rob J Hyndman and Han Lin Shang. Rainbow plots , bagplots and boxplots for functional data. Journal of Computational and Graphical Statistics, 19:29–45, 2009.
- Inselberg (1985) Alfred Inselberg. The Plane with Parallel Coordinates. The Visual Computer, 1(2):69–91, August 1985. doi: 10.1007/bf01898350.
- Kale et al. (2018) Alex Kale, Francis Nguyen, Matthew Kay, and Jessica Hullman. Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data. Transactions on Visualization and Computer Graphics, 2018.
- Montomoli et al. (2015) Francesco Montomoli, Mauro Carnevale, Antonio D’Ammaro, Michela Massini, and Simone Salvadori. Uncertainty Quantification in Computational Fluid Dynamics and Aircraft Engines, volume 92. 2015. ISBN 9783319146805. doi: 10.1007/978-3-319-00885-1.
- Moreland (2016) Kenneth Moreland. Why We Use Bad Color Maps and What You Can Do About It. In Proceedings of Human Vision and Electronic Imaging (HVEI), 2016. doi: 10.2352/ISSN.2470-1173.2016.16.HVEI-133.
- Popelin and Iooss (2013) Anne-laure Popelin and Bertrand Iooss. Visualization tools for uncertainty and sensitivity analyses on thermal-hydraulic transients. In SNA + MC 2013 - Joint International Conference on Supercomputing in Nuclear Applications + Monte Carlo, page 03403, jun 2013. ISBN 978-2-7598-1269-1. doi: 10.1051/snamc/201403403.
- Potter et al. (2012) Kristin Potter, Paul Rosen, and Chris R. Johnson. From quantification to visualization: A taxonomy of uncertainty visualization approaches. IFIP Advances in Information and Communication Technology, 377 AICT:226–247, 2012. doi: 10.1007/978-3-642-32677-6˙15.
- Raidou et al. (2016) Renata Georgia Raidou, Martin Eisemann, Marcel Breeuwer, Elmar Eisemann, and Anna Vilanova. Orientation-Enhanced Parallel Coordinate Plots. IEEE Transactions on Visualization and Computer Graphics, 22(1):589–598, 2016. doi: 10.1109/TVCG.2015.2467872.
- Ribés et al. (2015) Alejandro Ribés, Joachim Pouderoux, Anne-laure Popelin, and Bertrand Iooss. Understanding Ensembles by Joint Interactive Statistical Analysis of Curve Datasets and their Principal Component Analysis. 2015.
- Roy et al. (2017) Pamphile Tupui Roy, Nabil El Moçayd, Sophie Ricci, Jean-Christophe Jouhaud, Nicole Goutal, Matthias De Lozzo, and Mélanie C. Rochoux. Comparison of polynomial chaos and Gaussian process surrogates for uncertainty quantification and correlation estimation of spatially distributed open-channel steady flows. Stochastic Environmental Research and Risk Assessment, pages 1–29, oct 2017. doi: 10.1007/s00477-017-1470-4.
- Roy et al. (2018) Pamphile Tupui Roy, Sophie Ricci, Romain Dupuis, Robin Campet, Jean-Christophe Jouhaud, and Cyril Fournier. BATMAN: Statistical analysis for expensive computer codes made easy. The Journal of Open Source Software, 3(21):493, 2018. ISSN 2475-9066. doi: 10.21105/joss.00493. URL http://joss.theoj.org/papers/10.21105/joss.00493.
- Sacks et al. (1989) Jerome Sacks, Williams J. Welch, Toby J. Mitchell, and Henry P. Wynn. Design and Analysis of Computer Experiments. Statistical Science, 4(4):409–423, 1989. doi: 10.1214/ss/1177012413.
- Saltelli et al. (2007) Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. Global Sensitivity Analysis. The Primer. dec 2007. doi: 10.1002/9780470725184.
- Sullivan (2015) T.J. Sullivan. Introduction to Uncertainty Quantification, volume 63 of Texts in Applied Mathematics. 2015. doi: 10.1007/978-3-319-23395-6.
- Sun and Genton (2011) Ying Sun and Marc G. Genton. Functional Boxplots. Journal of Computational and Graphical Statistics, 20(2):316–334, 2011. doi: 10.1198/jcgs.2011.09224.
- Wand and Jones (1995) M. P. Wand and M. C. Jones. Kernel Smoothing. Boston, MA, 1995. doi: 10.1007/978-1-4899-4493-1.