Calibration refers to the problem of specifying the values of some features, known as calibration parameters, in a computational model, which are unobservable in the associated physical system (Kennedy and O’Hagan, 2001; Han et al., 2009a; Higdon et al., 2004a; Tuo and Wu, 2014). Here, a computational model refers to a set of mathematical formulations and assumptions implemented through computer codes, and is intended to behave similar to the physical system, i.e. they both generate the same response for a given input variable. If in the physical system, a calibration parameter assumes a range of values as a function of other variables in the system, we call this problem functional calibration. Functional calibration is motivated by examples in nano-manufacturing (Pourhabib et al., 2015), resistance spot-welding (Bayarri et al., 2007), and bioenergy production (Kumar et al., 2009). For example, in Poly-vinyl Alcohol (PVA)-treated buckypaper fabrication, we are interested in understanding the relation between the response, the tensile strength, and the input variable, the PVA amount. Here, the calibration parameter is the percentage of PVA absorbed, which may change in the physical system depending on the values of the input variable, the PVA amount (Pourhabib et al., 2015).
From a geometric perspective, functional calibration is a curve to surface matching problem. Here, all the possible values for the input variable and the response constitute a one-dimensional curve. Note that the reason the physical system is denoted by a curve in a two-dimensional space is we cannot measure or observe the calibration parameter, also referred to as the latent variable (Pourhabib et al., 2015), in the physical process. As such, what we actually observe only consists of input variables and the responses which form a curve in the input-response space. On the other hand, in the computational model, we can specify the values of both the input and the calibration parameter. Consequently, all the possible values of the variable, calibration parameter, and the computational model response together form a surface (see Figure 1). If the problem is point calibration, i.e., there exists one single value for the calibration parameter, then we need to match the curve from the physical system to the surface from the computational model, since the computational model is intended to represent the physical process.
We note that the physical curve we observe is in fact a projection of a curve in the three dimensional input-parameter-response space. The reason lies behind the nature of parameters in a physical process: for each input there exists an (unknown) value for the parameter, and these two features, i.e the input and parameter, determine one single response. However, since we do not observe the actual value of the parameter in the physical process, we only see a projected curve on input-response surface. Hence, this problem is to recover the original curve, or in other words, determine a non-isometric match of a curve to a surface. The non-isometry is due to the fact that the curve in the projected space has a different length than the original curve. Therefore, this is in principle different from isometric matching problems (Gruen and Akca, 2005; Bronstein et al., 2005; Baltsavias et al., 2008).
Furthermore, in functional calibration we do not observe a complete curve or surface, but a set of scattered data points obtained from the physical experiments or the computational model. The data points from the former lie on the projected curve we observe and the data from the latter lie on the input-parameter-response surface. Therefore, what we observe is incomplete data and we aim to match non-isometrically an incomplete curve to an incomplete surface, which is equivalent to solving the functional calibration problem.
The incompleteness of the data, that is scattered on a curve or a surface instead of a continuous function, can be addressed by using kernel interpolation, such as Gaussian processes (GPs)(Stein, 1999). In fact, to handle functional calibration, both parametric and non-parametric approaches (Pourhabib et al., 2015; Pourhabib et al., 2014)
use kernel interpolation. After interpolating the data points to get a continuous surface, the former assumes a parametric relationship between the inputs and calibration parameters and then estimates the unknown parameters of that function, whereas the latter, the non-parametric approach, formulates the problem using the concepts of reproducing kernel Hilbert spaces (RKHS) and then finds a solution using the representer’s theorem(Schölkopf et al., 2001).
However, using interpolation introduces a new source of error: assuming a GP, for example, relies on the joint Gaussian distribution which is not always the case in practice. Therefore, we argue if we operate on the very data points we will obtain a better matching in this case. In addition, the existing methods appeal to iterative numerical techniques for optimization which are prone to get trapped in local optima. The interaction between these two issues, interpolation and iterative optimization, further exacerbates the quality of the optimal solution, as the methods converge to a local optimum of the model based on the interpolated surface.
In this paper, we propose a graph-theoretic approach for non-isometric curve to surface matching from incomplete data for functional calibration. We reformulate the functional calibration problem as a shortest path problem on a directed acyclic graph (DAG) which can be solved efficiently with a fast polynomial-time algorithm. The shortest path found identifies anchor points from the computational model points that balance two basic criteria: First, the anchors should offer the best match to the response of the physical system, and second, they should facilitate fitting of a smooth function (not exhibit erratic fluctuations in values). The anchor points are then used to fit a response function that can then be used to make predictions on the physical process. Applying the overall framework developed in this paper on both synthetic and real data, we demonstrate that the new approach outperforms the alternative approaches in terms of prediction accuracy.
The rest of this paper is organized as follows: Section 2 formally defines the problem and reviews the relevant literature. Section 3 outlines our proposed approach to solve the non-isometric curve to surface matching. Section 4 presents the numerical results and makes suggestions regarding parameter selection. Section 5 concludes the paper.
2 Non-isometric matching
2.0.1 Problem Definition
We consider a physical system that operates according to a set of (possibly unknown) physical laws. In this system, we assume there is a functional relationship between a group of features and an output. The physical system has an input variable denoted by
. Input variables represent those values that can be observed or specified. As such, the vector of input variables are also called control variables. We also assume that once in the physical system the control variables are set (either observed or specified), the physical process generates a real valuedresponse. However, the response is not merely a function of those control variables, but a function of both the control variables and parameters . The latter denotes those features of the physical systems that are either hard to measure or control. Hence, to express this functional relationship we denote the response as . Likewise, we denote the output of the computational model by , although here, is also an input argument. An example of such a complex system is PVA-treated buckypaper fabrication, where we are interested in finding the relation between the input variable PVA amount and the response, namely the buckypaper tensile strength. Here, the percentage of PVA absorbed is a calibration parameter, since it cannot be measured in the physical system.
Note that, assuming our computational model is constructed according to the laws governing the physical system, in a computational model the same concepts hold: we have a “system” whose structure is determined by the interactions between some input/control variables and parameters. However, since in a computer code we are not restricted to measuring any physical features, we can set the values of both and arbitrarily.
Of course, the goal is to design a high-fidelity computational model that operates similar to the physical system. To this end, we need to obtain the physical responses at a set of physical inputs , and the computational model responses at the model inputs , where and , for and . In general, due to cost considerations, , i.e., obtaining data points from the computational model is generally cheaper than getting data from the physical experiment. As such, we also assume that for each there exists at least an such that , i.e., for any physical input we have at least one model input where the control variable in the computational model is the same as that in the physical system. The goal in calibration is to adjust the parameter , such that the computational model represents the physical system in the sense that the computational model can predict the physical response at any input location .
We call any method seeking a single value that matches the computational model with the physical system, point calibration (this class of methods is simply called calibration in the literature; see (Tuo and Wu, 2014; Bayarri et al., 2007; Joseph and Melkote, 2009; Higdon et al., 2004b; Han et al., 2009b; Xiong et al., 2009)). On the other hand, functional calibration refers to finding a function , where , such that the output of the computational model matches that of the physical system for any given , i.e. is a solution to
where is a space of functions and
is a loss function that measures the discrepancy between the responses of the physical system and the computational model. What we call functional calibration is also referred to as local calibration(Pourhabib et al., 2014) or latent variable estimation (Pourhabib et al., 2015).
Functional calibration has an interesting geometric interpretation. Note that as a function of mere constitutes a one-dimensional curve. On the other hand, as a function of both and is a two-dimensional surface. So, the optimization problem (1) is matching a one-dimensional curve to a two-dimensional surface. However, this is different from traditional matching problems (Gruen and Akca, 2005), since the length of the one-dimensional curve is not preserved. Specifically, if represents the true physical curve, what we observe is its projection , which is a one dimensional curve. On the other hand, the two-dimensional surface is set of all points . Now, we are trying to find such that the curve lies on the surface ; however, the length of the curve is larger than that of , and consequently the matching procedure is non-isometric. As such, we note that functional calibration for the case where and is non-isometric curve to surface matching.
In practice we do not have the entire curve or surface, but scattered data points from those sets. Recall that we are provided with the physical inputs and model inputs from the computational model. As such, we have , and sampled from the projected curve and the surface, respectively, where for , and for . Under this setting, the problem is then non-isometric matching of an incomplete curve to an incomplete surface, which we call non-isometric matching with incomplete data. Once the function is estimated, we can predict the physical response at any point by . See Figure 1 for an illustration.
2.0.2 Related Work
The characterization of function determines the solution approach to problem (1). One way is to characterize as an element of a reproducing kernel Hilbert space (RKHS) (Tuo and Wu, 2014). Assume denotes a symmetric positive semidefinite kernel, i.e.,
for any where is a measure (Rasmussen and Williams, 2006), which can define a space of smooth functions: For a given , the function is a real-valued function. Then, define the space of all functions constructed as a linear combination of for , as:
We can show that the closure of with respect to the norm induced by the inner product
is an RKHS (Wendland, 2005; Fasshauer, 2011). Then, non-parametric functional calibration (NFC) (Pourhabib et al., 2014), given a kernel, seeks a function in that RKHS as a solution to a variation of problem (1). Specifically, NFC solves
where , and , is a regularization parameter. The generalized representer’s theorem (Schölkopf et al., 2001) guarantees that any local optimum , i.e. a linear combination of the kernel evaluations at . Therefore, we need to find such that
which is a non-convex optimization problem in general, due to the fact that is an argument of the non-convex function .
Note that in optimization problem (3), we have which means we evaluate the computational model at . This poses no issue since we assumed for each there exists at least an such that . However, since we solve (3) using iterative numerical techniques, such as trust region methods (Byrd et al., 2000; Conn et al., 2000), or steepest descent (Luenberger and Ye, 2008), we may have to evaluate at points other than those in . Therefore, we need to create a surrogate model that interpolates the computational model. Specifically, let denote a real valued function defined on , where , for every , i.e. interpolates . Furthermore, if at any point , provides a “good guess” of what would be, we call a surrogate model for . A popular choice for the surrogate model is based on GP regression (Stein, 1999; Rasmussen and Williams, 2006).
An alternative to using a non-parametric form for , namely a function in the RKHS, is to assume a parametric form. For example, guided by the physics of the buckypaper fabrication process (Pourhabib et al., 2015), we can assume . Then, the functional calibration problem seeks such that
where we use again as a surrogate model as solving (4) requires appealing to iterative numerical techniques that need to evaluate the computational model at points not necessarily in . We call this approach parametric functional calibration (PFC) (which is also referred to as latent variable estimation in (Pourhabib et al., 2015)).
PFC is well suited for the buckypaper fabrication problem, however, for most applications we cannot assume a parametric form for the calibration function a priori. In addition, using a surrogate model introduces a new source of error, since we are interpolating the computational model surface. We cannot overcome the issue of using a surrogate model in PFC, or even NFC, since they are both solved using iterative numerical techniques that require evaluating the objective function at arbitrary points on its domain. Note that once we use a surrogate model in the optimization procedure, the point identified as optimal may be far way from the actual optimal point. In the following section, we propose an alternative framework in which we employ the computational model data points directly to determine the actual physical curve.
3 Graph Shortest Path for Non-isometric Matching
The intuition that the functional calibration problem is non-isometric curve to surface matching, and that we are provided with scatter data points from the curve and the surface, motivates us to view the problem through a discrete combinatorial lens. In this approach, we model functional calibration as a combinatorial optimization problem on a graph.
Unlike NFC and PFC discussed in Section 2.0.2, we do not identify a class of functions a priori. Instead we seek to identify anchor points, among computational model points, which, intuitively speaking, are “close” to the points on the true physical curve. In other words, the anchor points are positioned such that the true physical curve passes through neighborhoods of those points. To find these anchor points, we require two properties to hold: (i) the computational model response should be close to the physical response for a given input ; and (ii) the parameter values for two consecutive anchor points should be close to each other. The former ensures the computational model identifies the points that have the same response as that of the physical system, and the latter ensures the smoothness of the estimated physical curve. Next, we show how we translate this idea and the properties into a rigorous graph-theoretic framework.
Without loss of generality, we assume that both the physical inputs and model inputs are strictly ordered such that , for , and , for . Then, we construct a directed, edge-weighted graph, where , and the vertices in correspond to the model inputs in .
We partition the into clusters as follows: Consider any vertex corresponding to . Then, is assigned to a unique cluster for some as follows:
Note that, if the minimum in (5) is achieved by more than one , then the tie can be broken arbitrarily. As a consequence, each cluster is in 1-to-1 correspondence with the physical input . We can now describe the set of directed edges as follows:
The first term corresponds to all the directed edges that go from vertices in to vertices in . The second term corresponds to the directed edges from the artificial vertex 0 to each vertex in . The third term corresponds to directed edges from vertices in to the artificial vertex .
The final critical step is assigning weights to each edge . Consider two consecutive clusters and for any and let and . Define as follows.
where is a scaling parameter. The weights and are identically zero. Notice that the edge-weight for any edge between two consecutive clusters and consists of two parts: the first representing the difference between the model response and physical response; the second part represents the difference between the calibration parameters of and .
On this graph , we intend to solve the shortest (simple, directed) path problem from origin vertex to destination vertex . Every path from 0 to in has exactly arcs, by construction. Therefore, the internal vertices on the shortest path that is found will serve as the anchor points for fitting the function. See Figure 2 for illustration.
Observe that, as described here is a directed acyclic graph (DAG). Furthermore, we claim that is a topological ordering of vertices of . To establish this claim it suffices to show that if , then . Note that as we have assumed the inputs to be strictly ordered. Suppose and where . Hence, , and . Since, are points on a real line, we must have . Therefore, we can solve the shortest path problem in complexity using the reaching algorithm for DAGs (Lawler, 1976) (see Algorithm 1).
Suppose is the shortest path identified. The points serve as the anchor points for fitting the function. Note that we are only interested in identifying these “optimal” anchor points that provide us with the information to estimate the “best” function (within this discrete optimization framework).
Given the anchor points, one approach is to fit a piecewise-linear function with the anchor points as break-points, or identify linear pieces via regression within clusters, with the additional constraint that the linear pieces coincide at the boundary resulting in a continuous piecewise-linear function. Alternately, we can use GP regression to interpolate between these points. Based on our experiments, the latter, i.e., GP regression, provided a better solution in terms of predictive power. Specifically, we fit a GP on the anchor points , where are the index set and are noise-contaminated output of the GP , i.e. , for , where and .
We call this approach graph non-isometric matching (GNM), which characterizes in optimization (1) as the space of smooth GP posterior functions. The proximity of the anchor points to the physical responses is emphasized by our construction (clustering and edge-weights), enables a good match between the physical system and the computational model.
This section presents the numerical results for the proposed method, GNM, and compares its performance with two competing methods, PFC and NFC. We use one real and two synthetic datasets. The measure of performance is prediction accuracy; after using a training dataset we select a set of test data points, and then for each method we find the predicted response values . We compare the predicted values with the actual physical responses at those points, i.e. . Then, we compute the root mean squared error (RMSE)
for each method. A method with a smaller RMSE is deemed superior.
To create training and test cases we follow two approaches: we can select the test data set to be the set of inputs in the training data set, i.e. . The second approach is to use 4-fold cross validation (Hastie et al., 2001). We call the error found through the latter the test error and the former the training error.
For synthetic datasets, since we know the actual functional relationship between the calibration parameter and the input variable, we can also calculate the RMSE for the calibration parameter prediction. Specifically,
where is the estimated parameter at the input , i.e., , where is the estimated calibration function, and is the actual parameter.
4.0.1 Synthetic Data
We use two synthetic datasets, initially designed to test NFC (Pourhabib et al., 2014); however, we use different ranges for the input variables and slightly modify the way the input locations are selected as explained. First, we consider a physical system with a response function and an associated computational model . We note that when the computational model matches the physical response. As such, the true functional relationship is given by . We name this dataset SD1. The second synthetic dataset we consider displays a more complicated physical response and a nonlinear relationship between the calibration parameter and control variables. Specifically, the physical system follows and the associated computational model surface . Note that if the parameter the computational model matches the physical system. We name this dataset SD2. We use two-fold cross validation to select the regularization parameter , which suggests for SD1 and for SD2. Figure 1 illustrates the problem for SD2.
To calculate the training error, we use physical data points and data points for the computational model. To calculate the test error, we use physical data points and points from the computational model. For SD1 the physical data points are uniformly sampled from and the computational model data points are sampled from . For SD2 the physical data points are sampled from , and the computational model data points are sampled from .
Table 1 shows the results for both SD1 and SD2. For each training or test case, we present the results for response prediction, measured by RMSE, and parameter prediction, measured by RMSE
. For test cases, the numbers in parentheses are the standard deviations of the four folds. The table shows that GNM outperforms NFC and PFC for all the comparison cases. In the following, we analyze some of the results in more detail.
Comparing training cases of SD1 with SD2, we observe that the performance of PFC deteriorates on the dataset with a quadratic relationship between the control variables and the parameters, whereas for GNM and NFC the nonlinearity of function does not affect the performance since neither fixes a parametric functional from a priori. An interesting observation for the test errors in SD1 is while NFC has a relatively small RMSE, its RMSE is significantly worse. A closer look at the results revealed that the large value for RMSE can be attributed to two reasons. First, for the test cases each method is provided with a smaller number of data points. For NFC, this resulted in the algorithm finding a local optimum in the region which has a similar response surface as the training region, but different values of the parameters. This occurs due to the fact that NFC uses function approximation for the computational model, thereby it assumes the function following similar trends over a large region. Note that this problem does not occur for all the possible ranges for the inputs, like the ranges used in (Pourhabib et al., 2014) for building synthetic data. In fact, we realized this issue while trying different ranges for the data, and then used the aforementioned range in this paper to demonstrate this point. This can be a major drawback for some applications in which not only a good prediction for the response is desired, but also the relationship between the control variables and the parameters is sought (Pourhabib et al., 2015).
In summary, the significantly lower RMSE and RMSE for GNM, compared to NFC and PFC can be explained by noting that GNM does not model the computational model as a continuous surface. In addition, the way the graph is constructed adequately describes the non-isometric matching nature of this problem.
4.0.2 Real Data
We use a real dataset from fabrication of buckypaper which is a nano-manufactured product composed of thin sheets of carbon-nano tubes (Tsai et al., 2011). An approach to enhance the mechanical properties of buckypaper involves adding PVA to the fabrication process. A computational model is then used to model the relationship between the output, measured by Young’s modulus of the buckypaper, and the input, the PVA amount (Wang, 2013). Here, the calibration parameter is the percentage of the PVA absorbed in the process. Based on the understanding of the physics of the problem, this parameter is related, through a functional relationship, to the input variable, the PVA amount (Pourhabib et al., 2014). As such, this problem falls in the category of functional calibration or, based on the framework we proposed in this paper, non-isometric matching.
physical data points, where the input variables are uniformly distributed betweenand . The number of computational model data points is . Since we do not know the true functional relationship between the parameter and the input, we can only evaluate the methods by calculating their response prediction. Table 2 shows the results using both training error and 4-fold cross validation. For this dataset, NFC provides the best result for the training error. For the test case, GNM provides the best result. This demonstrates when we have a smaller number of training physical data points, the graph-theoretic approach is more reliable than both NFC and PFC.
4.1 Selection of regularization parameter
Here we discuss the impact of the numerical value of the regularization parameter in equation (7). If the weight for the edges in the graph is solely decided by the difference in the responses of the physical data points and the simulated data points in their neighborhood. This may result in a large difference between the second values of the parameters of the two consecutive points selected in model (7). On the other hand, if is a very large number, the selected computational model points, which are used to estimate the true physical curve, may be very close to each other and the estimated curve may deviate from the actual physical curve.
In our analysis we use 2-fold cross validation to select the parameter . Based on our analysis, this approach identifies a good value for the regularization parameter. To test this, Figure 3 displays the test error for the two synthetic datasets as a function of . The test error is calculated by randomly partitioning each dataset into of training points and for testing. The figure suggests that the value of the error initially reduces as a function of then it sharply increases. For SD2, due to shape of the computational model surface, even a very small number close to zero provides a small prediction error. The values for we obtained in Section 4.0.1 using cross validation error, which were and for SD1 and SD2, respectively, are very close to the optimal values in this figure.
In this paper, we developed a novel graph-theoretic approach for non-isometric curve to surface matching from incomplete data for functional calibration. The motivating example is the PVA-treated buckypaper fabrication process in which the incomplete curve is the set of physical data points and the incomplete surface is the set of computational model data points. In this framework, the functional calibration problem was recast as a shortest path problem on a directed acyclic graph. The solution to the shortest path problem is used to identify anchor points that are then used to fit a response function. On both synthetic and real data, we demonstrated that our new approach outperforms competing approaches in terms of prediction accuracy.
An interesting question, which is of importance in many manufacturing problems, is how to generalize GNM to match low dimensional surfaces to higher dimensional surfaces. This problem arises when the physical process has more than one input, such as the welding example discussed in (Bayarri et al., 2007). A major challenge in this case is that the points do not poses an inherent ordering like in the one-dimensional case. Therefore this generalization is not a straightforward task and will constitute our future research.
Author Balasundaram would like to acknowledge the support of the Air Force Office of Scientific Research grant FA9550-12-1-0103 and the National Science Foundation grant CMMI-1404971. This work was completed utilizing the High Performance Computing Center facilities of Oklahoma State University at Stillwater. The authors also acknowledge Dr. Chuck Zhang from Georgia Institute of Technology for providing the authors with the buckypaper fabrication data.
- Baltsavias et al. (2008) Baltsavias, E., A. Gruen, H. Eisenbeiss, L. Zhang, and L. Waser (2008). High-quality image matching and automated generation of 3d tree models. International Journal of Remote Sensing 29(5), 1243–1259.
- Bayarri et al. (2007) Bayarri, M. J., J. O. Berger, J. Cafeo, G. Garcia-Donato, F. Liu, J. Palomo, R. J. Parthasarathy, R. Paulo, J. Sacks, and D. Walsh (2007). Computer model validation with functional output. The Annals of Statistics 35(5), 1874–1906.
- Bayarri et al. (2007) Bayarri, M. J., J. O. Berger, R. Paulo, J. Sacks, J. A. Cafeo, J. Cavendish, C. H. Lin, and J. Tu (2007). A framework for validation of computer models. Technometrics 49(2), 138–154.
et al. (2005)
Bronstein, A. M., M. M. Bronstein, and R. Kimmel (2005).
Three-dimensional face recognition.
International Journal of Computer Vision64(1), 5–30.
- Byrd et al. (2000) Byrd, R. H., J. C. Gilbert, and J. Nocedal (2000). A trust region method based on interior point techniques for nonlinear programming. Mathematical Programming 89(1), 149–185.
- Conn et al. (2000) Conn, A. R., N. I. M. Gould, and P. L. Toint (2000). Trust-Region Methods. SIAM.
- Fasshauer (2011) Fasshauer, G. E. (2011). Positive definite kernels: Past, present and future. Dolomite Research Notes on Approximation 4, 21–63.
- Gruen and Akca (2005) Gruen, A. and D. Akca (2005). Least squares 3d surface and curve matching. ISPRS Journal of Photogrammetry and Remote Sensing 59(3), 151–174.
- Han et al. (2009a) Han, G., T. J. Santner, and J. J. Rawlinson (2009a). Simultaneous determination of tuning and calibration parameters for computer experiments. Technometrics 51(4), 464–474.
- Han et al. (2009b) Han, G., T. J. Santner, and J. J. Rawlinson (2009b). Simultaneous determination of tuning and calibration parameters for computer experiments. Technometrics 51(4), 464–474.
- Hastie et al. (2001) Hastie, T., R. Tibshirani, and J. Friedman (2001). Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer.
- Higdon et al. (2004a) Higdon, D., M. Kennedy, J. C. Cavendish, J. A. Cafeo, and R. D. Ryne (2004a). Combining field data and computer simulations for calibration and prediction. SIAM Journal of Scientific Computing 26, 448–466.
- Higdon et al. (2004b) Higdon, D., M. Kennedy, J. C. Cavendish, J. A. Cafeo, and R. D. Ryne (2004b). Combining field data and computer simulations for calibration and prediction. SIAM Journal on Scientific Computing 26(2), 448–466.
- Joseph and Melkote (2009) Joseph, V. R. and S. N. Melkote (2009). Statistical adjustments to engineering models. Journal of Quality Technology 41(4), 362–375.
- Kennedy and O’Hagan (2001) Kennedy, M. C. and A. O’Hagan (2001). Bayesian calibration of computer models (with discussion). Journal of the Royal Statistical Society: Series B 63(3), 425–464.
- Kumar et al. (2009) Kumar, A., H. Noureddini, Y. Demirel, D. Jones, M. Hanna, et al. (2009). Simulation of corn stover and distillers grains gasification with aspen plus. Transactions of the ASABE 52(6), 1989–1995.
- Lawler (1976) Lawler, E. (1976). Combinatorial Optimization: Networks and Matroids. New York: Holt, Rinehart, and Winston.
- Luenberger and Ye (2008) Luenberger, D. G. and Y. Ye (2008). Linear and nonlinear programming, Volume 116. Springer Science & Business Media.
- Pourhabib et al. (2015) Pourhabib, A., J. Z. Huang, K. Wang, C. Zhang, B. Wang, and Y. Ding (2015). Modulus prediction of buckypaper based on multi-fidelity analysis involving latent variables. IIE Transactions 47(2), 141–152.
- Pourhabib et al. (2014) Pourhabib, A., R. Tuo, J. Z. Huang, and Y. Ding (2014). Local calibration of computer models. Manuscript.
Rasmussen, C. E. and C. K. I. Williams (2006).
Gaussian Processes for Machine Learning. The MIT Press.
- Schölkopf et al. (2001) Schölkopf, B., R. Herbrich, and A. J. Smola (2001). A generalized representer theorem. In Computational learning theory, pp. 416–426. Springer.
- Stein (1999) Stein, M. L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer Verlag.
- Tsai et al. (2011) Tsai, C., C. Zhang, J. A. David, B. Wang, and R. Liang (2011). Elastic property prediction of single-walled carbon nanotube buckypaper/polymer nanocomposites: stochastic bulk response modeling. Journal of Nanosience and Nanotechnology 11(3), 2132–2141.
- Tuo and Wu (2014) Tuo, R. and C. F. J. Wu (2014). A theoretical framework for calibration in computer models: parametrization, estimation and convergence properties. Technical report, Chinese Academy of Sciences and Georgia Institute of Technology.
- Wang (2013) Wang, K. (2013). Statistics-enhanced multistage process models for integrated design and manufacturing of poly(vinyl alcohol) treated buckypaper. Ph. D. thesis, Florida State University, Tallahassee, FL.
- Wendland (2005) Wendland, H. (2005). Scattered Data Approximation. Cambridge University Press.
- Xiong et al. (2009) Xiong, Y., W. Chen, K.-L. Tsui, and D. W. Apley (2009, March). A better understanding of model updating strategies in validating engineering models. Computer Methods in Applied Mechanics and Engineering 198(15-16), 1327–1337.