Introduction
Agentbased models (ABMs) can be a useful tool for modeling and understanding how macroscale/aggregate features of complex systems emerge from microscale/individual decisions, interactions, and feedbacks (“generative” social science[1]). As a result, they have found use in many application areas, including land use change[2, 3, 4, 5, 6, 7, 8], ecology[9, 10, 11], flood risk[12, 13, 14, 15, 16], and climate change adaptation[17, 18, 19, 20, 21].
Models can be designed to address different questions about the modeled system, including prediction, explanation, and demonstration[22]. Marks[23] proposed a classification of simulation models as demonstrative or descriptive based on the model’s purpose. Demonstrative ABMs are used to illustrate that patterns of interest can be produced through locallevel rules and interactions. Descriptive ABMs are intended to reproduce observed phenomena for the purpose of explanation, prediction, or both. The descriptive model category includes both simpler, “strategic” models, intended and more complex, “tactical” models[24]. Early ABMs, such as the pioneering work on segregation[25], were primarily demonstrative[26, 23]. Over time, there has been an increase in descriptive models[26], the most famous of which is the Artificial Anasazi Model[27].
Both demonstrative and descriptive models require tests to ensure that the model works as intended (sometimes referred to as verification[28])Descriptive models also benefit from a comparison of model output against observations[23]. Careful validation demonstrates that the model reproduces measured data, though this is not the same thing as demonstrating that the model is a reproduction of system dynamics[28], as all models are approximations of real processes[29].
This observation that models are only capable of approximating, rather than reproducing real system dynamics, shows the importance of descriptive model calibration: the process of selecting model structures and parameter values. One common approach to calibration is to tune model parameters until model outputs are close to the empirical data[6, 30, 31], but these procedures can lead to overfitting the model to the calibration data due to neglecting the conditional and stochastic aspects of datageneration and observation[4]
. To avoid overfitting and account for the stochastic elements of a model, another approach is to choose a model structure and parameter values which are most probable given the observations and prior information about system dynamics
[32, 33].ABM calibration can be complicated by pathdependence and nonlinearities resulting from feedbacks. However, whether descriptive ABMs are intended to be used for explanation or prediction, these features suggest a need for quantification of model and parametric uncertainty, as observed patterns may be contingent on stochastic forcings or particular initial conditions. In this study, we focus on the question of how much data is required to probabilistically calibrate agentbased models. Here we focus on the overarching question: How much data is required to probabilistically calibrate agentbased models? We use a Bayesian approach to uncertainty quantification, based on the Bayesian interpretation of parameter values as random variables.
We use a Markov Chain Monte Carlo (MCMC) calibration method,, based on the MetropolisHastings algorithm
[34]. MCMC is an extremely general method for sampling from the posterior distribution. MCMC has been used for calibrating ABMs[35], and is the method we use to avoid approximation effects. However, MCMC may be computationally intractable for complex models featuring long runtimes or highdimensional parameter spaces. An additional complication is the need to specify a statistical likelihood function, which may be difficult for particular applications.The MetropolisHastings method has the ability to produce high fidelity approximations to the full joint probability density function of the model parameters
[36]. In general, there is a tradeoff between computational speed and accuracy of the resulting parameter distributions. Some alternative approaches to statistical calibration of ABMs, which are aimed at reducing computational requirements or likelihood specification, include statistical emulation[37, 38], particle filtering[39], and approximate Bayesian computation[40, 41, 31, 11]. While these methods reduce the computational burden, they come at a cost of potentially severe statistical approximations that can influence the parameter estimates
[42, 43, 44, 45].We address three specific questions. (i) How much data is required to statistically calibrate an ABM? The complexity of agent decision rules (in the sense of the number of parameters) and agentagent and agentenvironment interactions and feedbacks (in the sense of emergence) can reduce the ability to constrain model parameters or test hypotheses. (ii) Can we distinguish between models with varying levels of complexity, either in terms of highdimensional decision rules or the types of agent interactions with each other and the environment? (iii) How are calibration and hypothesistesting affected by the use of spatiallyaggregated data (as opposed to observations of individual agents), which may be all that are available due to datacollecting limitations or considerations of anonymity?
For a concrete example, we focus on the particular problem of modeling housing abandonment under flood risks, following the insightful work of Tonn & Guikema[16]. Housing abandonment poses potentially severe economic problems for settlements along rivers and coastlines[46]. Residents who haven’t experienced flooding themselves may abandon their homes if their neighbors do due to depreciating values or anticipation of future flooding. An associated ABM, and two nested submodels with fewer interactions and feedbacks, are illustrated by the influence diagrams in Figure 1.
We focus on the simpler submodels to address the question of calibrating relatively simple ABMs. Using the colors in Figure 1, these are the “no interactions” model, in black, and the “spatial interactions” model, in black and blue. In both cases, agents decide to vacate their homes using a probabilistic decision process (logistic regression), as opposed to maximizing utility or using heuristics (which are more common in ABMs in certain application areas, such as land use
[47]. Once a house is abandoned, there is a chance that it is occupied by a new agent in a subsequent year.We use these models in a perfect model experiment (see, for example, Olson et al [48] or Reed & Kollat[49], so that the datagenerating process and parameters are known. The pseudoobservations are generated using the spatialinteractions model for an artificial riparian settlement and realizations of annual flood height maxima from a generalized extreme value distribution. The parcel return periods and river heights are shown in Figure 2. Details of the datagenerating process are provided in the Methods section. The additional dynamic mechanism resulting from spatial interactions leads to increased probabilities of parcel abandonment for all return periods across realizations of the stochastic process, even for parcels that are far from floods (Figure 3).
Results
Calibration
The structure of the data (individualparcel versus spatiallyaggregated) strongly influences the final shape of the posterior distribution, both due to the number of data points and the different likelihood function specifications. Figure 4 shows the result of updating the prior distributions (specified in Table 1) with 50 years of pseudoobservations of 100 parcels. For certain key parameters (such as the logistic regression coefficient for the local flooding frequency), aggregated data (the total number of abandoned parcels at each time) leaves the posterior close to the prior (Figure 4 b). For individual parcel data, while the marginal posterior is sharpened much further (Figure 4 a).
While it appears from Figure 4 a that the original decision rules are not fully recovered (looking at the posterior density at the datagenerating value), it is important to keep in mind the influence of stochasticity in the realized data. Running the same model with the same parameters can yield model output with very different dynamics due to stochastic forcings, particularly in the presence of high levels of path dependence and positive feedbacks (see Figure 5). Between the strong influence of the stochastic elements in the model and the relative lack of sensitivity of the logistic regression to parameter values close to the datagenerating value, it is not necessarily surprising that the datagenerating value is assigned a relatively low density.
The full posterior parameter estimate illustrates one limitation of more deterministic approaches to calibration, particularly those which emphasize qualitative parameter selection, as many parameters are highly correlated. For example, the two logistic regression coefficients for flood frequency and proportion of neighboring abandoned parcels have a correlation coefficient of 0.66: a lower sensitivity to experienced floods can be offset by an increased sensitivity to neighbor behavior. Another example is the high positive correlation between the probability of a vacant lot being reoccupied and both the logistic regression intercept term and the coefficient for neighboring parcels (r=0.73 in both cases). Similar interactions would be missed by a deterministic calibration combined with oneatatime sensitivity analysis[50].
To validate the calibrated model, we look at the hindcasting ability of the posterior predictive distribution (shown in Figure
6). While the threeparameter nointeractions model is well constrained by smaller data sets, the lack of fit of the posterior predictive distribution compared to the pseudoobservations for increased amounts of data reveals the missing abandonment dynamic mechanism. Without spatial interactions, the nointeractions model calibration results in a higher sensitivity to experienced flooding to account for the data, which results in an overestimate of the number of abandoned parcels in later years. Meanwhile, the spatialinteractions model, which has one additional parameter, requires more data to constrain the model (25 observed parcels is insufficient with up to 50 years of data), but, once constrained, fits the pseudoobservations better than the nointeractions model. In general, having a larger spatial domain/numbers of agents facilitates calibration more than having a longer data record.Model Selection
More complex ABMs can be thought of as being constructed by adding new interactions and feedbacks to simpler ABMs, as illustrated in Figure 1. This allows us to view this type of model selection as hypothesis testing for the presence of additional feedback mechanisms[51]
. One standard method of comparing the fit of Bayesian models to data is by computing Bayes factors
[52]. The Bayes factor is the ratio of marginal likelihoods of two models (the integral of the data likelihood over the posterior).One important consideration when using Bayes factors is the role of the prior in the computation[33], particularly when they are used for pointnull hypothesis testing. Here, we use the same priors for corresponding parameters to reduce this effect.
For our perfect model experiment, we would expect additional (in terms of the number of observations) and spatially explicit (rather than aggregated) data to improve the ability to distinguish between the datagenerating spatialinteractions model and the simpler nointeractions model. In Figure 7, we show the logBayes factors (along with thresholds for evidence levels proposed by Kass & Raftery[52]) to summarize the evidence for the spatialinteractions model versus the nointeractions model. We neglect the case with 25 observed parcels due to unreasonably high estimates, despite the illconstraint on the spatialinteractions model parameters and the resulting qualitatively better fit of the noiterations model. For individualparcel data, with more than 25 observed parcels, there is at least strong evidence for the spatialinteractions model no matter how long the parcels were observed, which confirms the qualitative assessment (on the summary statistic of total abandoned parcels) obtained by comparing the hindcasts in Figures 6 c and 6 d.
On the other hand, when aggregated data is used for calibration, there is essentially no quantitative evidence for the spatialinteractions model. This is the case whether we compare the models using Bayes factors or a predictive information criterion such as the WatanabeAkaike information criterion (WAIC)[54, 55]. Predictive model comparison methods avoid the direct influence of the prior on the comparison and allows for an intuitive comparison between models which have different parameterizations[56]
. The onestandard error range of the difference in WAIC between the spatialinteractions and the nointeractions model is between 2 and 2, which can be interpreted as no difference in support between the two models
[57]. However, a qualitative assessment obtained by comparing Figures 6 a and 6 b might lead a modeler to conclude that the spatialinteractions model fits the observations better than the nointeractions model. This suggests that hindcasting can serve an important supporting role to quantitative model selection.Discussion
Probabilistic calibration is an important component of the descriptive agentbased modeling process due to the influence of stochastic noise via pathdependence and feedback loops (as illustrated in Figure 5). However, as our results illustrate, each additional parameter can considerably increase the calibration data requirements. Trying to include every hypothesized feedback mechanism in the final model choice, without supporting evidence, can pose problems from statistical as well as a decisiontheoretical points of view[29, 32, 33]. Starting with a simple model and adding complexity when supported by the data can produce more skillful hindcasts, projections, and more powerful insights[58, 24].
An additional concern is the specification of prior distributions. When less data is available (particularly in summarized or aggregated form), that data will have less power to update the prior distributions.This suggests that priors should be as informative as possible (with a strong warning that priors ought not to be more informative that can be supported). While we did not take prior correlations between parameters into account for this experiment, good priors for realworld problems will include prior information about correlations between parameters.
One approach to creating informed priors which include information about the relationships between parameters is probabilistic inversion[59, 60], in which expert assessments (or, as an alternative, the results of judgement and decisionmaking or economic experiments) can be used to update more generic priors in a way which is consistent with those assessments or experimental results. This allows the survey or experimental participants to provide information directly about outcomes rather than about model parameters, and allows for a separation of the data involved in the prior construction and Bayesian updating processes.
Methods
Models
The two ABMs used in this study are represented by the influence diagram in Figure 1. The simpler model, in which the probability of housing abandonment is determined only by the frequency of experienced floods over the previous ten years, is the “nointeractions” model, and is determined by three parameters: the logistic regression intercept, the logistic regression coefficient for flood frequency, and the probability that vacant houses are filled by a new agent.
The “spatialinteractions” model includes an additional logistic regression covariate, the fraction of neighboring plots which are vacant. As a result, this model has four parameters, including the coefficient for this neighboringvacancy covariate.
Data
We generated pseudoobservations for the perfect model experiment using the model with spatial interactions, to see if we could successfully test for this effect. Parcel residency was initialized by assuming that each parcel had a 99% probability of having a resident in year 0. We used varying combinations of observed years and parcels (see Figure 2 for the observed parcel domains). The combinations were 10, 25, and 50 years, and 25, 50, and 100 parcels. Annual maxima river heights were simulated from a generalized extreme value distribution with location parameter 865, scale parameter 11, and shape parameter 0.02. Datagenerating parameter values were 6 for the logistic intercept, 20 for the localflood coefficient, 4 for the neighboringvacancy coefficient, and 0.01 for the vacancyfill probability.
As data may not be available in individualized forms, we examine the power of data for calibration and hypothesis testing about model structures in both individual and aggregate forms. In the individual case, the data set contains observations of each observed parcel at each time. In the aggregate case, we observe the total number of abandoned parcels at each time.
Calibration
We use a Bayesian framework for model calibration, based on Bayes’ Theorem
[61]:where is the posterior density, is the data likelihood, and is the prior.
Priors are provided in Table 1. These priors were constructed using a rough understanding of the model dynamics, so that the resulting probabilities of abandonment seemed plausible. They are intentionally not centered on the known datagenerating parameter values.
Parameter  Prior Distribution 

Intercept  Normal(7, 1) 
Flood Coefficient  Normal(19, 2) 
Vacancy Coefficient  Normal(5, 2) 
Vacancy Fill Probability  Beta(1, 10) 
For both individualparcel and aggregate data, we model the probability of each parcel being vacant and compute the appropriate likelihood, treating each parcel’s vacant status at time as independent and identically distributed conditional on the state in time . This representation (marginalizing over agent states to represent the model dynamics as a Markov chain) is common for many ABMs[62]. In the individual data case, we use a binomial likelihood for each parcel at each time, with the probability of a vacant parcel determined using the Markovian representation after marginalizing. In the aggregate data case, we use a Poisson likelihood on the expected number of vacant parcels.
The models described above are simple enough for us to use MCMC for Bayesian computation. We use 150,000 MetropolisHastings iterations after a preliminary adaptive run [63] of 30,000 iterations, which is used to estimate the covariance jump matrix and starting value of the production run. The preliminary run is initialized at the maximumlikelihood estimate. These runs took from several hours to several days, depending on the model and data structure.
Model Selection
Marginal likelihoods for each model are estimated using the method of bridge sampling[53]. The importance density is a truncated multivariate normal with mean and covariance derived from the MCMC output. The truncation occurs along the vacancy fill probability dimension, to ensure that this parameter only takes values between 0 and 1. 5,000 posterior and importance samples were used in the bridge sampling estimator, which resulted in standard errors[64] for the logmarginal likelihoods of orders of magnitude smaller than 1e3. WAIC and the standard errors of the differences(Vehtari et al. 2016) were computed using 10,000 posterior samples.
Acknowledgements
The authors would like to thank Ben S. Lee, Joel RoopEckart, and Tony E. Wong for their valuable input and contributions. This work was partially supported by the National Science Foundation (NSF) through the Network for Sustainable Climate Risk Management (SCRiM) under NSF cooperative agreement GEO1240507 and the Penn State Center for Climate Risk Management. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF. All codes for pseudodata generation, model analysis and figure generation can be found at http://www.github.com/vsrikrish/ABM/tree/calibration.
Author contributions statement
V.S. and K.K. conceptualized the research. V.S. wrote the model and analysis codes. V.S. and K.K. designed the figures and wrote the paper.
Additional information
Competing interests: The authors declare no competing interests.
References
 [1] Epstein, J. M. Agentbased computational models and generative social science. Complexity 4, 41–60 (1999).
 [2] Parker, D. C., Manson, S. M., Janssen, M. A., Hoffmann, M. J. & Deadman, P. Multiagent systems for the simulation of landuse and landcover change: A review. Ann. Assoc. Am. Geogr. 93, 314–337, DOI: 10.1111/14678306.9302004 (2003). http://dx.doi.org/10.1111/14678306.9302004.
 [3] Evans, T. P. & Kelley, H. Multiscale analysis of a household level agentbased model of landcover change. J. Environ. Manage. 72, 57–72, DOI: 10.1016/j.jenvman.2004.02.008 (2004).
 [4] Brown, D. G., Page, S., Riolo, R., Zellner, M. & Rand, W. Path dependence and the validation of agent‐based spatial models of land use. Int. J. Geogr. Inf. Sci. 19, 153–174, DOI: 10.1080/13658810410001713399 (2005).
 [5] Evans, T. P. & Kelley, H. Assessing the transition from deforestation to forest regrowth with an agentbased model of land cover change for southcentral indiana (USA). Geoforum 39, 819–832, DOI: 10.1016/j.geoforum.2007.03.010 (2008).
 [6] Kelley, H. & Evans, T. The relative influences of landowner and landscape heterogeneity in an agentbased model of landuse. Ecol. Econ. 70, 1075–1087, DOI: 10.1016/j.ecolecon.2010.12.009 (2011).
 [7] Evans, M. R. et al. Do simple models lead to generality in ecology? Trends Ecol. Evol. 28, 578–583, DOI: 10.1016/j.tree.2013.05.022 (2013).
 [8] Brown, C., Alexander, P., Holzhauer, S. & Rounsevell, M. D. A. Behavioral models of climate change adaptation and mitigation in landbased sectors. Wiley Interdiscip. Rev. Clim. Change 8, e448 (2017).
 [9] Black, A. J. & McKane, A. J. Stochastic formulation of ecological models and their applications. Trends Ecol. Evol. 27, 337–345, DOI: 10.1016/j.tree.2012.01.014 (2012).
 [10] Grimm, V. Ten years of individualbased modelling in ecology: what have we learned and what could we learn in the future? Ecol. Modell. 115, 129–148, DOI: 10.1016/S03043800(98)001884 (1999).
 [11] van der Vaart, E., Johnston, A. S. A. & Sibly, R. M. Predicting how many animals will be where: How to build, calibrate and evaluate individualbased models. Ecol. Modell. 326, 113–123, DOI: 10.1016/j.ecolmodel.2015.08.012 (2016).
 [12] Aerts, J. C. J. H. et al. Integrating human behaviour dynamics into flood disaster risk assessment. Nat. Clim. Chang. 8, 193–199, DOI: 10.1038/s4155801800851 (2018).
 [13] Dubbelboer, J., Nikolic, I., Jenkins, K. & Hall, J. An AgentBased model of flood risk and insurance. JASSS 20, DOI: 10.18564/jasss.3135 (2017).
 [14] Haer, T., Botzen, W. J. W. & Aerts, J. C. J. H. The effectiveness of flood risk communication strategies and the influence of social networks—insights from an agentbased model. Environ. Sci. Policy 60, 44–52, DOI: 10.1016/j.envsci.2016.03.006 (2016).
 [15] Jenkins, K., Surminski, S., Hall, J. & Crick, F. Assessing surface water flood risk and management strategies under future climate change: Insights from an AgentBased model. Sci. Total Environ. 595, 159–168, DOI: 10.1016/j.scitotenv.2017.03.242 (2017).
 [16] Tonn, G. L. & Guikema, S. D. An AgentBased model of evolving community flood risk. Risk Anal. DOI: 10.1111/risa.12939 (2017).
 [17] Balbi, S., Giupponi, C., Perez, P. & Alberti, M. A spatial agentbased model for assessing strategies of adaptation to climate and tourism demand changes in an alpine tourism destination. Environmental Modelling & Software 45, 29–51, DOI: 10.1016/j.envsoft.2012.10.004 (2013).
 [18] Barthel, R. et al. An integrated modelling framework for simulating regionalscale actor responses to global change in the water domain. Environmental Modelling & Software 23, 1095–1121, DOI: 10.1016/j.envsoft.2008.02.004 (2008).
 [19] Gerst et al. Agentbased modeling of climate policy: An introduction to the ENGAGE multilevel model framework. Environmental Modelling & Software 44, 62–75, DOI: 10.1016/j.envsoft.2012.09.002 (2013).
 [20] Schneider, S. H., Easterling, W. E. & Mearns, L. O. Adaptation: Sensitivity to natural variability, agent assumptions and dynamic climate changes. Clim. Change 45, 203–221, DOI: 10.1023/a:1005657421149 (2000).
 [21] Ziervogel, G., Bithell, M., Washington, R. & Downing, T. Agentbased social simulation: a method for assessing the impact of seasonal climate forecast applications among smallholder farmers. Agric. Syst. 83, 1–26, DOI: 10.1016/j.agsy.2004.02.009 (2005).
 [22] Epstein, J. M. Why model? Journal of Artificial Societies and Social Simulation 11, 12 (2008).
 [23] Marks, R. E. Validation and model selection: Three similarity measures compared. Complexity Economics 0, 11 (2011).
 [24] Holling, C. S. CHAPTER 8  the strategy of building models of complex ecological systems. In Watt, K. E. F. (ed.) Systems Analysis in Ecology, 195–214, DOI: 10.1016/B9781483232836.500145 (Academic Press, 1966).
 [25] Schelling, T. C. Dynamic models of segregation. J. Math. Sociol. 1, 143–186, DOI: 10.1080/0022250X.1971.9989794 (1971).
 [26] Janssen, M. & Ostrom, E. Empirically based, agentbased models. Ecol. Soc. 11, DOI: 10.5751/ES01861110237 (2006).
 [27] Dean, J. S. et al. Understanding anasazi culture change through agentbased modeling. In Kohler, T. A. & Gumerman, G. J. (eds.) Dynamics in Human and Primate Societies, Santa Fe Institute Studies on the Sciences of Complexity, 179–205 (Oxford: Oxford University Press, 2000).
 [28] Oreskes, N., ShraderFrechette, K. & Belitz, K. Verification, validation, and confirmation of numerical models in the earth sciences. Science 263, 641–646, DOI: 10.1126/science.263.5147.641 (1994).
 [29] Box, G. E. P. Science and statistics. J. Am. Stat. Assoc. 71, 791–799, DOI: 10.1080/01621459.1976.10480949 (1976).
 [30] Schwarz, N. & Ernst, A. Agentbased modeling of the diffusion of environmental innovations — an empirical approach. Technol. Forecast. Soc. Change 76, 497–511, DOI: 10.1016/j.techfore.2008.03.024 (2009).
 [31] van der Vaart, E., Beaumont, M. A., Johnston, A. S. A. & Sibly, R. M. Calibration and evaluation of individualbased models using approximate bayesian computation. Ecol. Modell. 312, 182–190, DOI: 10.1016/j.ecolmodel.2015.05.020 (2015).
 [32] Jaynes, E. T. Probability Theory: The Logic of Science (Cambridge University Press, 2003).
 [33] Robert, C. P. The Bayesian choice: from decisiontheoretic foundations to computational implementation (Springer, New York, 2007), 2nd edn.
 [34] Hastings, W. K. Monte carlo sampling methods using markov chains and their applications. Biometrika 57, 97–109, DOI: 10.2307/2334940 (1970).
 [35] Keith, J. M. & Spring, D. Agentbased bayesian approach to monitoring the progress of invasive species eradication programs. Proc. Natl. Acad. Sci. U. S. A. 110, 13428–13433, DOI: 10.1073/pnas.1216146110 (2013).
 [36] Robert, C. P. The Metropolis–Hastings algorithm. In Wiley StatsRef: Statistics Reference Online, DOI: 10.1002/9781118445112.stat07834 (John Wiley & Sons, Ltd, 2014).

[37]
Lamperti, F., Roventini, A. &
Sani, A.
Agentbased model calibration using machine learning surrogates.
J. Econ. Dyn. Control 90, 366–389, DOI: 10.1016/j.jedc.2018.03.011 (2018).  [38] Oyebamiji, O. K. et al. Gaussian process emulation of an individualbased model simulation of microbial communities. J. Comput. Sci. 22, 69–84, DOI: 10.1016/j.jocs.2017.08.006 (2017).
 [39] Kattwinkel, M. & Reichert, P. Bayesian parameter inference for individualbased models using a particle markov chain monte carlo method. Environmental Modelling & Software 87, 110–119, DOI: 10.1016/j.envsoft.2016.11.001 (2017).
 [40] Fabretti, A. Markov chain analysis in agentbased model calibration by classical and simulated minimum distance. Knowl. Inf. Syst. DOI: 10.1007/s101150181258y (2018).
 [41] Sirén, J., Lens, L., Cousseau, L. & Ovaskainen, O. Assessing the dynamics of natural populations by fitting individualbased models with approximate bayesian computation. Methods Ecol. Evol. 41, 379, DOI: 10.1111/2041210X.12964 (2018).
 [42] Frazier, D. T., Martin, G. M., Robert, C. P. & Rousseau, J. Asymptotic properties of approximate bayesian computation. Biometrika 105, 593–607, DOI: 10.1093/biomet/asy027 (2018).
 [43] Künsch, H. R. Particle filters. Bernoulli 19, 1391–1403, DOI: 10.3150/12BEJSP07 (2013). 1309.7807.
 [44] Robert, C. P., Cornuet, J.M., Marin, J.M. & Pillai, N. S. Lack of confidence in approximate bayesian computation model choice. Proc. Natl. Acad. Sci. U. S. A. 108, 15112–15117, DOI: 10.1073/pnas.1102900108 (2011).
 [45] Singh, R., Quinn, J. D., Reed, P. M. & Keller, K. Skill (or lack thereof) of datamodel fusion techniques to provide an early warning signal for an approaching tipping point. PLoS One 13, e0191768, DOI: 10.1371/journal.pone.0191768 (2018).
 [46] Fowler, L. B. et al. Flood mitigation for pennsylvania’s rural communities: CommunityScale impact of federal policies. Tech. Rep., Technical report, The Center for Rural Pennsylvania, 2017. URL http://www. rural. palegislature. us/documents/reports/FloodMitigation2017. pdf. Accessed 01302018 (2017).
 [47] Groeneveld, J. et al. Theoretical foundations of human decisionmaking in agentbased land use models – a review. Environmental Modelling & Software 87, 39–48, DOI: 10.1016/j.envsoft.2016.10.008 (2017).
 [48] Olson, R. et al. What is the effect of unresolved internal climate variability on climate sensitivity estimates?: EFFECT OF INTERNAL VARIABILITY. J. Geophys. Res. D: Atmos. 118, 4348–4358, DOI: 10.1002/jgrd.50390 (2013).
 [49] Reed, P. M. & Kollat, J. B. Save now, pay later? multiperiod manyobjective groundwater monitoring design given systematic model errors and uncertainty. Adv. Water Resour. 35, 55–68, DOI: 10.1016/j.advwatres.2011.10.011 (2012).
 [50] Ten Broeke, G., Van Voorn, G. & Ligtenberg, A. Which sensitivity analysis method should I use for my agentbased model? Journal of Artificial Societies and Social Simulation 19, 5 (2016).
 [51] Cottineau, C., Reuillon, R., Chapron, P., ReyCoyrehourcq, S. & Pumain, D. A modular modelling framework for hypotheses testing in the simulation of urbanisation. Systems 3, 348–377, DOI: 10.3390/systems3040348 (2015).
 [52] Kass, R. E. & Raftery, A. E. Bayes factors. J. Am. Stat. Assoc. 90, 773–795, DOI: 10.1080/01621459.1995.10476572 (1995).
 [53] Meng, X. L. & Wing, H. W. Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Stat. Sin. 6, 831–860 (1996).
 [54] Watanabe, S. Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. J. Mach. Learn. Res. 11, 3571–3594 (2010).
 [55] Vehtari, A., Gelman, A. & Gabry, J. Practical bayesian model evaluation using leaveoneout crossvalidation and WAIC. Stat. Comput. 1–20, DOI: 10.1007/s1122201696964 (2016).
 [56] Gelfand, A. E. & Ghosh, S. K. Model choice: A minimum posterior predictive loss approach. Biometrika 85, 1–11, DOI: 10.1093/biomet/85.1.1 (1998).
 [57] Burnham, K. P. & Anderson, D. R. Multimodel inference: Understanding AIC and BIC in model selection. Sociol. Methods Res. 33, 261–304, DOI: 10.1177/0049124104268644 (2004).
 [58] Box, G. E. P. Robustness in the strategy of scientific model building. In Launer, R. L. & Wilkinson, G. N. (eds.) Robustness in Statistics, 201–236, DOI: 10.1016/B9780124381506.500182 (Academic Press, 1979).
 [59] Kraan, B. C. & Cooke, R. M. Uncertainty in compartmental models for hazardous materials  a case study. J. Hazard. Mater. 71, 253–268, DOI: 10.1016/S03043894(99)000825 (2000).
 [60] Fuller, R. W., Wong, T. E. & Keller, K. Probabilistic inversion of expert assessments to inform projections about antarctic ice sheet responses. PLoS One 12, e0190115, DOI: 10.1371/journal.pone.0190115 (2017).
 [61] Bayes, T. An essay towards solving a problem in the doctrine of chance. Philosophical Transactions of the Royal Society of London 53, 370–418 (1763).
 [62] Izquierdo, L. R., Izquierdo, S. S., Galan, J. M. & others. Techniques to understand computer simulations: Markov chain analysis. J. Artif. Organs (2009).
 [63] Vihola, M. Robust adaptive metropolis algorithm with coerced acceptance rate. Stat. Comput. 22, 997–1008, DOI: 10.1007/s1122201192695 (2012).
 [64] FruhwirthSchnatter, S. Estimating marginal likelihoods for mixture and markov switching models using bridge sampling techniques. Econom. J. 7, 143–167, DOI: 10.1111/j.1368423X.2004.00125.x (2004).
Comments
There are no comments yet.