Model-independent comparison of simulation output

09/30/2015
by   Nuno Fachada, et al.
0

Computational models of complex systems are usually elaborate and sensitive to implementation details, characteristics which often affect their verification and validation. Model replication is a possible solution to this issue. It avoids biases associated with the language or toolkit used to develop the original model, not only promoting its verification and validation, but also fostering the credibility of the underlying conceptual model. However, different model implementations must be compared to assess their equivalence. The problem is, given two or more implementations of a stochastic model, how to prove that they display similar behavior? In this paper, we present a model comparison technique, which uses principal component analysis to convert simulation output into a set of linearly uncorrelated statistical measures, analyzable in a consistent, model-independent fashion. It is appropriate for ascertaining distributional equivalence of a model replication with its original implementation. Besides model-independence, this technique has three other desirable properties: a) it automatically selects output features that best explain implementation differences; b) it does not depend on the distributional properties of simulation output; and, c) it simplifies the modelers' work, as it can be used directly on simulation outputs. The proposed technique is shown to produce similar results to the manual or empirical selection of output features when applied to a well-studied reference model.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/22/2016

micompr: An R Package for Multivariate Independent Comparison of Observations

The R package micompr implements a procedure for assessing if two or mor...
03/22/2016

SimOutUtils - Utilities for analyzing time series simulation output

SimOutUtils is a suite of MATLAB/Octave functions for studying and analy...
03/27/2018

Proving Equivalence Between Imperative and MapReduce Implementations Using Program Transformations

Distributed programs are often formulated in popular functional framewor...
03/27/2019

A Conceptual Framework for Assessing Anonymization-Utility Trade-Offs Based on Principal Component Analysis

An anonymization technique for databases is proposed that employs Princi...
04/14/2022

The replication of non-inferiority and equivalence studies

Replication studies are increasingly conducted to assess credibility of ...
05/31/2022

A Replication Study on Predicting Metamorphic Relations at Unit Testing Level

Metamorphic Testing (MT) addresses the test oracle problem by examining ...

Code Repositories

pphpc

Several implementations of the standard PPHPC agent-based model


view repo

simoututils

SimOutUtils - Utilities for analyzing time series simulation output


view repo

micompr

Multivariate independent comparison of observations.


view repo

micompr

Multivariate independent comparison of observations.


view repo

micompr

:exclamation: This is a read-only mirror of the CRAN R package repository. micompr — Multivariate Independent Comparison of Observations. Homepage: http://github.com/fakenmc/micompr Report bugs for this package: https://github.com/fakenmc/micompr/issues


view repo

Please sign up or login with your details

Forgot password? Click here to reset