1 Background
Monte Carlo simulation studies are computer experiments based on generating pseudorandom observations from a known truth. Statisticians usually mean Monte Carlo simulation study when they say Simulation study; throughout this article, we will just use simulation study
but this encapsulates Monte Carlo simulation studies. Simulation studies have several applications and represent an invaluable tool for statistical research nowadays: in statistics, establishing properties of current methods is key to allow them to be used – or not – with confidence. Sometimes it is not possible to derive exact analytical properties; for example, a large sample approximation may be possible, but evaluating the approximation in finite samples is required. Approximations often require assumptions as well: what are the consequences of violating such assumptions? Monte Carlo simulation studies come to the rescue and can help to answer these questions. They also can help answer questions such as: is an estimator biased in a finite sample? What are the consequences of model misspecification? Do confidence intervals for a given parameter achieve the advertised/nominal level of coverage? How does a newly developed method compare to an established one? What is the power to detect a desired effect size under complex experimental settings and analysis methods?
Simulation studies are being used increasingly in a wide variety of settings. For instance, searching on the database of peerreviewed research literature Scopus (https://www.scopus.com) with the query string TITLEABSKEY ("simulation study") AND SUBJAREA (math) yields more than 25000 results with a 25fold increase during the last 30 years, from 111 documents in 1988 to 2792 in 2018 (Figure 1). The increased availability of powerful computational tools and readytouse software to researchers surely contributed to the rise of simulation studies in the current literature.
Despite the popularity of simulation studies, they are often poorly designed, analysed, and reported. Morris et al. reviewed 100 research articles published in Volume 34 of Statistics in Medicine (2015) with at least one simulation study and found that information on datagenerating mechanisms (DGMs), number of repetitions, software, and estimands were often lacking or poorly reported, making critical appraise and replication of published studies a difficult task (Morris et al., 2019) . Another aspect of simulation studies that is often poorly reported or not reported at all is the Monte Carlo error of estimated performance measures, defined as the standard error of estimated performance, owing to the fact that a finite number of repetitions are used and so performance is estimated with uncertainty. Monte Carlo errors play an important role in understanding the role of chance in the results of simulation studies and have been showed to be severely underreported (Koehler et al., 2009).
The possibility of independently verifying results from scientific studies is a fundamental aspect of science (Laine et al., 2007); as a consequence, several reporting guidelines have emerged under the banner of the EQUATOR Network (http://www.equatornetwork.org) (Schulz et al., 2010; von Elm et al., 2007). Despite similar calls for harmonised reporting to allow for greater reproducibility in the area of computation science (e.g. Peng (Peng, 2011)) and several articles advocating for more rigour in specific aspects of simulation studies (Hoaglin and Andrews, 1975; Hauck and Anderson, 1984; DíazEmparanza, 2002; Burton et al., 2006; White, 2010; Smith and Marshall, 2011), design and reporting guidelines for simulation studies are lacking; Morris et al. introduced the ADEMP framework (Aims, Datagenerating mechanisms, Estimands, Methods, Performance measures) aiming to fill precisely that gap. In the Reporting section they compared the several ways of reporting results that they observed in their reviews, including results in text for small simulation studies, tabulating and plotting results, and even the nestedloop plot proposed by Rücker and Schwarzer for fullyfactorial simulation studies with many datagenerating mechanisms (Rücker and Schwarzer, 2014). They concluded by arguing that there is no correct way to present results, but we encourage careful thought to facilitate readability, considering the comparisons that need to be made.
As outlined in Spiegelhalter et al., there is little experimental evidence on how different types of visualisations are perceived (Spiegelhalter et al., 2011); despite that, they highlight the ease of improving understanding via interactive visualisations that can be adjusted by the user to best fit specific requirements. The recent advent of tools such as DataDriven Documents (D^{3}, or D3.js) (Bostock et al., 2011) and Shiny (Chang et al., 2019) has further facilitated the development of interactive visualisations.
The increased availability of powerful computational tools has not only contributed to a rise in the popularity of simulation studies, it has also allowed researchers to simulate an evergrowing number of datagenerating mechanisms and include several estimands and methods to compare: up to , 32, and 33, respectively, in the aforementioned review (Morris et al., 2019). With a large number of datagenerating mechanisms, estimands, or methods, analysing and reporting the results of a simulation study becomes cumbersome: what results shall we focus on so as not to bewilder readers? Which estimands and methods should we include in our tables and plots? How should we plot or tabulate several datagenerating mechanisms at once?
In an attempt to address these questions, we developed INTEREST, an INteractive Tool for Exploring REsults from Simulation sTudies. INTEREST is a browserbased interactive tool, and it requires first uploading a dataset with results from a simulation study; then, it estimates performance measures and it displays a variety of tables and plots automatically. The user can focus on specific datagenerating mechanisms, estimands, and methods: tables and plots are updated automatically. This article will introduce the implementation details of INTEREST in the Implementation section and the main features in the Results and discussion section, where we will further discuss its relevance. We also present a case study to motivate the use of INTEREST and illustrate its use in practice. Finally, we conclude the manuscript with some ending remarks in the Conclusions section.
2 Implementation
INTEREST was developed using the free statistical software R (R Core Team, 2019) and the R package Shiny (Chang et al., 2019). Shiny is an R package (and framework) that allows building interactive web apps straight from within R: the resulting applications can be hosted online, embedded in reports and dashboards, or just run as standalone apps.
The frontend of INTEREST has been built using the shinydashboard package (Chang and Borges Ribeiro, 2018); shinydashboard is based upon AdminLTE (https://adminlte.io/), an opensource admin control panel built on top of the Bootstrap framework (Version 3.x) and released under the MIT license.
The backend functionality of INTEREST is published as a standalone R package named rsimsum for easier longterm maintainability (Gasparini, 2018); rsimsum is freely available on the Comprehensive R Archive Network (CRAN) under the GNU General Public License Version 3 (https://www.gnu.org/licenses/gpl3.0).
INTEREST is available as an online application and as a standalone version for offline use. The online version is hosted at https://interest.shinyapps.io/interest/, and can be accessed via any web browser on any device (desktop computers, laptops, tablets, smartphones, etc.). The standalone offline version can be obtained from GitHub (https://github.com/ellessenne/interest) and can be run on any desktop computer and laptop with a local instance of R; if required, R can be downloaded for free from the website of the R project (R Core Team, 2019). INTEREST (as rsimsum) is published under the GNU General Public License Version 3.
3 Results and discussion
The main interface of INTEREST is presented in Figure 2. The interface is composed of a main area on the right and a navigation bar on the left; the navigation bar includes submenus for customising plots or modifying the default behaviour of INTEREST. We now introduce and describe the functionality of the application.
3.1 Data
The use of INTEREST starts by providing a tidy dataset (also known as long format, with variables in columns and observations in rows (Wickham, 2014); an example of tidy data is included in Table 1) with results from a simulation study via the Data tab from the side menu. A dataset can be provided to INTEREST in three different ways:

The user can upload a dataset. The uploaded file can be a commaseparated file (.csv), a Stata dataset (version 815, .dta), an SPSS dataset (.sav), a SAS dataset (.sas7bdat), or an R serialised object (.rds); the format will be inferred automatically from the extension of the uploaded file, and the autodetection is caseinsensitive. It is also possible to upload compressed files (ending in .gz, .bz2, .xz, or .zip) that are automatically decompressed;

The user can provide a URL link to a dataset hosted elsewhere. All considerations relative to the file format from point (1) are also valid here;

Finally, the user can paste a dataset (e.g. from Microsoft Excel) in a text box. The pasted data is assumed to be tabseparated.
Replication  DGM  Method  Estimate 
1  1  1  
2  1  1  
3  1  1  
1  2  1  
2  2  1  
3  2  1  
1  1  2  
2  1  2  
3  1  2  
1  2  2  
2  2  2  
3  2  2  
Once a dataset has been uploaded via one of the three methods outlined, the user will have to define the variables required by INTEREST and some optional variables, depending on the structure of the input dataset. The names of each column (i.e. variable) from the uploaded dataset automatically populate a set of selectlist inputs to assist the user. A variable defining a point estimate from the simulation study and a variable representing the standard error of such estimates are required, and the user has to define the true value of the estimand of interest as well. Additionally, a user can define a variable representing methods being compared with the current simulation study (and choose the comparator), and one or more variables defining datagenerating mechanisms (DGMs, e.g. sample size, true correlation, true baseline hazard function for survival models, etc.).
The View uploaded data side tab in INTEREST displays the dataset uploaded by the user using the R package DT, an R interface to the DataTables plugin for jQuery (Xie et al., 2019). The resulting table is interactive and can be sorted and filtered by the user. It is good practice to verify that the uploaded dataset is as expected before continuing with the analysis and any visual exploration.
3.2 Missing data
INTEREST includes a section for exploring missingness of estimates and/or standard errors from each repetition of a simulation study, which may occur, for example, due to nonconvergence of some repetitions. Missing values need to be carefully explored and handled at the initial stage of any analysis. Missingness may originate as a consequence of software failures: if so, the code could (or should) be made more robust to ensure fewer or no failures. Conversely, missing data may arise as a consequence of characteristics of the simulated data, yielding to nonconvergence of the estimation procedures. In other words, missing values may not be missing completely at random. A discussion on the interpretation of missing values can be found elsewhere (White et al., 2011; Morris et al., 2019).
The missing data functionality is based on the R package naniar (Tierney et al., 2019), and can be accessed via the Missing data tab. It comprises visual and tabular summaries; missing data visualisations available in INTEREST are the following:

Bar plots of number (or proportion) of missing values by method and datagenerating mechanism (if defined). Number and proportion of missing values are produced for each variable included in the data uploaded to INTEREST;

A plot to visualise the amount of missing data in the whole dataset;

A scatter plot with missing status depicted with different colours; to be able to plot missing values, they are replaced with values 10% lower than the minimum value in that variable. This plot allows identifying trends and patterns between variables in missing values (e.g. all estimates with a very large standard error have a missing point estimate);

A heat plot with methods on the horizontal axis and the datagenerating mechanisms on the vertical axis, with the colour fill representing the percentage of missingness in each tile.
Each plot can be further customised and exported (e.g. for use in slides and reports): more details in the Plots section below. Finally, INTEREST computes and outputs a table with the number, proportion, and the cumulative number of missing values per variable, stratifying by method and datagenerating mechanisms; the table can be easily exported to LaTeX format for further use (via the R package xtable (Dahl et al., 2019)).
3.3 Performance measures
INTEREST estimates performance measures automatically as soon as the user defines the required variables via the Data tab. Supported performance measures are presented in Table 2, and discussed in more detail elsewhere (Burton et al., 2006; White, 2010; Morris et al., 2019). In addition to that, INTEREST returns mean and median estimate, and mean and median squared error of the estimate. Finally, INTEREST computes and returns Monte Carlo standard errors by default. The list of performance measures estimated by INTEREST can be customised via the Options tab: by default, all are included.
Performance measure  Description 

Bias  Deviation between estimate and the true value 
Empirical standard error  Logrun standard deviation of the estimator 
Relative precision against a reference  Precision of a method B compared to a reference method A 
Mean squared error  The sum of squared bias and variance of the estimator 
Model standard error  Average estimated standard error 
Coverage  Probability that a confidence interval contains the true value 
Biaseliminated coverage  Coverage after removing bias, i.e. with confidence intervals centered on the estimated value rather than the true value of the estimand 
Power  Power of a significance test 
3.4 Tables
Estimated performance measures are presented in tabular form in the Performance measures side tab, once again using the R package DT. The table of estimated performance measures is relative to a given datagenerating mechanism, which can be modified using a select list input on the side. It is also possible to customise the number of significant digits and to select whether Monte Carlo standard errors should be excluded in each table or not via the Options tab.
Finally, it is possible to export the tables in two ways:

Export the table in LaTeX format, e.g. for use in reports, articles, or presentations, via the Export table tab and the R package xtable (Dahl et al., 2019). The caption of the table can be directly customised;

Export estimated performance measures as a dataset, e.g. to be used with a different software package of choice. The table of estimated performance measures can be exported as displayed by INTEREST or in tidy format, and in a variety of formats: commaseparated (.csv), tabseparated (.tsv), R (.rds), Stata (version 815, .dta), SPSS (.sav), and SAS (.sas7bdat).
3.5 Plots
INTEREST can produce a variety of plots to automatically visualise results from simulation studies. Plots produced by INTEREST can be categorised into two broad groups: plots of estimates (and their estimated standard errors) and plots of performance, following analysis. Plots for methodwise comparisons of estimated values and standard errors are:

Scatter plots;

Ridgeline plots (Wilke, 2018).
Each plot will include all datagenerating mechanisms by default and allows comparing serial trends and the relative performance of methods included in the simulation study.
Conversely, the following plots are supported for estimated performance:

Plots of performance measures with confidence intervals based on Monte Carlo standard errors. There are two variations of this plot: forest plots, and lolly plots. Both methods display the estimated performance measure alongside confidence intervals based on Monte Carlo standard errors; different methods are arranged side by side, either on the horizontal or on the vertical axis;

Heat plots of performance measures: these plots are mosaic plots where the several methods being compared (if defined) are on the horizontal axis and the datagenerating mechanisms are on the vertical axis. Then, each tile of the mosaic plot is coloured according to the value of a given performance measure. To the best of our knowledge, this is a novel way of visualising results from simulation studies, with an application in practice that can be found elsewhere (Gasparini et al., 2019);

Zip plots to visually explain coverage probabilities by plotting the confidence intervals directly. More information on zip plots is presented elsewhere (Morris et al., 2019);

Nested loop plots, useful to compare performance measures from studies with several DGMs at once. This visualisation is described in more detail elsewhere (Rücker and Schwarzer, 2014).
Finally, all plots can be exported for use in manuscript, reports, or presentations by simply clicking the Save plot button underneath a plot; all plots are exported by default in .png format, but other options are available via the Options tab. For instance, to suit a wide variety of possible use cases, INTEREST supports several alternative image formats such as pdf, svg, and eps. Through the Options tab it is also possible to customise the resolution of the plot for nonvectorial format (in dots per inch, dpi) and the physical size (height and width) of the plots to be exported. The Options tab allows further customisations: for instance, it is possible to (1) define a custom label for the xaxis and the yaxis and (2) change the overall appearance of the plot by applying one of the predefined themes (which are described in more detail in the User guide tab).
3.6 INTEREST for exploring results
INTEREST allows researchers to upload a dataset with the results of their Monte Carlo simulation study obtaining estimates of performance in a quick and straightforward way. This is very appealing, especially with simulation studies with several datagenerating mechanisms where it could be confusing to investigate all scenarios at once. Using the app it is possible to vary datagenerating mechanisms and obtain updated tables and plots in realtime, therefore allowing to quickly iterate and take into consideration all possible scenarios.
3.7 INTEREST for disseminating results
One of the intended usage scenarios for INTEREST consists of supplementing reporting of simulation studies. This is especially useful with large simulation studies, where it is most cumbersome to summarise all results in a manuscript: it is common to include in the main manuscript only a subset of results for conciseness. The remaining results are then relegated to supplementary material, web appendices, or not published at all  undermining dissemination and replicability of a study.
Furthermore, given that it is becoming increasingly common to publish the code of simulation study, one could publish the dataset with the results alongside the code used to obtain it. That dataset could then be uploaded to INTEREST by readers, who could then explore the full results of the study as they wish. Given the ubiquity of web services like GitHub (https://github.com) and datasharing repositories such as Zenodo (https://zenodo.org/), we encourage INTEREST users to publish online the full results of their simulation studies for other users to download and experiment with.
4 Future developments
Although INTEREST is fully functional in its current state, several future developments are being planned. For instance, we aim to include support for multiple estimands at once as currently supported by rsimsum via the multisimsum function. We also aim to improve the flexibility of INTEREST in terms of customisation (of tables and plots), e.g. by displaying the raw R code used to generate the plots behind the scenes. Finally, we are considering adding additional interactive features to the app via HTML widgets, D^{3}, or other approaches; there are several R packages that allow incorporating interactive graphs into Shiny apps such as htmlwidgets (Vaidyanathan et al., 2018), plotly (Sievert, 2018), and r2d3 (Luraschi and Allaire, 2018).
5 Case study
The case study included in this Section illustrates the use of INTEREST to analyse publicly available results of a simulation study. In particular, we will be using the results from the worked illustrative example included in Morris et al. (Morris et al., 2019).
The study dataset contains the results of a simulation study comparing three different methods for estimating the hazard ratio in a randomised trial with a time to event outcome. In particular, the methods being compared are proportional hazards survival models of the kind:
where is the log hazard ratio for the effect of a binary exposure (e.g. treatment). This class of models requires an assumption regarding the shape of the baseline hazard function : it can be assumed to follow a given parametric distribution, or it can be left unspecified (yielding therefore a Cox model). The aim of this simulation study consists of assessing the impact of such an assumption on the estimation of the log hazard ratio.
Morris et al. consider two distinct datagenerating mechanisms, varying the baseline hazard function:

An exponential baseline hazard with (DGM = 1);

A Weibull baseline hazard with (DGM = 2).
In both settings, data are simulated on 300 patients with a binary covariate (e.g. treatment) simulated using  simple randomisation with an equal allocation ratio. The log hazard ratio is set to be ; this is the true value of the estimand of interest.
Three distinct methods are fit to each simulated scenario: a parametric survival model that assumes an exponential baseline hazard, a parametric survival model that assumes a Weibull baseline hazard, and a Cox semiparametric survival model.
Finally, the performance measures of interest are bias, coverage, empirical and modelbased standard errors. Assuming that , 1600 repetitions are run to ensure that the Monte Carlo standard error of bias (the key performance measure of interest) is lower than 0.005.
The dataset with the results of this simulation study is publicly available, and can be downloaded from GitHub: https://github.com/tpmorris/simtutorial/raw/master/Stata/estimates.dta. Within the dataset published on GitHub, the exponential, Weibull, and Cox models are coded as model 1, 2, and 3, respectively. The abovementioned dataset is in Stata format; an R version is available as well (https://github.com/tpmorris/simtutorial/raw/master/R/estimates.rds), and INTEREST supports both.
The workflow of INTEREST starts by providing the dataset with the results of the simulation study. Given that the dataset is already available online, we can directly pass the URL above to INTEREST and then define the required variables (as illustrated in Figure 3); the uploaded dataset can then be verified via the View uploaded data tab (Figure 4).
We can also customise the performance measures reported by INTEREST via the Options tab (Figure 5), e.g. focussing on those outlined above as key performance measures (bias, coverage probability, empirical standard errors, modelbased standard errors).
The next step of the workflow consists of investigating missing values: this can be achieved via the Missing data tab. In particular, there is no missing data in the study dataset (Figure 6). We can, therefore, continue the analysis knowing that there is no pattern of serial missingness or nonconvergence issues in our data.
The performance measures of interest are tabulated in the Performance measures tab, e.g. for DGM = 2 (Figure 7). We can see that bias for the exponential model is much larger than the Weibull and Cox models: approximately 10% of the true value (in absolute terms) compared to less than 1%. Empirical and modelbased standard errors are quite similar for the Weibull and Cox models; conversely, the exponential model seemed to overestimate the modelbased standard error. Coverage was as advertised for all methods, at approximately 95%. By comparison, all models performed equally in the other scenario (DGM = 1); these results are omitted from the manuscript for brevity, but we encourage readers to replicate this analysis and verify our statement.
The Performance measures tab provides a LaTeX table ready to be pasted e.g. in a manuscript: the resulting table is included as Table 3. A dataset with all the estimated performance measures here tabulated can also be exported to be used elsewhere (Figure 8).
We can also visualise the results of this simulation study. First, we can produce a methodwise comparison of point estimates from each method using e.g. scatter plots (Figure 9) or BlandAltman plots (Figure 10). With both plots, it is possible to appreciate that for the DGM with the exponential model yields point estimates that are quite different compared to the Weibull and Cox models. Analogous plots can be obtained for estimated standard errors.
The performance measures tabulated in the Performance measures tab can also be plotted via the Plots tab. For instance, it is straightforward to obtain a forest plot for bias (as illustrated in Figure 11) which can be exported by clicking the Save plot button. The plots’ appearance can also be customised via the Options tab, e.g. by modifying the axes’ labels and the overall theme of the plot (Figure 12); the resulting forest plot, exported in .pdf format, is included as Figure 13. Several other data visualisations are supported by INTEREST, as described in the previous Sections: lolly plots, zip plots, and so on.
Performance Measure  1  2  3 

Bias in point estimate  0.0494 (0.0035)  0.0048 (0.0038)  0.0062 (0.0038) 
Empirical standard error  0.1381 (0.0024)  0.1516 (0.0027)  0.1511 (0.0027) 
Modelbased standard error  0.1539 (0.0001)  0.1541 (0.0001)  0.1542 (0.0001) 
Coverage of nominal 95% confidence interval  0.9600 (0.0049)  0.9556 (0.0051)  0.9575 (0.0050) 
6 Conclusions
As outlined in the introduction, Monte Carlo simulation studies are too often poorly analysed and reported (Morris et al., 2019). Given the increased use in methodological statistical research, we hope that INTEREST could improve reporting and disseminating results from simulation studies to a large extent. As illustrated in the case study, the exploration and analysis of the Monte Carlo simulation study of Morris et al. can be fully reproduced by using INTEREST. Estimated performance measures are tabulated automatically, and plots can be used to visualise the performance measures of interest. Moreover, the user is not constrained to a given set of plots and can fully explore the results with ease e.g. by varying DGMs to focus on or by choosing different data visualisations. Most interestingly, the only requirement to reproduce the simulation study described in the case study is a device with a web browser and connection to the Internet. To the best of our knowledge, there is no similar application readily available to be used by researchers and readers of published Monte Carlo simulation studies alike.
Acknowledgements
TPM is supported by the Medical Research Council (grant numbers MC_UU_12023/21 and MC_UU_12023/29). MJC is partially funded by the MRCNIHR Methodology Research Panel (MR/P015433/1).
We thank Ian R. White for discussions that lead to the inception and development of INTEREST.
References
 Altman and Bland (1983) Altman, D. G. and Bland, J. M. (1983). Measurement in medicine: The analysis of method comparison studies. The Statistician, 32(3):307, DOI: 10.2307/2987937, https://doi.org/10.2307%2F2987937.
 Bland and Altman (1999) Bland, J. M. and Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2):135160, DOI: 10.1177/096228029900800204, https://doi.org/10.1177%2F096228029900800204.
 Bostock et al. (2011) Bostock, M., Ogievetsky, V., and Heer, J. (2011). D^{3}: Datadriven documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):23012309, DOI: 10.1109/tvcg.2011.185.
 Burton et al. (2006) Burton, A., Altman, D. G., Royston, P., and Holder, R. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25(24):42794292, DOI: 10.1002/sim.2673.
 Chang and Borges Ribeiro (2018) Chang, W. and Borges Ribeiro, B. (2018). shinydashboard: Create Dashboards with shiny, https://CRAN.Rproject.org/package=shinydashboard. R package version 0.7.1.
 Chang et al. (2019) Chang, W., Cheng, J., Allaire, J., Xie, Y., and McPherson, J. (2019). shiny: Web Application Framework for R, https://CRAN.Rproject.org/package=shiny. R package version 1.3.2.
 Dahl et al. (2019) Dahl, D. B., Scott, D., Roosen, C., Magnusson, A., and Swinton, J. (2019). xtable: Export Tables to LaTeX or HTML, https://CRAN.Rproject.org/package=xtable. R package version 1.84.
 DíazEmparanza (2002) DíazEmparanza, I. (2002). Is a small Monte Carlo analysis a good analysis? Statistical Papers, 43(4):567577.
 Gasparini (2018) Gasparini, A. (2018). rsimsum: Summarise results from Monte Carlo simulation studies. Journal of Open Source Software, 3(26):739, DOI: 10.21105/joss.00739, https://doi.org/10.21105/joss.00739.
 Gasparini et al. (2019) Gasparini, A., Clements, M. S., Abrams, K. R., and Crowther, M. J. (2019). Impact of model misspecification in shared frailty survival models. Statistics in Medicine, DOI: 10.1002/sim.8309, https://doi.org/10.1002/sim.8309.
 Hauck and Anderson (1984) Hauck, W. W. and Anderson, S. (1984). A survey regarding the reporting of simulation studies. The American Statistician, 38(3):214216.
 Hoaglin and Andrews (1975) Hoaglin, D. C. and Andrews, D. F. (1975). The reporting of computationbased results in statistics. The American Statistician, 29(3):122126.
 Koehler et al. (2009) Koehler, E., Brown, E., and Haneuse, S. J. (2009). On the assessment of Monte Carlo error in simulationbased statistical analyses. The American Statistician, 63(2):155162, DOI: 10.1198/tast.2009.0030.
 Laine et al. (2007) Laine, C., Goodman, S. N., Griswold, M. E., and Sox, H. C. (2007). Reproducible research: Moving toward research the public can really trust. Annals of Internal Medicine, 146(6):450453, DOI: 10.7326/00034819146620070320000154.
 Luraschi and Allaire (2018) Luraschi, J. and Allaire, J. (2018). r2d3: Interface to D^{3} Visualizations, https://CRAN.Rproject.org/package=r2d3. R package version 0.2.3.
 Morris et al. (2019) Morris, T. P., White, I., and Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, pages 129, DOI: 10.1002/sim.8086.
 Peng (2011) Peng, R. D. (2011). Reproducible research in computational science. 334(6060):12261227, DOI: 10.1126/science.1213847.
 R Core Team (2019) R Core Team (2019). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, https://www.Rproject.org/.
 Rücker and Schwarzer (2014) Rücker, G. and Schwarzer, G. (2014). Presenting simulation results in a nested loop plot. BMC Medical Research Methodology, 14(1), DOI: 10.1186/1471228814129.
 Schulz et al. (2010) Schulz, K. F., Altman, D. G., Moher, D., and for the CONSORT Group (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. PLOS Medicine, 7(3):17, DOI: 10.1371/journal.pmed.1000251.
 Sievert (2018) Sievert, C. (2018). plotly for R, https://plotlyr.com.
 Smith and Marshall (2011) Smith, M. K. and Marshall, A. (2011). Importance of protocols for simulation studies in clinical drug development. Statistical Methods in Medical Research, 20(6):613622.
 Spiegelhalter et al. (2011) Spiegelhalter, D., Pearson, M., and Short, I. (2011). Visualizing uncertainty about the future. Science, 333(6048):13931400, DOI: 10.1126/science.1191181.
 Tierney et al. (2019) Tierney, N., Cook, D., McBain, M., and Fay, C. (2019). naniar: Data Structures, Summaries, and Visualisations for Missing Data, https://CRAN.Rproject.org/package=naniar. R package version 0.4.2.
 Vaidyanathan et al. (2018) Vaidyanathan, R., Xie, Y., Allaire, J., Cheng, J., and Russell, K. (2018). htmlwidgets: HTML Widgets for R, https://CRAN.Rproject.org/package=htmlwidgets. R package version 1.3.
 von Elm et al. (2007) von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsch, P., Vandenbroucke, J. P., and for the STROBE Initiative (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (Strobe) Statement: Guidelines for reporting observational studies. PLOS Medicine, 4(10):15, DOI: 10.1371/journal.pmed.0040296.
 White (2010) White, I. R. (2010). simsum: Analyses of simulation studies including Monte Carlo error. The Stata Journal, 10(3):369385.

White et al. (2011)
White, I. R., Royston, P., and Wood, A. M. (2011).
Multiple imputation using chained equations: Issues and guidance for practice.
Statistics in Medicine, 30(4):377399, DOI: 10.1002/sim.4067.  Wickham (2014) Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), DOI: 10.18637/jss.v059.i10.
 Wilke (2018) Wilke, C. O. (2018). ggridges: Ridgeline Plots in ggplot2, https://CRAN.Rproject.org/package=ggridges. R package version 0.5.1.
 Xie et al. (2019) Xie, Y., Cheng, J., and Tan, X. (2019). DT: A Wrapper of the JavaScript Library DataTables, https://CRAN.Rproject.org/package=DT. R package version 0.8.
Comments
There are no comments yet.