Workflow Techniques for the Robust Use of Bayes Factors

03/15/2021
by   Daniel J. Schad, et al.
0

Inferences about hypotheses are ubiquitous in the cognitive sciences. Bayes factors provide one general way to compare different hypotheses by their compatibility with the observed data. Those quantifications can then also be used to choose between hypotheses. While Bayes factors provide an immediate approach to hypothesis testing, they are highly sensitive to details of the data/model assumptions. Moreover it's not clear how straightforwardly this approach can be implemented in practice, and in particular how sensitive it is to the details of the computational implementation. Here, we investigate these questions for Bayes factor analyses in the cognitive sciences. We explain the statistics underlying Bayes factors as a tool for Bayesian inferences and discuss that utility functions are needed for principled decisions on hypotheses. Next, we study how Bayes factors misbehave under different conditions. This includes a study of errors in the estimation of Bayes factors. Importantly, it is unknown whether Bayes factor estimates based on bridge sampling are unbiased for complex analyses. We are the first to use simulation-based calibration as a tool to test the accuracy of Bayes factor estimates. Moreover, we study how stable Bayes factors are against different MCMC draws. We moreover study how Bayes factors depend on variation in the data. We also look at variability of decisions based on Bayes factors and how to optimize decisions using a utility function. We outline a Bayes factor workflow that researchers can use to study whether Bayes factors are robust for their individual analysis, and we illustrate this workflow using an example from the cognitive sciences. We hope that this study will provide a workflow to test the strengths and limitations of Bayes factors as a way to quantify evidence in support of scientific hypotheses. Reproducible code is available from https://osf.io/y354c/.

READ FULL TEXT
research
03/24/2021

Meta Analysis of Bayes Factors

Bayes Factors, the Bayesian tool for hypothesis testing, are receiving i...
research
10/19/2021

Bayes Factors can only Quantify Evidence w.r.t. Sets of Parameters, not w.r.t. (Prior) Distributions on the Parameter

Bayes factors are characterized by both the powerful mathematical framew...
research
10/19/2021

How to Guide Decisions with Bayes Factors

Some scientific research questions ask to guide decisions and others do ...
research
09/14/2018

Assessing Bayes factor surfaces using interactive visualization and computer surrogate modeling

Bayesian model selection provides a natural alternative to classical hyp...
research
03/04/2022

Data aggregation can lead to biased inferences in Bayesian linear mixed models

Bayesian linear mixed-effects models are increasingly being used in the ...
research
09/18/2019

Semantic and Cognitive Tools to Aid Statistical Inference: Replace Confidence and Significance by Compatibility and Surprise

Researchers often misinterpret and misrepresent statistical outputs. Thi...
research
01/20/2022

Using Machine Learning to Test Causal Hypotheses in Conjoint Analysis

Conjoint analysis is a popular experimental design used to measure multi...

Please sign up or login with your details

Forgot password? Click here to reset