Knowing what to know: Implications of the choice of prior distribution on the behavior of adaptive design optimization

03/22/2023
by   Sabina J. Sloman, et al.
0

Adaptive design optimization (ADO) is a state-of-the-art technique for experimental design (Cavagnaro, Myung, Pitt, Kujala, 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about a hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler's existing knowledge, specified in the form of a prior distribution. Informative priors align with the distribution of the focal construct in the participant population. This alignment is assumed by ADO's internal assessment of expected information gain. If the prior is instead misinformative, i.e., does not align with the participant population, ADO's estimates of expected information gain could be inaccurate. In many cases, the true distribution that characterizes the participant population is unknown, and experimenters rely on heuristics in their choice of prior and without an understanding of how this choice affects ADO's behavior. Our work introduces a mathematical framework that facilitates investigation of the consequences of the choice of prior distribution on the efficiency of experiments designed using ADO. Through theoretical and empirical results, we show that, in the context of prior misinformation, measures of expected information gain are distinct from the correctness of the corresponding inference. Through a series of simulation experiments, we show that, in the case of parameter estimation, ADO nevertheless outperforms other design methods. Conversely, in the case of model selection, misinformative priors can lead inference to favor the wrong model, and rather than mitigating this pitfall, ADO exacerbates it.

READ FULL TEXT
research
06/10/2022

On the safe use of prior densities for Bayesian model selection

The application of Bayesian inference for the purpose of model selection...
research
04/17/2023

Metrics for Bayesian Optimal Experiment Design under Model Misspecification

The conventional approach to Bayesian decision-theoretic experiment desi...
research
09/18/2018

Comparison between Suitable Priors for Additive Bayesian Networks

Additive Bayesian networks are types of graphical models that extend the...
research
05/20/2022

Robust Expected Information Gain for Optimal Bayesian Experimental Design Using Ambiguity Sets

The ranking of experiments by expected information gain (EIG) in Bayesia...
research
08/20/2011

Simulation-based optimal Bayesian experimental design for nonlinear systems

The optimal selection of experimental conditions is essential to maximiz...
research
01/23/2023

Sharing information across patient subgroups to draw conclusions from sparse treatment networks

Network meta-analysis (NMA) usually provides estimates of the relative e...
research
07/22/2021

Preliminary investigation into how limb choice affects kinesthetic perception

We have a limited understanding of how we integrate haptic information i...

Please sign up or login with your details

Forgot password? Click here to reset