Contra-Analysis: Prioritizing Meaningful Effect Size in Scientific Research

by   Bruce A. Corliss, et al.

At every phase of scientific research, scientists must decide how to allocate limited resources to pursue the research inquiries with the greatest potential. This prioritization dictates which controlled interventions are studied, awarded funding, published, reproduced with repeated experiments, investigated in related contexts, and translated for societal use. There are many factors that influence this decision-making, but interventions with larger effect size are often favored because they exert the greatest influence on the system studied. To inform these decisions, scientists must compare effect size across studies with dissimilar experiment designs to identify the interventions with the largest effect. These studies are often only loosely related in nature, using experiments with a combination of different populations, conditions, timepoints, measurement techniques, and experiment models that measure the same phenomenon with a continuous variable. We name this assessment contra-analysis and propose to use credible intervals of the relative difference in means to compare effect size across studies in a meritocracy between competing interventions. We propose a data visualization, the contra plot, that allows scientists to score and rank effect size between studies that measure the same phenomenon, aid in determining an appropriate threshold for meaningful effect, and perform hypothesis tests to determine which interventions have meaningful effect size. We illustrate the use of contra plots with real biomedical research data. Contra-analysis promotes a practical interpretation of effect size and facilitates the prioritization of scientific research.


Contra-Analysis for Determining Negligible Effect Size in Scientific Research

Scientific experiments study interventions that show evidence of an effe...

The Least Difference in Means: A Statistic for Effect Size Strength and Practical Significance

With limited resources, scientific inquiries must be prioritized for fur...

The Most Difference in Means: A Statistic for Null and Near-Zero Results

Two-sample p-values test for statistical significance. Yet p-values cann...

Powering population health research: Considerations for plausible and actionable effect sizes

Evidence for Action (E4A), a signature program of the Robert Wood Johnso...

FirstPersonScience: Quantifying Psychophysics for First Person Shooter Tasks

In the emerging field of esports research, there is an increasing demand...

Measuring Influence in Science: Standing on the Shoulders of Which Giants?

I study the measurement of the influence of scientists based on bibliogr...

FairPrep: Promoting Data to a First-Class Citizen in Studies on Fairness-Enhancing Interventions

The importance of incorporating ethics and legal compliance into machine...

Please sign up or login with your details

Forgot password? Click here to reset