Understanding and Supporting Debugging Workflows in Multiverse Analysis

10/07/2022
by   Ken Gu, et al.
0

Multiverse analysis-a paradigm for statistical analysis that considers all combinations of reasonable analysis choices in parallel-promises to improve transparency and reproducibility. Although recent tools help analysts specify multiverse analyses, they remain difficult to use in practice. In this work, we conduct a formative study with four multiverse researchers, which identifies debugging as a key barrier. We find debugging is challenging because of the latency between running analyses and detecting bugs, and the scale of metadata needed to be processed to diagnose a bug. To address these challenges, we prototype a command-line interface tool, Multiverse Debugger, which helps diagnose bugs in the multiverse and propagate fixes. In a second, focused study (n=13), we use Multiverse Debugger as a probe to develop a model of debugging workflows and identify challenges, including the difficulty in understanding the composition of a multiverse. We conclude with design implications for future multiverse analysis authoring systems.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
08/11/2021

Why are Some Bugs Non-Reproducible? An Empirical Investigation using Data Fusion

Software developers attempt to reproduce software bugs to understand the...
research
01/24/2020

Advaita: Bug Duplicity Detection System

Bugs are prevalent in software development. To improve software quality,...
research
03/23/2021

What we can learn from how programmers debug their code

Researchers have developed numerous debugging approaches to help program...
research
07/19/2023

An Analysis of Bugs In Persistent Memory Application

Over the years of challenges on detecting the crash consistency of non-v...
research
11/12/2020

Large-Scale Manual Validation of Bug Fixing Commits: A Fine-grained Analysis of Tangling

Context: Tangled commits are changes to software that address multiple c...
research
05/19/2023

Bugs as Features (Part II): A Perspective on Enriching Microbiome-Gut-Brain Axis Analyses with Multidisciplinary Techniques

The microbiome-gut-brain-axis field is multidisciplinary, benefiting fro...

Please sign up or login with your details

Forgot password? Click here to reset