Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality

by   Carlo A. Furia, et al.

Statistical analysis is the tool of choice to turn data into information, and then information into empirical knowledge. To be valid, the process that goes from data to knowledge should be supported by detailed, rigorous guidelines, which help ferret out issues with the data or model, and lead to qualified results that strike a reasonable balance between generality and practical relevance. Such guidelines are being developed by statisticians to support the latest techniques for Bayesian data analysis. In this article, we frame these guidelines in a way that is apt to empirical research in software engineering. To demonstrate the guidelines in practice, we apply them to reanalyze a GitHub dataset about code quality in different programming languages. The dataset's original analysis (Ray et al., 2014) and a critical reanalysis (Berger at al., 2019) have attracted considerable attention – in no small part because they target a topic (the impact of different programming languages) on which strong opinions abound. The goals of our reanalysis are largely orthogonal to this previous work, as we are concerned with demonstrating, on data in an interesting domain, how to build a principled Bayesian data analysis and to showcase some of its benefits. In the process, we will also shed light on some critical aspects of the analyzed data and of the relationship between programming languages and code quality. The high-level conclusions of our exercise will be that Bayesian statistical techniques can be applied to analyze software engineering data in a way that is principled, flexible, and leads to convincing results that inform the state of the art while highlighting the boundaries of its validity. The guidelines can support building solid statistical analyses and connecting their results, and hence help buttress continued progress in empirical software engineering research.


page 1

page 2

page 3

page 4


Bayesian Data Analysis in Empirical Software Engineering Research

Statistics comes in two main flavors: frequentist and Bayesian. For hist...

Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions

There is abundant observational data in the software engineering domain,...

Data of low quality is better than no data

Missing data is not uncommon in empirical software engineering research ...

On the Impact of Programming Languages on Code Quality

This paper is a reproduction of work by Ray et al. which claimed to have...

Toward Inclusion of Children as Software Engineering Stakeholders

Background: A growing amount of software is available to children today....

Toward a principled Bayesian workflow in cognitive science

Experiments in research on memory, language, and in other areas of cogni...

RepoMiner: a Language-agnostic Python Framework to Mine Software Repositories for Defect Prediction

Data originating from open-source software projects provide valuable inf...

Please sign up or login with your details

Forgot password? Click here to reset