Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions

by   Carlo A. Furia, et al.

There is abundant observational data in the software engineering domain, whereas running large-scale controlled experiments is often practically impossible. Thus, most empirical studies can only report statistical correlations – instead of potentially more insightful and robust causal relations. This paper discusses some novel techniques that support analyzing purely observational data for causal relations. Using fundamental causal models such as directed acyclic graphs, one can rigorously express, and partially validate, causal hypotheses; and then use the causal information to guide the construction of a statistical model that captures genuine causal relations – such that correlation does imply causation. We apply these ideas to analyzing public data about programmer performance in Code Jam, a large world-wide coding contest organized by Google every year. Specifically, we look at the impact of different programming languages on a participant's performance in the contest. While the overall effect associated with programming languages is weak compared to other variables – regardless of whether we consider correlational or causal links – we found considerable differences between a purely statistical and a causal analysis of the very same data. The takeaway message is that even an imperfect causal analysis of observational data can help answer the salient research questions more precisely and more robustly than with just purely statistical techniques.


page 1

page 2

page 3

page 4


Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality

Statistical analysis is the tool of choice to turn data into information...

Applications of statistical causal inference in software engineering

This paper reviews existing work in software engineering that applies st...

Do learned representations respect causal relationships?

Data often has many semantic attributes that are causally associated wit...

Bayesian Data Analysis in Empirical Software Engineering Research

Statistics comes in two main flavors: frequentist and Bayesian. For hist...

On the Impact of Programming Languages on Code Quality

This paper is a reproduction of work by Ray et al. which claimed to have...

Using R for teaching and research

R is a language and environment for statistical computing and graphics, ...

Please sign up or login with your details

Forgot password? Click here to reset