Quantifying Performance Changes with Effect Size Confidence Intervals

07/21/2020
by   Tomas Kalibera, et al.
0

Measuring performance quantifying a performance change are core evaluation techniques in programming language and systems research. Of 122 recent scientific papers, as many as 65 included experimental evaluation that quantified a performance change using a ratio of execution times. Few of these papers evaluated their results with the level of rigour that has come to be expected in other experimental sciences. The uncertainty of measured results was largely ignored. Scarcely any of the papers mentioned uncertainty in the ratio of the mean execution times, and most did not even mention uncertainty in the two means themselves. Most of the papers failed to address the non-deterministic execution of computer programs (caused by factors such as memory placement, for example), and none addressed non-deterministic compilation. It turns out that the statistical methods presented in the computer systems performance evaluation literature for the design and summary of experiments do not readily allow this either. This poses a hazard to the repeatability, reproducibility and even validity of quantitative results. Inspired by statistical methods used in other fields of science, and building on results in statistics that did not make it to introductory textbooks, we present a statistical model that allows us both to quantify uncertainty in the ratio of (execution time) means and to design experiments with a rigorous treatment of those multiple sources of non-determinism that might impact measured performance. Better still, under our framework summaries can be as simple as "system A is faster than system B by 5.5 confidence", a more natural statement than those derived from typical current practice, which are often misinterpreted. November 2013

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/09/2021

Conditional Inference: Towards a Hierarchy of Statistical Evidence

Statistical uncertainty has many sources. P-values and confidence interv...
research
03/01/2021

Accelerating Distributed-Memory Autotuning via Statistical Analysis of Execution Paths

The prohibitive expense of automatic performance tuning at scale has lar...
research
03/22/2018

A Comprehensive Analysis of Deep Regression

Deep learning revolutionized data science, and recently, its popularity ...
research
03/31/2021

A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods

The quality of a summarization evaluation metric is quantified by calcul...
research
06/24/2021

Bootstrap confidence intervals for multiple change points based on moving sum procedures

In this paper, we address the problem of quantifying uncertainty about t...
research
08/22/2019

Regression Analysis of Unmeasured Confounding

When studying the causal effect of x on y, researchers may conduct regre...
research
08/12/2021

Optimising experimental design in neutron reflectometry

Using the Fisher information (FI), the design of neutron reflectometry e...

Please sign up or login with your details

Forgot password? Click here to reset