Towards effective assessment of steady state performance in Java software: Are we there yet?

by   Luca Traini, et al.

Microbenchmarking is a widely used form of performance testing in Java software. A microbenchmark repeatedly executes a small chunk of code while collecting measurements related to its performance. Due to Java Virtual Machine optimizations, microbenchmarks are usually subject to severe performance fluctuations in the first phase of their execution (also known as warmup). For this reason, software developers typically discard measurements of this phase and focus their analysis when benchmarks reach a steady state of performance. Developers estimate the end of the warmup phase based on their expertise, and configure their benchmarks accordingly. Unfortunately, this approach is based on two strong assumptions: (i) benchmarks always reach a steady state of performance and (ii) developers accurately estimate warmup. In this paper, we show that Java microbenchmarks do not always reach a steady state, and often developers fail to accurately estimate the end of the warmup phase. We found that a considerable portion of studied benchmarks do not hit the steady state, and warmup estimates provided by software developers are often inaccurate (with a large error). This has significant implications both in terms of results quality and time-effort. Furthermore, we found that dynamic reconfiguration significantly improves warmup estimation accuracy, but still it induces suboptimal warmup estimates and relevant side-effects. We envision this paper as a starting point for supporting the introduction of more sophisticated automated techniques that can ensure results quality in a timely fashion.


Comprehending Test Code: An Empirical Study

Developers spend a large portion of their time and effort on comprehendi...

SAT-Based Extraction of Behavioural Models for Java Libraries with Collections

Behavioural models are a valuable tool for software verification, testin...

An Empirical Study on Maintainable Method Size in Java

Code metrics have been widely used to estimate software maintenance effo...

Atoms of Confusion in Java

Although writing code seems trivial at times, problems arise when humans...

Profiling and Optimizing Java Streams

The Stream API was added in Java 8 to allow the declarative expression o...

Blindspots in Python and Java APIs Result in Vulnerable Code

Blindspots in APIs can cause software engineers to introduce vulnerabili...

Dev2vec: Representing Domain Expertise of Developers in an Embedding Space

Accurate assessment of the domain expertise of developers is important f...

Please sign up or login with your details

Forgot password? Click here to reset