A Case Study on the Stability of Performance Tests for Serverless Applications

07/28/2021
by   Simon Eismann, et al.
0

Context. While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective. In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method. We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results. We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion. Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/28/2020

Sizeless: Predicting the optimal size of serverless functions

Serverless functions are a cloud computing paradigm that reduces operati...
research
07/03/2023

A numerical variability approach to results stability tests and its application to neuroimaging

Ensuring the long-term reproducibility of data analyses requires results...
research
07/18/2019

Fragility of Layout-Based and Visual GUI Test Scripts: An Assessment Study on a Hybrid Mobile Application

Context: Albeit different approaches exist for automated GUI testing of ...
research
03/09/2022

Combining dependent p-values resulting from multiple effect size homogeneity tests in meta-analysis for binary outcomes

Testing effect size homogeneity is an essential part when conducting a m...
research
08/17/2021

Stable combination tests

This paper proposes a stable combination test, which is a natural extens...
research
12/06/2019

FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair

Much research on software testing makes an implicit assumption that test...
research
12/09/2021

A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests

Test flakiness forms a major testing concern. Flaky tests manifest non-d...

Please sign up or login with your details

Forgot password? Click here to reset