Test them all, is it worth it? A ground truth comparison of configuration sampling strategies

by   Axel Halin, et al.

Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them, if the effort required to do so remains acceptable. Not only this: we believe there is lots to be learned by systematically and exhaustively testing a configurable system. In this article, we report on the first ever endeavor to test all possible configurations of an industry-strength, open source configurable software system, JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70 identify the feature interactions that cause the errors. We show that sampling testing strategies (like dissimilarity and 2-wise) (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster's lead developers.


A Pairwise T-Way Test Suite Generation Strategy Using Gravitational Search Algorithm

Software faults are commonly occurred due to interactions between one or...

Feature-Interaction Aware Configuration Prioritization for Configurable Code

Unexpected interactions among features induce most bugs in a configurabl...

Achievement of Minimized Combinatorial Test Suite for Configuration-Aware Software Functional Testing Using the Cuckoo Search Algorithm

Context: Software has become an innovative solution nowadays for many ap...

Towards Adversarial Configurations for Software Product Lines

Ensuring that all supposedly valid configurations of a software product ...

T-Wise Presence Condition Coverage and Sampling for Configurable Systems

Sampling techniques, such as t-wise interaction sampling are used to ena...

Statically Verifying Continuous Integration Configurations

Continuous Integration (CI) testing is a popular software development te...

A Comprehensive Empirical Evaluation of Generating Test Suites for Mobile Applications with Diversity

Context: In search-based software engineering we often use popular heuri...

Please sign up or login with your details

Forgot password? Click here to reset