A Comprehensive Empirical Evaluation of Generating Test Suites for Mobile Applications with Diversity

by   Thomas Vogel, et al.

Context: In search-based software engineering we often use popular heuristics with default configurations, which typically lead to suboptimal results, or we perform experiments to identify configurations on a trial-and-error basis, which may lead to better results for a specific problem. We consider the problem of generating test suites for mobile applications (apps) and rely on , a state-of-the-art approach to this problem that uses a popular heuristic (NSGA-II) with a default configuration. Objective: We want to achieve better results in generating test suites with while avoiding trial-and-error experiments to identify a more suitable configuration of . Method: We conducted a fitness landscape analysis of to analytically understand the search problem, which allowed us to make informed decisions about the heuristic and configuration of when developing . We comprehensively evaluated in a head-to-head comparison with on 34 apps. Results: Analyzing the fitness landscape of , we observed a lack of diversity of the evolved test suites and a stagnation of the search after 25 generations. realizes mechanisms that preserve the diversity of the test suites being evolved. The evaluation showed that achieves better or at least similar test results than concerning coverage and the number of revealed faults. However, typically produces longer test sequences and requires more execution time than . Conclusions: The understanding of the search problem obtained by the fitness landscape analysis helped us to find a more suitable configuration of without trial-and-error experiments. By promoting diversity of test suites during the search, improved or at least similar test results in terms of faults and coverage can be achieved.


page 1

page 2

page 3

page 4


Does Diversity Improve the Test Suite Generation for Mobile Applications?

In search-based software engineering we often use popular heuristics wit...

An Algorithm to Effect Prompt Termination of Myopic Local Search on Kauffman-s NK Landscape

In the NK model given by Kauffman, myopic local search involves flipping...

Basic Block Coverage for Search-based Unit Testing and Crash Reproduction

Search-based techniques have been widely used for white-box test generat...

Learning How to Search: Generating Effective Test Cases Through Adaptive Fitness Function Selection

Search-based test generation is guided by feedback from one or more fitn...

Test them all, is it worth it? A ground truth comparison of configuration sampling strategies

Many approaches for testing configurable software systems start from the...

Multi-Objectivizing Software Configuration Tuning (for a single performance concern)

Automatically tuning software configuration for optimizing a single perf...

Bet and Run for Test Case Generation

Anyone working in the technology sector is probably familiar with the qu...