A Comprehensive Empirical Evaluation of Generating Test Suites for Mobile Applications with Diversity
share
Context: In search-based software engineering we often use popular heuristics with default configurations, which typically lead to suboptimal results, or we perform experiments to identify configurations on a trial-and-error basis, which may lead to better results for a specific problem. We consider the problem of generating test suites for mobile applications (apps) and rely on , a state-of-the-art approach to this problem that uses a popular heuristic (NSGA-II) with a default configuration. Objective: We want to achieve better results in generating test suites with while avoiding trial-and-error experiments to identify a more suitable configuration of . Method: We conducted a fitness landscape analysis of to analytically understand the search problem, which allowed us to make informed decisions about the heuristic and configuration of when developing . We comprehensively evaluated in a head-to-head comparison with on 34 apps. Results: Analyzing the fitness landscape of , we observed a lack of diversity of the evolved test suites and a stagnation of the search after 25 generations. realizes mechanisms that preserve the diversity of the test suites being evolved. The evaluation showed that achieves better or at least similar test results than concerning coverage and the number of revealed faults. However, typically produces longer test sequences and requires more execution time than . Conclusions: The understanding of the search problem obtained by the fitness landscape analysis helped us to find a more suitable configuration of without trial-and-error experiments. By promoting diversity of test suites during the search, improved or at least similar test results in terms of faults and coverage can be achieved.
READ FULL TEXT