Empirical Evaluation of Mutation-based Test Prioritization Techniques
share
We propose and empirically investigate a new test case prioritization technique that combines both mutation-based and diversity-based approaches. Our diversity-aware mutation-based technique relies on the notion of mutant distinguishment, which aims to distinguish one mutant's behavior from another, rather than from the original program. We empirically investigate the relative cost and effectiveness of the mutation-based prioritization techniques (i.e., using both the traditional mutant kill and the proposed mutant distinguishment) with 352 real faults and 553,477 developer-written test cases. The empirical evaluation considers both the kill-only and the diversity-aware mutation criteria in various settings: single-objective greedy, single-objective hybrid, as well as multi-objective that seeks to prioritize using both criteria simultaneously. The results show that there is no single dominant technique across all the studied faults. To this end, we use visual aids to study and infer the reasons explaining why and when each one of the mutation-based prioritization criteria performs poorly.
READ FULL TEXT