What It Would Take to Use Mutation Testing in Industry–A Study at Facebook

10/26/2020
by   Moritz Beller, et al.
0

Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook's, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook's rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two found information of automatically detected test holes interesting in principle. As such, almost half of the 26 would actually act on the mutant presented to them by adapting an existing or creating a new test. The others did not for a variety of reasons often outside the scope of mutation testing. It remains a practical challenge how we can include such external information to increase the true actionability rate on mutants.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 5

03/15/2021

Extreme mutation testing in practice: An industrial case study

Mutation testing is used to evaluate the effectiveness of test suites. I...
02/22/2021

Practical Mutation Testing at Scale

Mutation analysis assesses a test suite's adequacy by measuring its abil...
03/12/2021

Does mutation testing improve testing practices?

Various proxy metrics for test quality have been defined in order to gui...
04/15/2021

On the Use of Mutation in Injecting Test Order-Dependency

Background: Test flakiness is identified as a major issue that compromis...
12/06/2019

FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair

Much research on software testing makes an implicit assumption that test...
03/13/2019

Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?

Mutation testing is a means to assess the effectiveness of a test suite ...
12/28/2021

Cerebro: Static Subsuming Mutant Selection

Mutation testing research has indicated that a major part of its applica...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.