What It Would Take to Use Mutation Testing in Industry–A Study at Facebook

by   Moritz Beller, et al.

Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants. At industrial systems the scale and size of Facebook's, doing this is infeasible. We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers. To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically. We combine the mutations with instrumentation that measures which tests exactly visited the mutated piece of code. Results on more than 15,000 generated mutants show that more than half of the generated mutants survive Facebook's rigorous test suite of unit, integration, and system tests. Moreover, in a case study with 26 developers, all but two found information of automatically detected test holes interesting in principle. As such, almost half of the 26 would actually act on the mutant presented to them by adapting an existing or creating a new test. The others did not for a variety of reasons often outside the scope of mutation testing. It remains a practical challenge how we can include such external information to increase the true actionability rate on mutants.


Extreme mutation testing in practice: An industrial case study

Mutation testing is used to evaluate the effectiveness of test suites. I...

Practical Mutation Testing at Scale

Mutation analysis assesses a test suite's adequacy by measuring its abil...

Does mutation testing improve testing practices?

Various proxy metrics for test quality have been defined in order to gui...

On the Use of Mutation in Injecting Test Order-Dependency

Background: Test flakiness is identified as a major issue that compromis...

FlakiMe: Laboratory-Controlled Test Flakiness Impact Assessment. A Case Study on Mutation Testing and Program Repair

Much research on software testing makes an implicit assumption that test...

Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?

Mutation testing is a means to assess the effectiveness of a test suite ...

Please sign up or login with your details

Forgot password? Click here to reset