Predictive Test Selection

10/11/2018

∙

by   Mateusz Machalica, et al.

∙

0

∙

share

Change-based testing is a key component of continuous integration at Facebook. However, a large number of tests coupled with a high rate of changes committed to our monolithic repository make it infeasible to run all potentially-impacted tests on each change. We propose a new predictive test selection strategy which selects a subset of tests to exercise for each change submitted to the continuous integration system. The strategy is learned from a large dataset of historical test outcomes using basic machine learning techniques. Deployed in production, the strategy reduces the total infrastructure cost of testing code changes by a factor of two, while guaranteeing that over 95 changes are still reported back to developers. The method we present here also accounts for the non-determinism of test outcomes, also known as test flakiness.