Accelerating Continuous Integration with Parallel Batch Testing

by   Emad Fallahzadeh, et al.

Continuous integration at scale is costly but essential to software development. Various test optimization techniques including test selection and prioritization aim to reduce the cost. Test batching is an effective alternative, but overlooked technique. This study evaluates parallelization's effect by adjusting machine count for test batching and introduces two novel approaches. We establish TestAll as a baseline to study the impact of parallelism and machine count on feedback time. We re-evaluate ConstantBatching and introduce DynamicBatching, which adapts batch size based on the remaining changes in the queue. We also propose TestCaseBatching, enabling new builds to join a batch before full test execution, thus speeding up continuous integration. Our evaluations utilize Ericsson's results and 276 million test outcomes from open-source Chrome, assessing feedback time, execution reduction, and providing access to Chrome project scripts and data. The results reveal a non-linear impact of test parallelization on feedback time, as each test delay compounds across the entire test queue. ConstantBatching, with a batch size of 4, utilizes up to 72 maintain the actual average feedback time and provides a constant execution reduction of up to 75 feedback time with up to 91 reduction of up to 99 feedback time with up to 81 reduction of up to 67 TestCaseBatching to reduce the required testing machines efficiently. Analyzing historical data to find the threshold where adding more machines has minimal impact on feedback time is also crucial for resource-effective testing.


page 1

page 2

page 3

page 4


DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing

Continuous integration testing is an important step in the modern softwa...

Comparative Study of Machine Learning Test Case Prioritization for Continuous Integration Testing

There is a growing body of research indicating the potential of machine ...

An approach for Test Impact Analysis on the Integration Level in Java programs

Test Impact Analysis is an approach to obtain a subset of tests impacted...

Towards Testing of Deep Learning Systems with Training Set Reduction

Testing the implementation of deep learning systems and their training r...

Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Continuous Integration (CI) requires efficient regression testing to ens...

Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics

Automated test execution scheduling is crucial in modern software develo...

An Integration Test Order Strategy to Consider Control Coupling

Integration testing is a very important step in software testing. Existi...

Please sign up or login with your details

Forgot password? Click here to reset