Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defects

by   Steffen Herbold, et al.

Machine learning is nowadays a standard technique for data analysis within software applications. Software engineers need quality assurance techniques that are suitable for these new kinds of systems. Within this article, we discuss the question whether standard software testing techniques that have been part of textbooks since decades are also useful for the testing of machine learning software. Concretely, we try to determine generic smoke tests that can be used to assert that basic functions can be executed without crashing. We found that we can derive such tests using techniques similar to equivalence classes and boundary value analysis. Moreover, we found that these concepts can also be applied to hyperparameters, to further improve the quality of the smoke tests. Even though our approach is almost trivial, we were able to find bugs in all three machine learning libraries that we tested and severe bugs in two of the three libraries. This demonstrates that common software testing techniques are still valid in the age of machine learning and that they are suitable to find and prevent severe bugs, even in mature machine learning libraries.



There are no comments yet.


page 1

page 2

page 3

page 4


A Comprehensive Study on Deep Learning Bug Characteristics

Deep learning has gained substantial popularity in recent years. Develop...

Security testing using JUnit and Perl scripts

In this paper, I describe a recent practical experience where JUnit was ...

Nirikshak: An Autonomous Testing Framework

Quality Assurance (QA) is an important part of any product. But even wit...

Tutorials on Testing Neural Networks

Deep learning achieves remarkable performance on pattern recognition, bu...

DepOwl: Detecting Dependency Bugs to Prevent Compatibility Failures

Applications depend on libraries to avoid reinventing the wheel. Librari...

Machine Learning and Evolutionary Computing for GUI-based Regression Testing

ReTest is a novel testing tool for Java applications with a graphical us...

Boundary Value Exploration for Software Analysis

For software to be reliable and resilient, it is widely accepted that te...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.