TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

07/28/2018
by   Augustus Odena, et al.
4

Machine learning models are notoriously difficult to interpret and debug. This is particularly true of neural networks. In this work, we introduce automated software testing techniques for neural networks that are well-suited to discovering errors which occur only for rare inputs. Specifically, we develop coverage-guided fuzzing (CGF) methods for neural networks. In CGF, random mutations of inputs to a neural network are guided by a coverage metric toward the goal of satisfying user-specified constraints. We describe how fast approximate nearest neighbor algorithms can provide this coverage metric. We then discuss the application of CGF to the following goals: finding numerical errors in trained neural networks, generating disagreements between neural networks and quantized versions of those networks, and surfacing undesirable behavior in character level language models. Finally, we release an open source library called TensorFuzz that implements the described techniques.

READ FULL TEXT

page 2

page 7

page 8

research
02/17/2019

Towards Improved Testing For Deep Learning

The growing use of deep neural networks in safety-critical applications ...
research
11/13/2020

diagNNose: A Library for Neural Activation Analysis

In this paper we introduce diagNNose, an open source library for analysi...
research
06/20/2019

testRNN: Coverage-guided Testing on Recurrent Neural Networks

Recurrent neural networks (RNNs) have been widely applied to various seq...
research
09/02/2023

Pressmatch: Automated journalist recommendation for media coverage with Nearest Neighbor search

Slating a product for release often involves pitching journalists to run...
research
04/30/2018

Concolic Testing for Deep Neural Networks

Concolic testing alternates between CONCrete program execution and symbO...
research
08/03/2021

Tutorials on Testing Neural Networks

Deep learning achieves remarkable performance on pattern recognition, bu...
research
07/01/2021

The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models

Supervised learning models often make systematic errors on rare subsets ...

Please sign up or login with your details

Forgot password? Click here to reset