The Relevance of Classic Fuzz Testing: Have We Solved This One?

08/14/2020
by   Barton P. Miller, et al.
0

As fuzz testing has passed its 30th anniversary, and in the face of the incredible progress in fuzz testing techniques and tools, the question arises if the classic, basic fuzz technique is still useful and applicable? In that tradition, we have updated the basic fuzz tools and testing scripts and applied them to a large collection of Unix utilities on Linux, FreeBSD, and MacOS. As before, our failure criteria was whether the program crashed or hung. We found that 9 crash or hang out of 74 utilities on Linux, 15 out of 78 utilities on FreeBSD, and 12 out of 76 utilities on MacOS. A total of 24 different utilities failed across the three platforms. We note that these failure rates are somewhat higher than our in previous 1995, 2000, and 2006 studies of the reliability of command line utilities. In the basic fuzz tradition, we debugged each failed utility and categorized the causes the failures. Classic categories of failures, such as pointer and array errors and not checking return codes, were still broadly present in the current results. In addition, we found a couple of new categories of failures appearing. We present examples of these failures to illustrate the programming practices that allowed them to happen. As a side note, we tested the limited number of utilities available in a modern programming language (Rust) and found them to be of no better reliability than the standard ones.

READ FULL TEXT
research
12/23/2021

A Modeling Framework for Reliability of Erasure Codes in SSD Arrays

To help reliability of SSD arrays, Redundant Array of Independent Disks ...
research
02/02/2023

A novel failure indexing approach with run-time values of program variables

Failures with different root causes can disturb multi-fault localization...
research
03/10/2019

Does Unit-Tested Code Crash? A Case Study of Eclipse

Context: Software development projects increasingly adopt unit testing a...
research
02/18/2021

Learning Logic Programs by Explaining Failures

Scientists form hypotheses and experimentally test them. If a hypothesis...
research
08/03/2021

Towards Substructural Property-Based Testing

We propose to extend property-based testing to substructural logics to o...
research
05/02/2021

Assessing Exception Handling Testing Practices in Open-Source Libraries

Modern programming languages (e.g., Java and C#) provide features to sep...
research
07/31/2022

The Unnecessity of Assuming Statistically Independent Tests in Bayesian Software Reliability Assessments

When assessing a software-based system, the results of statistical infer...

Please sign up or login with your details

Forgot password? Click here to reset