Universal Fuzzing via Large Language Models

by   Chunqiu Steven Xia, et al.

Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4All, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4All is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are wellsuited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4All on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4All has identified 76 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 47 bugs already confirmed by developers as previously unknown.


Why Aren't Regular Expressions a Lingua Franca? An Empirical Study on the Re-use and Portability of Regular Expressions

This paper explores the extent to which regular expressions (regexes) ar...

Exposing Bugs in JavaScript Engines through Test Transplantation and Differential Testing

Context. JavaScript is a popular programming language today with several...

Large Language Models of Code Fail at Completing Code with Potential Bugs

Large language models of code (Code-LLMs) have recently brought tremendo...

K-CONFIG: Using Failing Test Cases to Generate Test Cases in GCC Compilers

The correctness of compilers is instrumental in the safety and reliabili...

MTFuzz: Fuzzing with a Multi-Task Neural Network

Fuzzing is a widely used technique for detecting software bugs and vulne...

Controlled Update of Software Components using Concurrent Exection of Patched and Unpatched Versions

Software patching is a common method of removing vulnerabilities in soft...

Scoped Extension Methods in Dynamically-Typed Languages

Context. An extension method is a method declared in a package other tha...

Please sign up or login with your details

Forgot password? Click here to reset