On the Unusual Effectiveness of Type-aware Mutations for Testing SMT Solvers

04/19/2020
by   Dominik Winterer, et al.
0

We propose type-aware operator mutation, a simple, but unusually effective approach for testing SMT solvers. The key idea is to mutate operators of conforming types within the seed formulas to generate well-typed mutant formulas. These mutant formulas are then used as the test cases for SMT solvers. We realized type-aware operator mutation within the OpFuzz tool and used it to stress-test Z3 and CVC4, two state-of-the-art SMT solvers. Type-aware operator mutations are unusually effective: During nine months of extensive testing with OpFuzz, we reported 909 bugs in Z3 and CVC4, out of which 632 bugs were confirmed and 531 of the confirmed bugs were fixed by the developers. The detected bugs are highly diverse – we found bugs of many different types (soundness bugs, invalid model bugs, crashes, etc.), logics and solver configurations. We have further conducted an in-depth study on the bugs found by OpFuzz. The study results show that the bugs found by OpFuzz are of high quality. Many of them affect core components of the SMT solvers' codebases, and some required major changes for the developers to fix. Among the 909 bugs found by OpFuzz, 130 were soundness bugs, the most critical bugs in SMT solvers, and 501 were in the default modes of the solvers. Notably, OpFuzz found 16 critical soundness bugs in CVC4, which has proved to be a very stable SMT solver.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/07/2023

HornFuzz: Fuzzing CHC solvers

Many advanced program analysis and verification methods are based on sol...
research
04/13/2020

Detecting Critical Bugs in SMT Solvers Using Blackbox Mutational Fuzzing

Formal methods use SMT solvers extensively for deciding formula satisfia...
research
05/20/2021

Testing DBMS Performance with Mutations

Because database systems are the critical component of modern data-inten...
research
12/07/2020

Exposing Bugs in JavaScript Engines through Test Transplantation and Differential Testing

Context. JavaScript is a popular programming language today with several...
research
07/26/2022

Finding Deep-Learning Compilation Bugs with NNSmith

Deep-learning (DL) compilers such as TVM and TensorRT are increasingly u...
research
04/06/2021

A new perspective on the competent programmer hypothesis through the reproduction of bugs with repeated mutations

The competent programmer hypothesis states that most programmers are com...
research
06/13/2019

Astra Version 1.0: Evaluating Translations from Alloy to SMT-LIB

We present a variety of translation options for converting Alloy to SMT-...

Please sign up or login with your details

Forgot password? Click here to reset