RERS-Fuzz : Combining Greybox Fuzzing with Interval Analysis for error reachability in reactive softwares

by   Animesh Basak Chowdhury, et al.

Fuzz Testing is a well-studied area in the field of Software Maintenance and Evolution. In recent years, coverage-based Greybox fuzz testing has gained immense attention by discovering critical security level and show-stopper bugs in industrial grade software. Greybox fuzz-testing uses coverage maximization as objective function and achieve the same by employing feedback-driven evolutionary algorithms. In our work, we have utilized the power of Greybox fuzz testing, combined with interval analysis for solving reachability problem in sequential and industrial RERS (Rigorous Examination of Reactive Software) 2019 benchmarks.



page 1

page 2

page 3

page 4


Security testing using JUnit and Perl scripts

In this paper, I describe a recent practical experience where JUnit was ...

Industry Practice of Coverage-Guided Enterprise-Level DBMS Fuzzing

As an infrastructure for data persistence and analysis, Database Managem...

Automatically Checking Conformance on Asynchronous Reactive Systems

Software testing is an important issue in software development process t...

Program State Abstraction for Feedback-Driven Fuzz Testing using Likely Invariants

Fuzz testing proved its great effectiveness in finding software bugs in ...

Does the performance of TDD hold across software companies and premises? A group of industrial experiments on TDD

Test-Driven Development (TDD) has been claimed to increase external soft...

FuzzSplore: Visualizing Feedback-Driven Fuzzing Techniques

Fuzz Testing techniques are the state of the art in software testing for...

Efficient Binary-Level Coverage Analysis

Coverage analysis plays an important role in the software testing proces...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

RERS-Fuzz is an automated test-generation tool targeted for coverage maximization in reactive softwares. The techniques employed are heavily based on key concepts of Greybox fuzz-testing [6]. The underlying idea is to employ evolutionary algorithms for generating interesting test-inputs that help in exploration of newer code segments. In Greybox fuzz-testing, a set of random test-inputs are considered as initial population. The fitness function is the code coverage measure observed by an individual test-input. For every test-input, fitness value is calculated and best-fit test-inputs are retained for further fuzzing. The retained test-inputs are subjected to various mutation and crossover operations for generation of newer test-inputs. The cycle repeats until certain user-defined goals are met or statistical measures on coverage metrics are achieved.

We have mainly used the following tools and techniques and modified them inline with our requirements to solve reachability problems on RERS 2019 benchmarks :-

  • American Fuzzy Lop (AFL v2.52b) [1].

  • Instrim [2].

  • VeriFuzz [3]

  • LLVM Interval Analysis [4].

In subsequent sections, we briefly describe about our approach, highlighting core techniques, tool architecture and strengths and weaknesses associated with them.

2 Approach

State-of-art greybox fuzzers, like afl-fuzz uses a stronger notion of structural code coverage of program’s control-flow-graph (CFG), called branch-pair coverage as fitness function. The tool maintains a shared 64kB memory where each byte entry represents a logarithmic visit count of a typical branch-pair by all test-inputs generated so far. The amount of shared memory is designed in such a way that entire shared memory can be stored in cache-memory and hence execution speed of fuzz-testing tool remains unhampered. However, as per our observation, the number of conditional statements in RERS benchmarks are typically very high. Hence, instrumenting each conditional statement of program would lead to heavy collision [5]. Secondly, the ranges of input values driving the reactive softwares is comparatively small. Hence, application of heavy-weight and sophisticated mutation and crossover operations would amount to generation of useless test-inputs rejected by the reactive system. Finally, in reactive softwares, code-coverage does not necessarily ensure that newer states are also covered.

In our tool, we have tried of identify the problems where plain usage of greybox fuzz testing tool would fail. We have added our own techniques on top of core-fuzzing engine to suitably tune the application of evolutionary fuzz testing on RERS benchmarks. The following are the key techniques of our approach.

Minimal Node Instrumentation: As discussed earlier, the software-under-test (SUT) is subjected to node instrumentation for measuring code coverage during a test-run. However, in RERS benchmarks the number of conditional statements are significantly high and complex. In order to reduce cache collision and execution overhead arising from instrumentation, number of instrumentation points have to be reduced. In our tool, we have used lightweight instrumentation [2] which is efficient and instruments minimal conditional statements of SUT, without loosing any coverage information. The technique have posed the minimal instrumentation as path differentiation problem and identify minimal number of nodes in CFG for which any two branch-pairs can be differentiated.

Interval Analysis assisted Mutation: Reactive softwares belong to class of program where input space is very restricted [3]. It is possible to tune mutation and crossover operation within the ranges of input taken by  SUT. We have taken help of LLVM’s interval analysis [4] to identify ranges of input consumed by RERS to go into deeper program segments. The values are then passed on to our parametric fuzz engine, which in turn does crossover and mutation operation over those restricted ranges.

State Instrumentation: The minimal node instrumentation takes care of efficient branch-pair coverage in program’s CFG. But, it may happen that a test-input have exhibited no new branch-pair coverage in its run, but it have explored a new state. We define state as, unique values assignments to all the global variables in SUT. In addition to existing 64k shared memory for branch-pair instrumentation, we have additionally deployed 64k shared memory, which maintains a binary valued entry whether a state has been explored or not. Accordingly, our fitness function is also updated with state coverage information. Any test-input with no new branch-pair coverage but new state space coverage are now retained for further fuzzing.

3 Tool Implementation

Figure 1: RERS-Fuzz : Overview

The overall flow of RERS-Fuzz has been shown in figure 1. The fuzzing module has been developed on top greybox fuzzer afl-fuzz [1]. The green blocks in the fuzzing module denotes that techniques have been modified and tuned with RERS programs. Red blocks denote the algorithm is unchanged and in line with afl-fuzz core algorithm. We have additionally used Instrim111 package for minimal node instrumentation on top of LLVM-7 framework. For Interval Analysis module, we used the LLVM package developed for interval analysis222 The tool has been developed using C, C++ and python.

4 Strengths and Weakness

RERS-Fuzz participated in Reachability Track of RERS 2019 competition. The tool could find out counterexamples for 217 academic reachability benchmarks. In industrial reachability track, it has emerged as overall winner, scoring 2038 points.

The core strength of RERS-Fuzz is its capability to scale up on large industrial benchmarks and find sufficient number of error reachable locations. We have run all sequential and industrial reachability benchmarks for a timeline of 8 hours on Intel i7 3720 Octa-Core Processor (2.6GHz) and presented our results. The entry for the specification with error_reachable denotes that error location is reachable. Entries with UNKNOWN denote we are unable to conclude whether the error location is reachable or not.

Weaknesses involve the inability of the tool to generate proofs for unreachability of error location. Besides that, we have observed that running certain benchmarks for a longer time period can yield more number of error reachable locations. However there are no assurances that preserving test-inputs with newer state-space or branch-pair would definitely aid to uncover error-reachable locations during evolution. With our technique, we try to maximize a stronger notion code-coverage and state-space coverage of RERS

benchmarks. Today, our tool lacks focused search towards reachable error location. We believe that in future, such problems can be modelled as appropriate fitness function and hence fuzzing can be made directed. Also, such high quality test-vectors would definitely aid various invariant learning algorithms, which in turn would help prove correctness of such reactive software.


  • [1] Zaleskwi, M.: American Fuzzy Lop
  • [2] Hsu, C. C., Wu, C. Y., Hsiao, H. C., & Huang, S. K. (2018). Instrim: Lightweight instrumentation for coverage-guided fuzzing. In Symposium on Network and Distributed System Security (NDSS), Workshop on Binary Analysis Research.
  • [3] Chowdhury, A.B., Medicherla, R.K., & Venkatesh, R. (2019) VeriFuzz : Program Aware Fuzzing. In TACAS 2019, Part III, LNCS 11429, pp. 1–6, 2019.
  • [4] LLVM Compiler Framework (Interval Analysis)
  • [5] Zaleskwi, M.: Technical Whitepaper of American Fuzzy Lop
  • [6] DeMott, J., Enbody, R., Punch, W.F.: Revolutionizing the field of grey-box attack surface testing with evolutionary fuzzing. BlackHat and Defcon (2007)