Testing DBMS Performance with Mutations

by   Xinyu Liu, et al.

Because database systems are the critical component of modern data-intensive applications, it is important to ensure that they operate correctly. To this end, developers extensively test these systems to eliminate bugs that negatively affect functionality. In addition to functional bugs, however, there is another important class of bugs: performance bugs. These bugs negatively affect the response time of a database system and can therefore affect the overall performance of the system. Despite their impact on end-user experience, performance bugs have received considerably less attention than functional bugs. In this paper, we present AMOEBA, a system for automatically detecting performance bugs in database systems. The core idea behind AMOEBA is to construct query pairs that are semantically equivalent to each other and then compare their response time on the same database system. If the queries exhibit a significant difference in their runtime performance, then the root cause is likely a performance bug in the system. We propose a novel set of structure and predicate mutation rules for constructing query pairs that are likely to uncover performance bugs. We introduce feedback mechanisms for improving the efficacy and computational efficiency of the tool. We evaluate AMOEBA on two widely-used DBMSs, namely PostgreSQL and CockroachDB. AMOEBA has discovered 20 previously-unknown performance bugs, among which developers have already confirmed 14 and fixed 4.



There are no comments yet.


page 9

page 10


Security testing using JUnit and Perl scripts

In this paper, I describe a recent practical experience where JUnit was ...

Actor Database Systems: A Manifesto

Interactive data-intensive applications are becoming ever more pervasive...

Testing Database Engines via Pivoted Query Synthesis

Relational databases are used ubiquitously. They are managed by database...

Detecting Optimization Bugs in Database Engines via Non-Optimizing Reference Engine Construction

Database Management Systems (DBMS) are used ubiquitously. To efficiently...

On the Unusual Effectiveness of Type-aware Mutations for Testing SMT Solvers

We propose type-aware operator mutation, a simple, but unusually effecti...

Who is Debugging the Debuggers? Exposing Debug Information Bugs in Optimized Binaries

Despite the advancements in software testing, bugs still plague deployed...

SnapFuzz: An Efficient Fuzzing Framework for Network Applications

In recent years, fuzz testing has benefited from increased computational...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.