Mutation Testing for Ethereum Smart Contract

08/10/2019 ∙ by Haoran Wu, et al. ∙ Nanjing University 0

Smart contract is a special program that manages digital assets on blockchain. It is difficult to recover the loss if users make transactions through buggy smart contracts, which cannot be directly fixed. Hence, it is important to ensure the correctness of smart contracts before deploying them. This paper proposes a systematic framework to mutation testing for smart contracts on Ethereum, which is currently the most popular open blockchain for deploying and running smart contracts. Fifteen novel mutation operators have been designed for Ethereum Smart Contracts (ESC), in terms of keyword, global variable/function, variable unit, and error handling. An empirical study on 26 smart contracts in four Ethereum DApps has been conducted to evaluate the effectiveness of mutation testing. The experimental results show that our approach can outperform the coverage-based approach on defect detection rate (96.01 defects and we found 117 out of 729 real bug reports are related to our operators. These show the great potential of using mutation testing for quality assurance of ESC.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Blockchain, the foundation of Bitcoin, has been gaining increasing attention from both industry and research community [63]. As an immutable ledger, the blockchain technology allows transactions to take place in a decentralized manner [34]. Many blockchain-based applications beyond cryptocurrencies like Bitcoin have been proposed. An emerging area of blockchain technology is smart contract [28]. A smart contract is a special program (running on blockchains) that controls users’ digital assets. It could be automatically and correctly executed by a network of mutually distrusting nodes without the need of an external trusted authority [63]. Smart contract has increasingly been finding numerous important applications (e.g., crowdfunding, voting, gaming, etc) in the real world [6]. More and more developers are devoting themselves to developing various kinds of smart contracts. For example, there have been as many as 1 million smart contracts being deployed and running on Ethereum – the most popular open blockchain platform for deploying and running smart contracts [61].

Despite the increasing popularity of smart contracts applications in practice, the quality of smart contracts was reported to be worrisome. According to Nikoli et al. study, among the sampled 3,759 smart contracts, 3,686 smart contracts had an 89% probability of containing vulnerabilities

[35]. Any potential vulnerability within a smart contract may lead to a significant financial loss. For example, hackers stole as much as $150 million from Ethereum by exploiting a loophole within the DAO contract [4]. On the other hand, due to the nature of blockchain, any transactions through smart contacts could not be reverted. Which means if users experience loss on blockchain, they cannot recover their losses. In addition, the code of smart contract is immutable once deployed onto blockchain. All these called for effective approaches to guard/evaluate the quality of smart contracts in practice.

Compared with conventional software systems, the environment for developing ESC is still immature, 16 and most developers are young and lack of experience. Nikoli et al. sampled 3,759 smart contracts, of which 3,686 smart contracts had a 89% probability of containing vulnerabilities [35], indicating that developing a high-quality ESC is full of challenges. As stated, a small defect might result in extremely serious consequences. Meanwhile, testing is widely recognized as one of the most important means of ensuring software quality [39]. Thus, the Smart Contract Under Deployment (SCUD) must be given adequate testing, regardless of cost, aiming to detect defects as more as possible. Subsequently, the following question should be answered, that is, how to evaluate the adequacy of ESC test suites? Mutation testing provides a costly but effective approach for test adequacy evaluation [13]. It achieves this by checking the ability of the given tests to reveal some artificial defects [40]. Currently, mutation testing has been applied on a series of programming languages such as C [8], C++ [12], Java [42] and JavaScript [31], and has also been adapted for a set of programming paradigms such as Object-Oriented [29], Functional [26] and Aspect-Oriented [38] programming. Therefore, it is desirable to employ mutation testing for evaluating ESC test adequacy.

Testing approaches were often used to guard/evaluate the quality of software programs. As an effective testing approach, mutation testing was commonly used within traditional software programs [8, 12, 42]. However, the potential of mutation testing has not been explored in the context of smart contracts. Towards this end, we proposed to apply mutation testing on Ethereum Smart Contract (ESC). Specifically, we designed a systematical framework for mutating Ethereum smart contract. Our framework included four parts. First, for a given Smart Contract Under Test (SCUT), we firstly transformed it into an Abstract Syntax Tree (AST) and performed mutation on the AST. Then, the mutated ASTs were transformed back to their corresponding source code version mutants. Next, we built a testnet for each mutant, during which the testnets was initialized with the same state and contracts that SCUT depended on or relied on SCUT were deployed in a specific order. After that, we executed each mutant, recorded their execution results on the given test suite, and collected the mutation scores as the final evaluation results.

An important part of mutation testing is to define mutation operators, as mutation operators defined how to introduce syntactic changes to the original programs [40]. The goal of mutation operators is to simulate the potential threats as completely as possible, and different operators are generally needed if different programming languages are adopted to develop programs [40]. Unlike traditional programs that were developed in general-purpose programming languages such as Java and Python, smart contracts were developed in specific-purpose languages such as Solidity or Vyper [45]. To adapt to new programming languages of smart contracts, we designed 15 novel mutation operators (in addition to 10 existing mutation operators) in terms of keyword, global variable/function, variable unit, and error handling, to effectively mutate smart contracts.

Twenty-six smart contracts from four Ethereum Decentralized Applications (DApps) were used to evaluate the effectiveness of our proposed mutation testing framework. We found that our approach could outperform the coverage-based approach on defect detection rate (96.01% vs. 55.68%). And the newly-proposed 15 mutation operators were effective to reveal real defects, where 117 real bug reports were related to our 15 mutation operators. These results to a great extent showed the great potential of using mutation testing to guard the quality of Ethereum smart contract.

Our main contributions are as follows:

  • Idea. We introduce the idea of applying mutation testing to ensure the quality of ESC.

  • Implementation. We implement the first ESC-oriented mutation testing system, and design 15 novel mutation operators to simulate the potential defects in ESC.

  • Study. We carried out an empirical study on four real-world DApps. The study results verified that mutation testing could indeed help improve ESC testing evaluation, and the novel operators were able to simulate real ESC defects well.

The rest of this study is organized as follows. Section II describes the basics of Ethereum smart contract and mutation testing. Our mutation testing approach is presented in detail in Section III, and each novel mutation operator is detailed in Section IV. Experimental design and results analysis are presented in Section V. After that, we discuss the threats to validity and give the related work in Section VI and VII, respectively. Finally, Section VIII concludes the paper and outlines directions for future research.

2 Background

2.1 Ethereum Smart Contract

Smart contract encapsulates predefined state and transition rules, scenarios that trigger contract execution (e.g. arrival at a specific time or occurrence of a specific event), and response actions under a specific scenario. It was first proposed by Nick Szabo, who aims to flexibly create and manage intellectual assets in a decentralized environment [56]. Blockchain provides the first feasible decentralized technique, making it possible to automatically and correctly execute smart contracts without relying on a trusted authority [63]. Currently, many mainstream blockchain systems (e.g., BitCoin and Ethereum) have supported smart contract technique. With its Turing-complete programming languages and a series of sophisticated building tools, Ethereum has become a reasonably attractive blockchain platform for constructing various kinds of Decentralized Applications (Dapp).

Fig. 1: ESC Deployment and Execution

Ethereum users rely on accounts to carry out transactions. There are two types of accounts, External Account (EA) and Contract Account (CA), in Ethereum. The former is controlled by the users with private keys, and the latter is controlled by ESC code [7]. Both EA and CA record the state information such as account balance, and CA has an additional unchanged field, codeHash, which points to the corresponding ESC. CA creation and its corresponding ESC execution are transaction-driven. An Ethereum transaction can be denoted by a quadruple , where and denote the addresses of sender’s account and receiving account respectively, and denotes the number of Wei to be transferred to . is an CA creation transaction when . At this time, denotes SCUD. Otherwise (i.e., ), if is a CA, denotes the input to ’s corresponding ESC code. ESC deployment can be deemed as the process of CA creation. Fig.1 depicts the ways of deploying as well as executing the ESC .

  • ESC Deployment. EA launches a CA creation transaction . Once has been packed into a block and the block has been persisted in Ethereum, CA creation as well as deployment are finished.

  • ESC Execution. EA launches a message call transaction . Once has been packed into a block and the block has been persisted in Ethereum, takes as input and starts to run. Executing results in two additional transactions, and .

Ethereum provides four smart contract programming languages (i.e., Solidity, Vyper, Bamboo and Flint), where each of them is influenced by one or several popular languages [45]. For example, Solidity is mainly influenced by JavaScript. Among them, Solidity is the most popular language on Ethereum. Different from JavaScript, which is often used in developing Web applications and runs in the Web browser, Solidity is specific to ESC and runs in the Ethereum Virtual Machine (EVM). Thus, though JavaScript and Solidity share a similar syntax, they still have a number of differences, which are detailed as follows:

Unit Type Suffix Description
Ether wei, finney, szabo, ether 1 szabo = 1e12 wei, 1 finney = 1e15 wei, 1 ether = 1e18 wei
Time seconds, minutes, hours, days, weeks 1 minutes = 60 seconds, 1 hours = 60 minutes, 1 days = 24 hours, 1 weeks = 7 days
TABLE I: Specific Variable Units in Solidity
Specifier Type Keyword Description
function visibility public accessed by all
external accessed by using this and externally
internal accessed by this and derived contracts
private accessed by only this contract
function state pure not view and modify state
view not modify state
payable receive ethers
reference location memory lifetime is limited to a function call
storage persisted into the blockchain itself
calldata required for params of external functions
lvalue delete assigns the initial value for its type
TABLE II: Specific Keywords in Solidity
  • Variable Type. Different from JS, Solidity is a statically and strongly typed language, indicating that the variable’s type is specified during compilation (executing for JavaScript) and implicit type conversion is not allowed. Moreover, for ease of handling blockchain data, some novel variable types (e.g., address) are added in Solidity.

  • Variable Unit. Ethereum employs an intrinsic currency, Ether, to incentive computation within the network [62]. Then, as Table I depicts, it defines some sub-denominations of Ether. It also defines some time units.

  • Keyword. As Table II depicts, Solidity defines a set of novel keywords in terms of function visibility, function state, reference location, and lvalue to optimize the storage and reduce the gas cost of ESC.

  • Global Variable and Global Function. For ease of accessing blockchain data (e.g., block number, difficulty and gasLimit) and carrying out mathematical calculations (e.g., addmod, mulmod and keccak256), Solidity provides a set of global variables and global functions.

  • Error Handling. In addition to assert functions, Solidity provides require() for handling errors. ESC continues when is satisfied and otherwise stops. Function require is used for variable checking, while assert is mainly used to detect unknown errors.

2.2 Mutation Testing

Mutation testing is a fault-based software testing technique which can be used to evaluate the adequacy of test cases. These so-called mutants are based on well-defined mutated operators that either simulate typical application errors or force a validity test. The goal is to help testers identify limitations in the testing process or test suite.

When applying mutation testing, testers first design mutation operators according to the characteristics of the program under test. Generally, mutation operators only make minor changes to the program under test in accordance with the grammar. Applying mutation operators to the program under test can generate a large number of defective programs called mutants. Then the equivalent mutants (i.e., the mutants that have no effect on the execution result of the program) are excluded. Next, the original program and each mutant are tested with a given test suite. If the test result of a mutant is different from the original program, it is said to be killed. This indicates that existing test cases can detect the defect. If the test result of a mutant is the same as that of the original program, it is said to be survived, and new test cases need to be designed to kill it. Finally, mutation testing calculates the mutation score according to the following formula: . The higher the mutation score, the higher the adequacy of test cases.

Mutation testing has been shown to subsume other test criteria by incorporating appropriate mutation operators [60, 21, 5, 25]. Thus, designing effective mutation operators is one of the most important tasks when applying mutation testing to a new field[15]. In this paper, we first summarize a set of general mutation operators that can be used in Solidity based on the existing mutation operators for JavaScript [31], and then propose several specific mutation operators according to the characteristics of Solidity.

Operator Description
AOR Arithmetic Operator Replacement
AOI Arithmetic Operator Insertion
ROR Relational Operator Replacement
COR Conditional Operator Replacement
LOR Logical Operator Replacement
ASR Assignment Operator Replacement
SDL Statement Deletion
RVR Return Value Replacement
CSC Condition Statement Change
TABLE III: General Mutation Operators

Table III shows the general mutation operators. AOR is divided into two kinds of mutation operators, AOR (Binary Arithmetic Operator Replacement) and AOR (Short-cut Arithmetic Operator Replacement). The AOR operator replaces basic binary arithmetic operators (, , , , and ) with other binary arithmetic operators while the AOR operator replaces short-cut arithmetic operators (op, op, op, and op). The ROR operator replaces relational operators (, , , , , and ) with other relational operators or replace the entire predicate with true and false. The COR operator replaces binary conditional operators (, , , ) with other binary conditional operators. The ASR operator replaces short-cut assignment operators (, , , , and ) with other short-cut assignment operators. The SDL operator can delete an executable statement by commenting it out, and the CSC operator can force the statement in a conditional judgment to be true or false.

3 Mutating Smart Contracts

Fig. 2: Performing Mutation Analysis on Smart Contracts

Traditional JavaScript mutation engine usually parses the source file to an AST (Abstract Syntax Tree), and then performs mutant generation on it [32]. The generated AST mutants are then transformed back into source files for execution and testing. However, ESC mutation analysis cannot be performed the same way as for JavaScript programs. First, smart contracts must be deployed on a blockchain before being executed. Second, different from JavaScript, Solidity is not a scripting language. Thus, ESC must be compiled before being deployed. This significantly influences the process of ESC mutation analysis. Fig. 2 illustrates how our ESC mutation analysis engine works. It treats the Smart Contract Under Test (SCUT) and a passed Test Suite (TS), which contains tests, as inputs, and finally outputs the mutation score. Below are the steps of mutation analysis on ESC. Note that steps (2) and (4) are different from the traditional JavaScript mutation testing process.

  • AST Generation. SCUT is parsed into AST format for alleviating the loss of mutation precision, which may cause by the redundant statements, such as annotations and empty lines. To ensure the reliability of parsing result, solidity-parser-antlr [53], a Solidity parser built on top of a robust ANTLR4 grammar, is selected in this step.

  • AST Mutation. The mutation is performed on the AST formatted SCUT, creating a new copy of the file for each AST mutant. In addition to thirteen new ESC mutation operators, which defined in 4, we also reuse nine JavaScript-oriented operators [32] for ESC mutation.

  • Mutant Generation and Filtration. Each AST mutants will be transformed into a source file version, which has the identical semantic with SCUT excepting has been injected with a pre-defined fault. Subsequently, an equivalence checking is conducted to filter out the equivalent mutants.

  • Compilation and Deployment. ESC works on the Ethereum blockchain, and its execution result depends on the blockchain state, such as account balance and block height. Thus, to avoid the influence of blockchain state, for either SCUT or each source file version mutant, we should build a unique Ethereum testnet before its being compiled and deployed. Mutants that cause compilation errors are discarded immediately and not used in the following testing.

  • SCUT and Mutants Testing. Once an ESC has been successfully deployed, tests in TS can be executed. For each mutant, we record its execution results (i.e., passed or failed) in every test. Given a mutant and a test , is marked as have been killed by if it failed on .

  • Mutation Score Calculation. After collecting all the execution results, we can get a matrix of execution results, where depicts the execution result of mutant on test . Finally, the mutation score is computed as a percentage of the mutants killed by the tests to the number of non-equivalent mutants.

4 Smart Contract Mutation Operators

To improve the effectiveness of mutation testing for smart contracts, we defined a new set of mutation operators for Ethereum smart contract. We read the Solidity documentation and look over the issues related to smart contracts on GitHub and Stack Exchange to design the mutation operators for the specific defects that may be generated by smart contracts. Details are as follows:

4.1 Keyword Operators

There are several keywords in Solidity that cannot be found in JavaScript. Six mutation operators are designed for these keywords.

4.1.1 Function State Keyword Change (FSC)

Functions can be declarable as view, which means that function will not modify the state of the contract, i.e. not modify variables, not emit events, etc. Functions can also be declarable as pure, which means that function can neither read from nor modify the state. Pure functions can only call other pure functions [23].

FSC operator change the state of a function by replace keyword view to pure. Table IV shows an example FSC mutant, if such mutant survives, it means you should use pure instead of view (ESE111Ethereum Stack Exchange (ESE):

s function func(uint x, uint y) view returns (uint){
s  return x * (y + 42);
s }
s function func(uint x, uint y) pure returns (uint){
s  return x * (y + 42);
s }
TABLE IV: Example of FSC Mutant

4.1.2 Function Visibility Keyword Change (FVC)

Since Solidity has two kinds of function calls (internal ones that do not create an actual EVM call and external ones that do), there are four types of visibility for functions and state variables. external functions are part of the contract interface, which means they can be called from other contracts and via transactions. public functions are part of the contract interface and can be either called internally or via messages. internal functions can only be accessed from this contract and contracts deriving from it. private functions can only be visible for the contract where they are defined (ESE#32353). It is important to note that incorrect use of visibility keywords does not always lead to errors initially, but it can lead to faulty behavior when the contract is integrated with other classes, modified, or inherited from [55].

We design the FVC operator by imitating the Access Modifier Change operator in MuJava [30]. Table V shows an example FVC mutant where the visibility keyword internal is changed into private, which makes it impossible to call function setData() from C’s derived contracts.

s contract C{
s  function f(uint a) private pure returns (uint b){}
s function setData(uint a) internal{data = a;}
s  uint public data;
s }
s contract C{
s  function f(uint a) private pure returns (uint b){}
s function setData(uint a) private{data = a;}
s  uint public data;
s }
TABLE V: Example of FVC Mutant

4.1.3 Data Location Keyword Replacement (DLR)

Every reference type, i.e. arrays and structs, has an additional annotation in Solidity, the “data location”, about where it is stored. There are three data locations: memory (whose lifetime is limited to a function call), storage (the location where the state variables are stored and is persistent between function calls) and calldata (special data location that contains the function arguments, only available for external function call parameters) [48].

Data locations are not only relevant for the persistence of data, but also for the semantics of assignments. For example, Assignments between storage and memory (or from calldata) always create an independent copy while Assignments from memory to memory only create references (ESE#1231). Uninitialized local storage variables can point to unexpected storage locations in the contract, which can lead to intentional or unintentional vulnerabilities [58]. Table VI shows an example DLR mutant where the data location of mySandwich is replaced from storage to memory. This mutant turns mySandwich from a reference into a copy. DLR mutants challenge testers to design test cases to check whether the original variables have changed.

s function eatSandwich(uint _index) public{
s Sandwich storage mySandwich = sandwiches[_index];
s  mySandwich.status = “Eaten!”;
s }
s function eatSandwich(uint _index) public{
s Sandwich memory mySandwich = sandwiches[_index];
s  mySandwich.status = “Eaten!”;
s }
TABLE VI: Example of DLR Mutant

4.1.4 Variable Type Keyword Replacement (VTR)

Since Solidity is a statically and strongly typed language, the type of each variable needs to be specified. Solidity supports three fixed-size types, fixed-size integers, fixed-point numbers and fixed-size byte arrays. All these types are restricted to a certain range. For example, an 8-bit unsigned integer can store values between 0 and 255 (2-1). When the result of some arithmetic falls outside that supported range, an overflow occurs (ESE#7293).

Integers in Solidity are divided into signed integers and unsigned integers of various sizes. The consequence of an integer overflow is that the most significant bits of the result are lost, which can cause real-world vulnerabilities such as batchOverflow[3]. Fixed point numbers and fixed-size byte arrays are no difference. To ensure that the correct value keywords are used, we define a series of VTR operators such as replace uint to int and replace bytes32 to bytes8. Table VII shows an example of VTR mutant. This kind of mutants requires testers to consider negative number and truncation.

s for (uint256 i=0; i¡length; i++){
s  A.push(B[i]);
s }
s for (uint8 i=0; i¡length; i++){
s  A.push(B[i]);
s }
TABLE VII: Example of VTR Mutant

4.1.5 Payable Keyword Deletion (PKD)

payable is a modifier that allows a function to be called with a non-zero value (that you can access via msg.value). If a function needs currency operation, it must have a payable keyword, so that it can receive Ethernet currency normally (ESE#20847). The PKD operator deletes the payable keyword of a function at a time, and then any transaction trying to send ether will be rejected. The sample program is shown in Table VIII.

s function deposit() payable{
s  deposits[msg.sender] += msg.value;
s }
s function deposit(){
s  deposits[msg.sender] += msg.value;
s }
TABLE VIII: Example of PKD Mutant

4.1.6 Delete Keyword Deletion (DKD)

In Solidity, delete a assigns the initial value for the type to a. For integers, it is equivalent to a = 0, but it can also be used on arrays, where it assigns a dynamic array of length zero or a static array of the same length with all elements set to their initial values (ESE#58495). It is important to note that deleting a really behaves like an assignment to a, i.e. it stores a new object in a. Table IX shows an example of DKD mutant which makes the return value change from 0 to 3. The DKD operator is similar to the Member Variable Initialization Deletion (JID) in MuJava [30].

s function testDel() returns (uint){
s  uint a = 3;
s delete a;
s  return a;
s }
s function testDel() returns (uint){
s  uint a = 3;
s //delete a;
s  return a;
s }
TABLE IX: Example of DKD Mutant

4.2 Global Variables and Functions Operators

There are some special variables and functions which always exist in the global namespace [54]. Thus, we define three mutation operators for them.

4.2.1 Global Variable Change (GVC)

Solidity uses several global variables to provide information about the blockchain (ESE#2664). For example, now is the current block timestamp (alias for block.timestamp), block.number shows the current block number, and msg.value shows the number of wei sent to the contract. A GVC operator changes a global variable by assigning it a format-compliant random value, and thus may cause different execution results. The sample program is shown in Table X, testers can kill the mutant by executing function getNow().

s function getNow() public constant returns (uint){
s return now;
s }
s function getNow() public constant returns (uint){
s return 0;
s }
TABLE X: Example of GVC Mutant

4.2.2 Mathematical Functions Replacement (MFR)

addmod and mulmod are two mathematical global functions in Solidity. addmod(uint x, uint y, uint k) computes where the addition is performed with arbitrary precision and does not wrap around at 2**256. Similarly, mulmod(uint x, uint y, uint k) computes [51]. We design MFR operators to exchange them to keep developers from accidentally using incorrect functions. Table XI shows an example of MFR mutant, testers can kill the mutant by testing the return value of the function.

s function Test(uint x, uint y, uint k) view public returns (uint){
s return addmod(x, y, k);
s }
s function Test(uint x, uint y, uint k) view public returns (uint){
s return mulmod(x, y, k);
s }
TABLE XI: Example of MFR Mutant

4.2.3 Address Variable Replacement (AVR)

There are three global variables related to address in Solidity. block.coinbase is the current block miner’s address, msg.sender is the sender of the message (current call), and tx.origin is the sender of the transaction (full call chain). The different between msg.sender and tx.origin is that msg.sender can be a contract but tx.origin can never be a contract. In a simple call chain, , msg.sender in D will be C, and tx.origin will always be A (ESE#1891).

An AVR operator replaces an address variable with another one. For example, in Table XII, msg.sender is replaced with tx.origin. To kill this mutant, a test case needs to be designed where msg.sender is not equal to tx.origin.

s function sendTo(address receiver, uint amount) public{
s require(msg.sender == owner);
s  receiver.transfer(amount);
s }
s function sendTo(address receiver, uint amount) public{
s require(tx.origin == owner);
s  receiver.transfer(amount);
s }
TABLE XII: Example of AVR Mutant

4.3 Variable Unit Operators

Solidity has two unique variable units, Ether Units and Time Units. We defined the following two mutation operators to simulate the defects caused by using the wrong unit.

4.3.1 Ether Unit Replacement (EUR)

Units are indispensable whether it is to construct transactions for the transfer of Ethernet currency or to invoke intelligent contracts for the issuance of tokens. Ether’s unit suffixes are wei, finney, szabo, ether. The conversion format is [50]. Table XIII shows an EUR mutant where the unit keyword finney has been replaced by szabo. To kill this mutant, this.balance should be set between 70 finney and 70 szabo.

s if(this.balance >= 70 finney){
s  uint sendProfit = this.balance;
s }
s if(this.balance >= 70 szabo){
s  uint sendProfit = this.balance;
s }
TABLE XIII: Example of EUR Mutant

4.3.2 Time Unit Replacement (TUR)

Time is also one of the characteristics of smart contracts. The time units supported in solidity are seconds, minutes, hours, days and weeks, where seconds is the default unit [52]. The TUR operator replace a time unit suffix with another one, the sample program is shown in Table XIV.

s function f(uint start, uint timeAfter){
s if (block.timestamp >= start + timeAfter * 1 days){}
s }
s function f(uint start, uint timeAfter){
s if (block.timestamp >= start + timeAfter * 1 weeks){}
s }
TABLE XIV: Example of TUR Mutant

4.4 Error Handling Operators

Solidity uses state-reverting exceptions to handle errors. Such an exception will undo all changes made to the state in the current call. The convenience functions assert and require can be used to check for conditions and throw an exception if the condition is not met [49]. We define two kinds of mutation operators related to error handling.

4.4.1 Require Statement Deletion (RSD)

The require function is often used to ensure valid conditions on inputs or contract state variables, or to validate return values from calls to external contracts. If the condition in require() is not satisfied, the state change is revoked to check for errors caused by input or external components, and an error message can be provided at the same time. The RSD operator is inspired from the mutation operators related to if statement, but the difference is that require reverts the entire state changes in the function while if doesn’t (ESE#60585).

As is shown in Table XV, the RSD operator deletes the whole require statement by commenting them out to ensure that the following statements always execute. In addition, change the statement in require() to false is another operator, named Require Statement Change (RSC), which ensures the following statement will never get executed.

4.4.2 Assert Statement Deletion (ASD)

The assert function works in a similar way to require, but require is used to check conditions on inputs while assert is used for internal error checking (ESE#16457). The assert statement operators are also divided into two kinds. The ASD operator can enforce the execution of the following statements by deleting the assert statement (for example, line s in Fig. XV). Relatively, the Assert Statement Change (ASC) operator can make the following statements never execute by changing the condition statement to false.

s function sendHalf(address a) public payable returns (uint){
s require(msg.value%2 == 0, “Even value required.”);
s  uint balanceBefore == this.balance;
s  addr.transfer(msg.value / 2);
s  assert(this.balance == balanceBefore - msg.value / 2);
s  return this.balance;
s }
s function sendHalf(address a) public payable returns (uint){
s //require(msg.value%2 == 0, “Even value required.”);
s  uint balanceBefore == this.balance;
s  addr.transfer(msg.value / 2);
s  assert(this.balance == balanceBefore - msg.value / 2);
s  return this.balance;
s }
TABLE XV: Example of RSD Mutant

4.5 Summary

In summary, as depicted in table XVI, we proposed 15 ESC specific mutation operators, including six corresponds to keyword, three corresponds to global variable/function, two corresponds to variable unit, and four corresponds to error handling in Ethereum smart contract.

Type Operator Description
Keyword FSC Function State Keyword Chang
FVC Function Visibility Keyword Chang
DLR Data Location Keyword Replacement
VTR Variable Type Keyword Replacement
PKD Payable Keyword Deletion
DKD Delete Keyword Deletion
Global Variable
and Function
GVC Global Variable Change
MFR Mathematical Functions Replacement
AVR Address Variable Replacement
Variable Unit EUR Ether Unit Replacement
TUR Time Unit Replacement
Error Handling RSD Require Statement Deletion
RSC Require Statement Change
ASD Assert Statement Deletion
ASC Assert Statement Change
TABLE XVI: Specific Mutation Operators

5 Empirical Study

In this section, we assess the proposed mutation testing approach and the value of each mutation operator for ESC. To that end, we first present the addressed research questions followed by the experimental subjects and experiments performed.

5.1 Research Questions

The goal of this section is to answer the following research questions:

  • RQ1: Are mutation testing effective in evaluating the adequacy of ESC test-suite? For a test-suite, its test adequacy can be measured by its defect detect capability. We aim to verify that mutation testing performs well in measuring the adequacy of ESC test-suite, and expect mutation testing is stronger than coverage based approach.

  • RQ2: Are the specific mutation operators work well in mutation testing for ESC? For each specific mutation operators, we intend to know (1) the non-equivalent mutant generation rate when applying it in ESC mutation testing and (2) if they lead to real bugs in practice. This fact would allow us to analyze the correlation between mutants and real defects.

5.2 Experimental Subjects

SkinCoin 225 66 35
SmartIdentity 180 34 87
AirSwap 330 76 17
CryptoFin 348 58 44
TABLE XVII: Experimental Subjects

Our study regards a set of smart contracts in four different real-world Ethereum DApps (i.e., SkinCoin [44], SmartIdentity [47], AirSwap [2] and CryptoFin [10]) as experimental subjects. SkinCoin is a universal cryptocurrency for instant trading skins in games and making bets on e-sports events. SmartIdentity relies on Ethereum blockchain to represent an identity using a smart contract. CryptoFin is a collection of Solidity libraries, with an initial focus on arrays. AirSwap is a peer-to-peer trading network built on Ethereum. These DApps are selected because each of which provides not only a set of ESCs but also accompanies a well-designed test-suite. Thus, we do not need to manually design an ESC test-suite, which is too subjective to generate a fair experimental result and conclusion. Tools for automated ESC test generating such as ContractFuzzer [24] and MTG [61] are also not used because the generated results are not complete, i.e., missing assert statements. Tests that failed on the original DApp are removed from the subject (e.g., we removed 15 tests from AirSwap’s test-suite, which contains 32 tests). Table XVII summarizes the characteristics of DApps used in the experiments. For each DApp, its name (col. 1), lines of code (col. 2), branches of code (col. 3), and size of test-suite (col. 4) are described. All the smart contracts have been open sourced on GitHub.

5.3 Experiment 1: Mutation Testing Effectiveness

In this experiment, we compare mutation testing to coverage based approach for evaluating the effectiveness of mutation testing.

5.3.1 Setup

Mutants Generation. Given DApp under test, we first generated all the possible mutants by using the total of 25 mutation operators, of which 10 are general mutation operators, and 15 are proposed by us. After removing the equivalent mutants and those were failed during compiling, we got useful mutants. Mutants Partition and Execution. Then, we randomly selected useful mutants to build mutant set , and the remaining useful mutants build mutant set . Each mutant in was tested by the accompanying Test-Suite .  Generation.  was generated by randomly selecting a subset of that satisfies test targets w.r.t. line coverage and branch coverage, i.e., line coverage and branch coverage of are equal to that of respectively.  Generation.  was generated by randomly selecting a subset of that had the same mutation score with on . Comparison Regarding as the verification set and finally calculating the mutation scores of and on for comparison. For accurately, we ran the analysis ten times to obtain the results on average to reduce bias introduced by randomization.

5.3.2 Results

DApp General ESC Specific Total
SkinCoin 314 274 106 106 420 380
SmartIdentity 267 225 86 86 353 311
AirSwap 406 363 175 175 581 538
Cryptofin 464 406 214 183 678 589
Total 1451 1268 581 550 2032 1818
TABLE XVIII: Mutant Number of Each DApp

Table XVIII summarizes the generated mutants. For each DApp, numbers of mutants (col. 2, 4 and 6) that were generated by general operators, ESC specific operators and total operators are presented, together with the ones of non-equivalent mutants (col. 3, 5 and 7). The former five columns show an 84.27% to 89.41% non-equivalent mutating rate for general operators and a comparable 85.50% to 100% non-equivalent mutating rate for ESC specific operators, indicating the proposed operators will not increase the cost of detecting equivalent mutants during mutation testing. In total, 1818 mutants were selected from the mutant pool, which contains 2032 compilable mutants, to conduct the first experiment.

LCov BCov LCov BCov LCov BCov
SkinCoin 80.44 63.33 34.7 43.1 - - 15.2 23.6 79.33 59.98 - 40.3
SmartIdentity 91.67 85.29 63.6 47.4 - - 25.7 23.5 91.67 85.29 - 44.6
AirSwap 45.45 17.11 32.2 28.8 - - 19.3 17.6 45.45 17.11 - 28.8
Cryptofin 89.78 76.47 47.8 53.2 - - 28.4 31.3 88.71 76.47 - 51.7
Average 76.84 60.55 44.6 43.1 - - 22.2 24.0 76.29 59.71 - 41.4
TABLE XIX: Coverage Results and Mutation Scores of , and

Table XIX presents the results of line coverage and branch coverage of (col. 2-3) and (col. 10-11) when applying ESC under test, as well as the mutation scores of (col. 4-5), (col. 8-9) and (col. 13) when applying and . For the mutants that were killed by , can kill most of them (i.e., in average), whereas can only kill half of them (i.e., in average), indicating that outperforms in defect detection. To determine whether the observed difference is statistically significant or not, we applied the paired Wilcoxon test and carried out the two-tailed alternative hypothesis. The value of the test is 0.005. Therefore, we can accept the alternative hypothesis that significantly outperforms and can make a conclusion that mutation testing is more effective in evaluating the adequacy of test-suite.

5.4 Experiment 2: Mutation Operator Effectiveness

In this experiment, we assessed each mutation operator by rates of survival and equivalent. We also conducted a survey on open-sourced communities to analysis the corresponds between real defects and mutants.

5.4.1 Setup

Mutants Classification.

For both general mutants and ESC specific mutants, we further classified them and their testing results based on their mutation operators. Besides experiment, we inspected real defect reports for evaluating operator effectiveness.

Defect Reports Collection. We searched the defect reports from GitHub [20, 46, 59, 19, 18], DASP [11] and PeckShield [41], and finally collected 729 closed reports w.r.t. ESC. For each report, we determined it into one type of mutation operator.

Operator Number of Mutants MS
All Equ. Killed Live
General Mutants
AOR 257 4 117 136 46.2
AOI 416 108 128 159 44.6
ROR 410 41 148 221 40.1
COR 29 0 5 24 17.2
LOR 0 0 0 0 0.0
ASR 34 0 18 16 52.9
SDL 229 30 78 121 39.2
RVR 36 0 11 25 30.6
CSC 40 0 26 14 65.0
Subtotal 1451 183 531 716 42.6
ESC Specific Mutants
FSC 4 0 0 4 0.0
FVC 136 0 67 69 49.3
DLR 6 0 1 5 16.7
VTR 101 31 16 54 22.9
PKD 1 0 0 1 0.0
DKD 6 0 2 4 33.3
GVC 65 0 34 31 52.3
MFR 55 0 19 36 34.5
AVR 127 0 35 103 25.4
EUR 20 0 4 16 20.0
TUR 6 0 0 6 0.0
RSD 13 0 3 10 23.1
RSC 13 0 5 8 38.5
ASD 19 0 2 17 10.5
ASC 19 0 13 6 68.4
Subtotal 581 31 201 370 35.2
Total 2032 214 732 1086 40.3
TABLE XX: Experimental Results of Mutation Operators

5.4.2 Results

Table XX presents the statistics for each mutation operator, where columns 2-5 respectively depicts the numbers of all, equivalent, killed and live mutants and the last column depicts the mutation score. The first group is the mutants generated by general mutation operators. In addition to LOR, other general mutation operators generate at least one mutant. Among them, COR has the lowest mutation score of 17.2, but this only shows that the test cases are not sufficient in this respect, and does not mean that COR operator is ineffective. AOI generates the most mutants. However, 108 of the 416 mutants generated by AOI are equivalent, this is because the inserted arithmetic operators do not work on variables that have an impact on the execution results. The 10 traditional mutation operators generate a total of 1451 mutants, of which 1247 were non-equivalent, so the non-equivalent mutant generation rate is 85.94%.

The second group corresponds to ESC specific mutation operators. FSC, PKD and TUR have the lowest mutation scores of 0, indicating that none of their mutants was killed by . These three operators only generated eleven mutants, so this low percentage probably isn’t meaningful. FVC and AVR generated 136 and 127 mutants respectively, accounting for nearly half of all mutants. In addition, VTR generated 101 mutants, but 31 of them are equivalent mutants. For example, VTR can replace uint in (uint i = 0; i < 20; i++) with int and uint8, thus result in two mutants. However, both mutants will not affect the result of program execution, so they are logically equivalent. ASC generated 19 mutants and got the highest mutation score of 68.4, while ASD generates 19 mutants with the lowest mutation score of 10.5. This probably indicates that many test cases only trigger assert(true) without considering assert(false), which may lead to a hidden defect. The average mutation score of the special mutation operators on is 35.2, which is lower than that of the general mutation operators (42.6). This shows that testers pay less attention to the problems caused by solidity characteristics when writing test cases. This reflects the significance of our mutation operators.

Further, we conducted a survey to look up various issues and bug reports related to ESC from open source communities. Among 729 reports, 117 are related to our mutation operator, including 41 Keyword Operators bugs, 35 Global Variables and Functions Operators bugs, 9 Variable Unit Operators bugs and 32 Error Handling Operators bugs. Representative samples are as follows:

  • SWC-100 (FVC) Functions that do not have a function visibility type specified are public by default. This can lead to a vulnerability if a developer forgot to set the visibility and a malicious user is able to make unauthorized or unintended state changes.

  • SWC-109 (DLR) Uninitialized local storage variables can point to unexpected storage locations in the contract, which can lead to unintentional vulnerabilities.

  • DASP#item-3 (VTR/MFR) An overflow condition gives incorrect results and, particularly if the possibility has not been anticipated, can compromise a program’s reliability and security.

  • SWC-120 (GVC) block.timestamp is insecure, as a miner can choose to provide any timestamp within a few seconds and still get his block accepted by others. Use of blockhash, block.difficulty and other fields is also insecure, as they’re controlled by the miner.

  • DASP#item-2 (AVR/RSD/RSC) Access Control vulnerabilities can occur when contracts use tx.origin instead of msg.sender to validate callers, handle large authorization logic with lengthy require and make reckless use of delegatecall in proxy libraries or proxy contracts.

  • SWC-123 (ASD/ASC) The Solidity assert() function is meant to assert invariants. Properly functioning code should never reach a failing assert statement. A reachable assertion mean that a bug exists in the contract that allows it to enter an invalid state or the assert statement is used incorrectly (e.g. to validate inputs).

The result shows that our newly proposed Solidity mutation operator can inject real defects, and thus can help developers avoid some common mistakes.

6 Threats to Validity

Internal validity. There are two main internal threats to validity. Firstly, whether we mutate the smart contract correctly. Our mutation operations are implemented at the AST level, where AST is generated with solidity-parser-antlr, the mutation and restoration of AST are implemented by ourselves. To verify that our operators were correctly implemented, we checked every mutant by hand to ensure it was mutated as expected. Secondly, due to the constant update of the solidity version, the parser and mutation tool need to be updated accordingly. Meanwhile, the design of mutation operators should also consider the impact of the version update to meet the latest test requirements.

Construct validity. There are two construct threats to validity. Firstly, the test code of all smart contracts is attached to original the project, while the quality of test cases will have an impact on our experiment (for example, low-quality test cases will make all the generated mutants survive). Secondly, the identification of equivalent mutants is performed manually, so we can not ensure that all equivalent mutants are excluded.

External validity. Our experiments were conducted on 26 smart contracts from four DApps, so it is not possible to guarantee the representative of selected subjects. We managed to select DApps from four different areas to maximize the representation of the experiment.

7 Related Work

In this section, we will describe related work in two areas: Mutation Testing and Smart Contract Testing.

7.1 Mutation Testing

Mutation testing was first discovered and made public by DeMillo, Lipton and Sayward [14], and explored extensively by Offutt and others [36]. It is based on two premises: the competent programmer hypothesis [14] and the coupling effect hypothesis [37]. The skilled programmer hypothesis assumes that the defect code written by the programmer is very close to the correct code, and that the defect can be removed with only minor modifications. Based on this assumption, mutation testing can simulate the actual programming behavior of skilled programmers only by modifying the amplitude code of the program under test. Coupling effect [37] hypothesis points out that complex faults are coupled with simple faults, so a test data set that detects simple faults (such as those introduced by mutation) will detect complex faults, i.e., the combination of several simple faults.

Mutation testing has been applied to many programming languages such as C [8], C++ [12], C# [16], Java [42], JavaScript [31], Ruby [27], Android [15] and web applications [43]. It has also been adapted for some very popular programming paradigms such as Object-Oriented [29], Functional [26], aspect-oriented and declarative programming [38][57]. However, to the best of our knowledge, there is no research paper that introduce mutation testing to the smart contract.

With appropriate mutation operators exposing potential defects, mutation testing can provide a strong standard for evaluating test adequacy [40]. In addition to the adequacy evaluation of a test suite, mutation testing can also simulate the real defects of the software under test by applying mutation operators, thus assisting the validity evaluation of the test methods proposed by researchers. For example, Andrews et al. [5] and Do et al.[17] have proved that mutation defects generated by mutation operators are similar to real defects in effectiveness evaluation.

7.2 Smart Contract Testing

Chia et al. list four approaches to help blockchain test engineers in his paper [9]. The first approach is to improve the Documentation on smart contract for developers and testers, the second approach is to fuzz the inputs of the smart contracts, the third approach is to mutate the code of smart contracts, and the fourth approach is to search the blockchain for traces of already deployed smart contracts. There are already some research results in these areas. For example, Jiang et al.[24] proposed the ContractFuzzer for the fuzzing test, which can detect vulnerabilities in smart contracts through random fuzzing. Wang et al.[61] guides the automatic generation of efficient test cases by tracking the execution information of smart contracts on the chain. As for mutation testing, as far as we know, there has been no existing literature on the application of mutation testing for smart contracts. It has been proposed before to develop a mutation tool for smart contracts [33], but the attempt has been abandoned. The challenge is for the mutation generator to understand enough of the semantics of the smart contract language to generate only useful mutants.

Currently, we have found two mutation testing tools available for smart contract on GitHub, eth-mutants [1] and universalmutator [22]. However, eth-mutants only implements boundary condition mutation operators, which means that it can only replaces and for and and vice-versa. And universalmutator is a regexp based tool for mutating generic source code across numerous languages, it has designed several mutation operators for solidity, but it’s not enough. Therefore, it is necessary to conduct an in-depth research to improve the effectiveness of mutation testing for smart contracts.

8 Conclusion and Future Work

As an effective method of improving the adequacy of testing, mutation testing is hard to be widely used in industry because of its high cost. But in the field of Ethereum, mutation testing can work well because ESC is almost impossible to be modified on Ethereum. In this paper, we proposed a novel mutation approach for ESC, together with a set of ESC specific mutation operators. The empirical study on a set of smart contracts in four real-world DApp verified that mutation testing works well in evaluating the adequacy of ESC test-suite and the proposed ESC operators can reflect real defects in practice. We believe that these mutation operators can indeed help testers discover potential defects in smart contracts and write more adequacy test cases to ensure the security of blockchain applications.


  • [1] (2018) A mutation testing tool for solidity contracts. Note: Cited by: §7.2.
  • [2] (2019) AirSwap. Note: Cited by: §5.2.
  • [3] (2018) ALERT: new batchoverflow bug in multiple erc20 smart contracts. Note: Cited by: §4.1.4.
  • [4] S. Ammous (2016) Blockchain technology: what is it good for?. Cited by: §1.
  • [5] J. H. Andrews, L. C. Briand, and Y. Labiche (2005) Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th International Conference on Software Engineering (ICSE), St. Louis, Missouri, USA, pp. 402–411. Cited by: §2.2, §7.1.
  • [6] M. Bartoletti and L. Pompianu (2017) An empirical analysis of smart contracts: platforms, applications, and design patterns. In Proceedings of the 21st International Conference on Financial Cryptography and Data Security (FC), pp. 494–509. Cited by: §1.
  • [7] V. Buterin (2014) Ethereum white paper: A next-generation smart contract and decentralized application platform. Cited by: §2.1.
  • [8] T. T. Chekam, M. Papadakis, Y. Le Traon, and M. Harman (2017) An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings of the 39th IEEE/ACM International Conference on Software Engineering (ICSE), Buenos Aires, Argentina, pp. 597–608. Cited by: §1, §1, §7.1.
  • [9] V. Chia, P. Hartel, Q. Hum, S. Ma, G. Piliouras, D. Reijsbergen, M. van Staalduinen, and P. Szalachowski (2018) Rethinking blockchain security: position paper. arXiv. Cited by: §7.2.
  • [10] (2019) CryptoFin. Note: Cited by: §5.2.
  • [11] (2019) Decentralized application security project. Note: Cited by: §5.4.1.
  • [12] P. Delgado-Pérez, I. Medina-Bulo, F. Palomo-Lozano, A. García-Domínguez, and J. J. Domínguez-Jiménez (2017) Assessment of class mutation operators for C++ with the MuCPP mutation system. Information and Software Technology 81 (2017), pp. 169–184. Cited by: §1, §1, §7.1.
  • [13] P. Delgado-Pérez, S. Segura, and I. Medina-Bulo (2017) Assessment of C++ object-oriented mutation operators: a selective mutation approach. Software Testing, Verification and Reliability 27 (4-5), pp. e1630. Cited by: §1.
  • [14] R. A. DeMillo, R. J. Lipton, and F. G. Sayward (1978) Hints on test data selection: help for the practicing programmer. Computer 11 (4), pp. 34–41. Cited by: §7.1.
  • [15] L. Deng, J. Offutt, P. Ammann, and N. Mirzaei (2017) Mutation operators for testing android apps. Information and Software Technology 81, pp. 154–168. Cited by: §2.2, §7.1.
  • [16] A. Derezinska and K. Kowalski (2011) Object-oriented mutation applied in common intermediate language programs originated from C#. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 342–350. Cited by: §7.1.
  • [17] H. Do and G. Rothermel (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Transactions on Software Engineering 32 (9), pp. 733–752. Cited by: §7.1.
  • [18] (2019) Ethereum-goethereum-issues. Note: Cited by: §5.4.1.
  • [19] (2019) Ethereum-solidity-issues. Note: Cited by: §5.4.1.
  • [20] (2019) Examples of solidity security issues. Note: Cited by: §5.4.1.
  • [21] P. G. Frankl, S. N. Weiss, and C. Hu (1997) All-uses vs mutation testing: An experimental comparison of effectiveness. Journal of Systems and Software 38 (3), pp. 235–253. Cited by: §2.2.
  • [22] A. Groce, J. Holmes, D. Marinov, A. Shi, and L. Zhang (2018) An extensible, regular-expression-based tool for multi-language mutant generation. In Proceedings of the 40th IEEE/ACM International Conference on Software Engineering: Companion (ICSE-Companion), pp. 25–28. Cited by: §7.2.
  • [23] (2016) Introduce a real constant keyword and rename the current behaviour. Note: Cited by: §4.1.1.
  • [24] B. Jiang, Y. Liu, and W. Chan (2018) Contractfuzzer: fuzzing smart contracts for vulnerability detection. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), pp. 259–269. Cited by: §5.2, §7.2.
  • [25] R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser (2014) Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), pp. 654–665. Cited by: §2.2.
  • [26] D. Le, M. A. Alipour, R. Gopinath, and A. Groce (2014) MuCheck: an extensible tool for mutation testing of haskell programs. In Proceedings of the 23rd International Symposium on Software Testing and Analysis (ISSTA), pp. 429–432. Cited by: §1, §7.1.
  • [27] N. Li, M. West, A. Escalona, and V. H. Durelli (2015) Mutation testing in practice using Ruby. In Proceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–6. Cited by: §7.1.
  • [28] L. Luu, D. Chu, H. Olickel, P. Saxena, and A. Hobor (2016) Making smart contracts smarter. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 254–269. Cited by: §1.
  • [29] Y. Ma, Y. Kwon, and J. Offutt (2002) Inter-class mutation operators for Java. In Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE), pp. 352–363. Cited by: §1, §7.1.
  • [30] Y. Ma, J. Offutt, and Y. R. Kwon (2005) MuJava: an automated class mutation system. Software Testing, Verification and Reliability 15 (2), pp. 97–133. Cited by: §4.1.2, §4.1.6.
  • [31] S. Mirshokraie, A. Mesbah, and K. Pattabiraman (2013) Efficient javascript mutation testing. In Proceedings of the 6th IEEE International Conference on Software Testing, Verification and Validation (ICST), Luxembourg, pp. 74–83. Cited by: §1, §2.2, §7.1.
  • [32] S. Mirshokraie, A. Mesbah, and K. Pattabiraman (2015) Guided mutation testing for JavaScript Web applications. IEEE Transactions on Software Engineering 41 (5), pp. 429–444. Cited by: item (2), §3.
  • [33] (2016) Mutation testing support for solidity. Note: Cited by: §7.2.
  • [34] S. Nakamoto (2008) Bitcoin: a peer-to-peer electronic cash system. Cited by: §1.
  • [35] I. Nikolić, A. Kolluri, I. Sergey, P. Saxena, and A. Hobor (2018) Finding the greedy, prodigal, and suicidal contracts at scale. In Proceedings of the 34th Annual Computer Security Applications Conference (ACSAC), San Juan, PR, USA, pp. 653–663. Cited by: §1, §1.
  • [36] A. J. Offutt and R. H. Untch (2001) Mutation testing for the new century. pp. 34–44. Cited by: §7.1.
  • [37] A. J. Offutt (1992) Investigations of the software testing coupling effect. ACM Transactions on Software Engineering and Methodology 1 (1), pp. 5–20. Cited by: §7.1.
  • [38] E. Omar and S. Ghosh (2012) An exploratory study of higher order mutation testing in aspect-oriented programming. In Proceedings of the 23rd IEEE International Symposium on Software Reliability Engineering (ISSRE), pp. 1–10. Cited by: §1, §7.1.
  • [39] A. Orso and G. Rothermel (2014) Software testing: a research travelogue (2000–2014). In Proceedings of the 2nd Workshop on Future of Software Engineering (FOSE), Hyderabad, India, pp. 117–132. Cited by: §1.
  • [40] M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman (2019) Mutation testing advances: An analysis and survey. In Advances in Computers, Vol. 112, pp. 275–378. Cited by: §1, §1, §7.1.
  • [41] (2019) PeckShield. Note: Cited by: §5.4.1.
  • [42] (2019) Pitest: state of the art mutation testing system for the jvm. Note: Cited by: §1, §1, §7.1.
  • [43] U. Praphamontripong and J. Offutt (2010) Applying mutation testing to Web applications. In Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), pp. 132–141. Cited by: §7.1.
  • [44] (2019) SkinCoin. Note: Cited by: §5.2.
  • [45] (2019) Smart contract languages. Note: Cited by: §1, §2.1.
  • [46] (2019) Smart contract weakness classification and test cases. Note: Cited by: §5.4.1.
  • [47] (2019) SmartIdentity. Note: Cited by: §5.2.
  • [48] (2019) Solidity data location. Note: Cited by: §4.1.3.
  • [49] (2019) Solidity error handling: assert, require, revert and exceptions. Note: Cited by: §4.4.
  • [50] (2019) Solidity ether units. Note: Cited by: §4.3.1.
  • [51] (2019) Solidity mathematical and cryptographic functions. Note: Cited by: §4.2.2.
  • [52] (2019) Solidity time units. Note: Cited by: §4.3.2.
  • [53] (2019) Solidity-parser-antlr. Note: Cited by: item (1).
  • [54] (2019) Special variables and functions. Note: Cited by: §4.2.
  • [55] (2019) State variable default visibility. Note: Cited by: §4.1.2.
  • [56] N. Szabo (1996) Smart contracts: building blocks for digital markets. Cited by: §2.1.
  • [57] J. Tuya, M. J. Suárez-Cabal, and C. De La Riva (2007) Mutating database queries. Information and Software Technology 49 (4), pp. 398–417. Cited by: §7.1.
  • [58] (2019) Uninitialized storage pointer. Note: Cited by: §4.1.3.
  • [59] (2019) USCC-submissions-2017. Note: Cited by: §5.4.1.
  • [60] P. J. Walsh (1985) A measure of test case completeness (software, engineering). Cited by: §2.2.
  • [61] X. Wang, H. Wu, W. Sun, and Y. Zhao (2019) Towards generating cost-effective test-suite for Ethereum smart contract. In Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Hangzhou, China, pp. 549–553. Cited by: §1, §5.2, §7.2.
  • [62] G. Wood (2014) Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper, pp. 1–32. Cited by: item (2).
  • [63] Z. Zheng, S. Xie, H. Dai, X. Chen, and H. Wang (2018) Blockchain challenges and opportunities: a survey. International Journal of Web and Grid Services 14 (4), pp. 352–375. Cited by: §1, §2.1.