In the recent years, crowdsourcing has been gaining attention as a promising modern business model that enables individuals and organizations to receive services from a large group of people or crowd. Crowdsourcing services support a variety of tasks ranging from software development to logo designs [chittilappilly2016survey]. For example, LEGO Ideas is attracting a lot of fan designers to enter prize contests by submitting original proposals for new LEGO Ideas sets. A few top-ranked proposals in the contest have resulted in successful commercialization [LEGO]. Likewise, since 2005, UNIQLO has continuously held annual Global T-Shirt Design Competitions (UTGP) and its 14th UTGP in 2019 has attracted over 18,000 entries from all over the world [UTGP]. The increasing popularity of crowdsourcing solutions is also driving the rise of crowdsourcing intermediate platforms such as Upwork [Upwork], 99designs [99designs] and designContest [designContest] that connect business clients to designers including individual freelancers and design agencies. For instance, via designContest, a startup company may set up a logo design contest with a description of its requirements and include a monetary reward. After receiving a large number of entries from interested designers, the client may invite its target audience such as its followers on Twitter to vote for the favorite design and finally pick one or multiple winning entries based on the voting result. Nearly all such crowdsourcing intermediate platforms make profits by charging fees from clients and designers. Upwork reported in second quarter 2019 that its Gross Services Volume (GSV) grew 20% year-over-year to $518.8 million [UIR]. On a $500 crowdsourcing project in Upwork, the platform would charge $100 as the service fee. Similarly, 99designs would charge $75 from a $500 project as the platform fee. Such high service fees significantly increase the cost of running crowdsourcing projects online. Clients and designers have no choice but accept it as the mutually distrusted parties need a trustworthy intermediary for avoiding dishonest behaviors such as free-riding and false-reporting [zhang2012reputation].
Recent advancements in blockchain technology have led to the development of numerous open smart contract platforms including Ethereum [buterin2014next, wood2014ethereum]. Ethereum has become a promising technology for decentralizing traditional centralized online services and as result, in the recent years, we have witnessed a rapid proliferation of numerous decentralized applications including decentralized crowdsourcing systems [calado2018tamper, duan2019aggregating, feng2019mcs, li2018crowdbc, lu2018zebralancer, wang2018blockchain, xu2019blockchain]. Such services offer clients and designers an option to reduce the high intermediary fee required in centralized crowdsourcing systems. However, blockchains provide tamper resistance properties only at a cost. For example, Ethereum charges each transaction a small fee based on the complexity. In Ethereum, tens of thousands of miners follow the Proof-of-Work (PoW) consensus protocol [nakamoto2008bitcoin]
to compete for solving puzzles and each winner receives a monetary reward for packaging the recent transactions (i.e., transferring fund, executing functions or creating smart contracts) into a new block appended to the end of the blockchain. Fees charged from transactions within a new block in the blockchain are paid to the competition winner who packages the block. People trust that no one can tamper with the blockchain as the probability of a single miner to win in several consecutive competitions to be able to change the network consensus about the blockchain state is negligible. As part of the competition reward, transaction fees help incentivize miners to invest more computation resources into the competitions, which in turn increases the difficulty of competitions and improves the overall safety of the blockchain. Also, transaction fees help protect Ethereum against DDoS and Sybil attacks[douceur2002sybil] as the cost of creating transactions will have a cost of . Despite their significance to the safety of Ethereum, transaction fees turn out to be an obstacle to existing decentralized crowdsourcing systems, especially when crowdsourcing projects are scaled up. Consider that a client sets up a decentralized design contest in Ethereum, where each entry needs to be submitted by a designer via a transaction that costs a small fee, say $1. The total fee charged in this contest would be cheaper than a contest in 99designs only when there are less than 75 entries. In other words, intermediary fees are not eliminated by decentralizing crowdsourcing but are paid to a decentralized infrastructure instead of a centralized service provider for the same purpose of acquiring trust. As a result, it is not surprising to see that sometimes there may be no considerable economic advantage in decentralizing crowdsourcing. For example, if we take the price of ether 111 The native cryptocurrency in Ethereum, denoted by . as its mean value during the first half of the year 2019 recorded in Etherscan [etherscan], a crowdsourced image tagging task completed via CrowdBC [li2018crowdbc] could very well spend over four times the price charged by the centralized Amazon Mechanical Turk [AMT]. Similarly, aggregating data from 1,000 providers via decentralized crowdsensing [duan2019aggregating] could cost up to $170.
This paper aims at addressing the challenging question: how to provide a reliable solution that decouples the cost of decentralizing crowdsourcing from the scale of the crowd? Our research illustrates that the root cause of the high cost in existing decentralized crowdsourcing systems is the lack of cost-efficient solutions to exploit decentralized trust. Crowdsourcing projects usually involve steps that aggregate data (i.e., contest entries or sensed data) from the crowd of scale or perform calculations on aggregated data (i.e., determine winning entries based on votes). Such steps become cost-intensive in smart contract platforms like Ethereum as they either charge accumulated small transaction fees (TYPE ) or a single large transaction fee (TYPE ), both resulting in cost. In this paper, we propose a novel suite of protocols called NF-Crowd that reliably resolves the scalability issue by reducing the lower bound of the total cost of a decentralized crowdsourcing project to . We prove that as long as participants of a project powered by NF-Crowd are rational, the lower bound of cost could be reached regardless of the scale of the crowd. We also demonstrate that as long as at least one participant of a project powered by NF-Crowd is honest, the project cannot be aborted and the results are guaranteed to be correct. We design NF-Crowd protocols for a representative type of project named crowdsourcing contest with open community review (CC-OCR). We implement the protocols over the Ethereum official test network. Our results demonstrate that NF-Crowd protocols can reduce the cost of running a CC-OCR project to less than $2 regardless of the scale of the crowd, providing a significant cost benefit in adopting decentralized crowdsourcing solutions.
The rest of this paper is organized as follows: We start by introducing preliminaries in Section II. In Section III, we present a strawman protocol for CC-OCR projects and categorize the cost-intensive steps resulting in cost. Then, in Section IV, we propose the NF-Crowd protocol for CC-OCR projects that reduces the lower bound of cost to . We implement and evaluate the NF-Crowd protocols over the Ethereum official test network in Section V. Finally, we discuss related work in Section VI and conclude in Section VII.
In this section, we discuss the preliminaries about smart contracts and introduce the key assumptions, key cryptographic tools and notations used in our work. Though we discuss smart contracts in the context of Ethereum [wood2014ethereum], our solutions are applicable to a wide range of other smart contract platforms as well.
Ii-a Account types
There are two types of accounts in Ethereum, namely External Owned Accounts (EOAs) and Contract Accounts (CAs). To interact with the Ethereum blockchain, a user needs to create an EOA and control it via a pair of keys. Specifically, the public key can generate a 20-byte address to uniquely identify the EOA and the private key can be used by the user to sign transactions or other types of messages. Then, any user can create a smart contract by sending out a contract creation transaction from a controlled EOA. The 20-byte address of the created smart contract becomes the unique identity of the contract account (CA).
The state of Ethereum blockchain can only be changed by the external world (i.e., EOAs) using transactions. A transaction is a serialized binary message sent from an EOA (i.e., sender) that contains the following elements:
nonce: a sequence number issued by the EOA (i.e., transaction creator) to prevent transaction replay;
gas price: the price of gas the EOA is willing to pay;
gas limit: the maximum amount of gas the EOA can afford;
recipient: the recipient account address;
value: the amount of ether to send to the recipient;
data: the binary data payload;
vrs: the ECDSA digital signature of the EOA.
Depending on the value at recipient
(i.e., EOA or CA or 0x0), transactions can be classified into three categories.
Fund transfer transaction: A transaction with an EOA as recipient and a non-empty value is a fund transfer transaction, which is used to transfer an amount of ether from the sender EOA to the recipient EOA. On the other hand, data carried by a transaction is usually ignored by Ethereum clients and wallets that help users control their EOAs.
Function invocation transaction: When a transaction involves a CA as recipient as well as a non-empty data, it is usually a function invocation transaction for calling a function within an existing smart contract. Specifically, when the transaction is used to call a function with arguments, such as:
the data payload of the transaction would be in the form of
namely concatenation of the encoded string , also called as function selector, and the encoded values for all function arguments. When a function invocation transaction additionally carries a non-empty value and the invoked function is marked payable such as:
The amount of ether indicated by value would be transferred from the sender EOA to the recipient CA.
Contract creation transaction: In Ethereum, there is a special type of transaction for creating new smart contracts. Such a transaction, usually referred to as a contract creation transaction, carries a special recipient address 0x0, an empty value and a non-empty data payload. A smart contract (or contract) in Ethereum is a piece of program created using a high-level contract-oriented programming language such as Solidity [Solidity2017]. After compiling into a low-level bytecode language called Ethereum Virtual Machine (EVM) code, the created contract is filled into a contract creation transaction as the data payload.
A user, after filling recipient, value and data of a transaction, will then input nonce, gas price and gas limit and finally sign the transaction with the private key of the sender EOA to get signature vrs. After that, a complete transaction is created. To make the transaction get executed to change the state of the Ethereum blockchain, the transaction should be broadcast to the entire Ethereum network formed by tens of thousands of miner nodes. Following the Proof-of-Work (PoW) consensus protocol [nakamoto2008bitcoin], miners in Ethereum competitively solve a blockchain puzzle and the winner packages the received transactions into a block and appends the new block to the end of Ethereum blockchain. From then on, it is hard to tamper with the blockchain state updated by the transaction (i.e., transferred fund, executed function or created contract). Thus, transactions and smart contracts in Ethereum are executed transparently in a decentralized manner and the results are deterministic.
Ii-C Transaction fees
In order to either deploy a new contract or call a deployed contract in Ethereum, one needs to spend Gas, or transaction fees. Based on the complexity of the contract or that of the called function, an amount of ether needs to be spent to purchase an amount of Gas as a transaction fee, which is then paid to the winning miner. The Gas system is important for Ethereum as it helps to incentivize miners to stay honest, to nullify denial-of-service attacks and to encourage efficiency in smart contract programming. On the other hand, the Gas system requires protocols, especially the multi-party ones, to be designed with higher scalability in Ethereum. This is due to the fact that even a single-round multi-party protocol could spend a lot of money to run in case of a large number of participants.
Ii-D Off-chain channels
In Ethereum, nodes forming the underlying P2P network can send messages to each other via off-chain channels established through the Whisper protocol [Whisper2017]. By default, messages are broadcast to the entire P2P network. A node can set up a filter to only accept messages marked with a specific 4-byte topic. Besides, a node can locally generate a pair of asymmetric Whisper keys and reveal the public Whisper key to the blockchain, which allows other nodes to privately communicate with it.
Ii-E Key assumptions
We make the following key assumptions in this paper:
We assume that the underlying blockchain system satisfies liveness, consistency and immutability properties [garay2017bitcoin].
We also make the synchrony assumption that ensures that there exists a known upper bound on the delay of messages.
Finally, we assume that at least one reviewer is honest.
|a client who sets up a design contest|
|a designer who wants to win a reward|
|a reviewer who casts vote to entries|
|a smart contract|
|function within contract|
|broadcast information via off-chain channels|
|transmit infomation via private off-chain channels|
|invoke a function within a smart contract|
|an address of an EOA or a CA|
|a keccak hash value|
|a content-addressed IPFS link|
Ii-F Cryptographic tools
In the next two sections, we start by presenting a strawman protocol for a CC-OCR crowdsourcing scenario. We then introduce the idea of NF-Crowd for reducing cost to . Before discussing the proposed protocols, we summarize the notations that will be used in the rest of this paper in Table I.
Iii CC-OCR: A strawman protocol
In this section, we first describe the crowdsourcing contest with open community review (CC-OCR) as a three-phase process. We then propose a strawman protocol that decentralizes CC-OCR projects over Ethereum. We finally identify and categorize the cost-intensive steps within the strawman protocol that results in its total cost.
Iii-a CC-OCR as a three-phase process
We describe a CC-OCR project as a three-phase process similar to the procedure of LEGO Ideas [LEGO]:
CC-OCR.initial: The client () initiates a design contest with a detailed description of its requirements as well as a reward.
CC-OCR.entry: The interested designers (s) within the community submit their entries to the contest.
CC-OCR.review: The client () invites selected members from the community as reviewers (s) to cast votes to the entries (one vote per ) and pick one or multiple winning entries based on the voting result. Here, the known identities of reviewers prevent them from performing a Sybil attack and hence, the one vote per reviewer could be guaranteed.
Iii-B The strawman CC-OCR protocol
We first propose a strawman protocol to decentralize CC-OCR projects over Ethereum. We sketch the protocol in Fig. 1 and present the formal description in Fig. 2. The regulations of the strawman protocol are programmed as an agency smart contract , through which a client can hold a contest to receive entries and votes from a community.
CC-OCR.initial: Client initiates a contest by sending out a function invocation transaction from an owned EOA to call the function at CA . The transaction would carry two arguments at its payload: (1) a list of
indicating ending times of phases or epochs in the contest; (2) an IPFS link guiding designers to a file describing the requirements of this contest. The transaction would also include a non-emptyto transfer an amount of ether to CA to reward winning entries. From then on, the arguments are permanently recorded in the blockchain and this new contest can be uniquely identified via the client address and the number of contests that have been created by at CA .
CC-OCR.entry: After reading the descriptions, interested designers from the community could submit entries to the contest. The protocol requires a designer to create an entry object EO by first generating a one-time asymmetric key pair, then encrypting the detailed with the public key and finally computing the IPFS link of encrypted . Such an EO could make confidential before the review phase and also reduce the size of data stored on the chain. After that, the designer could call with a transaction carrying arguments EO as well as that specifies the participating contest. It is worth noting that any entry submmitted after the specified by for the entry phase would be rejected because includes a time checker function:
which requires miners to only accept the transaction if the timestamp is smaller than input via .
CC-OCR.review: The protocol divides the review phase into three epochs. The first epoch is for designers to reveal private keys so that reviewers can read their entries. Then, during the second epoch, each reviewer creates a vote object VO which in the strawman protocol is simply the address of the designer that wants to vote for. The created VO would be submitted to CA via . Finally, client shall call during the third epoch and the function would traverse the votes received by all the entries and pick one or multiple entries obtaining the greatest number of votes as winners, who could later withdraw the deposited by via .
Iii-C The cost-intensive steps
Despite the simplicity, the strawman CC-OCR protocol involves two key representative types of cost-intensive steps, defined as TYPE and TYPE , respectively.
Definition 1 (Type ).
A protocol step is said to be in TYPE if the number of transactions created at this step increases along with the scale of the crowd.
Definition 2 (Type ).
A protocol step is said to be in TYPE if the cost of a single transaction created at this step increases along with the scale of the crowd.
It is easy to see that both step 2 and 4 of the strawman protocol in Fig. 2 are in TYPE while step 5 is in TYPE . At step 2 (4), data is aggregated from the crowd and each designer (reviewer) needs to submit data via a function invocation transaction () that spends a certain amount of ether and hence, the accumulated cost at this step increases along with the scale of the crowd (i.e., number of transactions). At step 5, function contains a for loop to iterate over all entries to pick the winners and each iteration spends a certain amount of ether and therefore, the accumulated cost at this step also increases along with the scale of the crowd (i.e., number of iterations).
A TYPE step is associated with at least one TYPE step, but not vice versa.
We can see that even though steps 2 and 4 are in TYPE , there is a difference between them. Specifically, step 2 leverages the blockchain as a bulletin board to post entries that meet the requirements of the contest (e.g., before ) in a deterministic and trustworthy way and hence, data received at step 2 (i.e., EO) is not associated with any future step in the protocol. In contrast, data aggregated at step 4 (i.e., VO) is associated with step 5 as the votes are collected for the purpose of being the inputs of a function that outputs poll winners.
Next, we introduce the NF-crowd CC-OCR protocol that leverages cost-cutting strategies to convert the two types of cost-intensive steps into cost-efficient steps and reduce the lower bound of the total cost of a decentralized CC-OCR project to .
Iv CC-OCR: An NF-Crowd protocol
We begin by presenting the high-level ideas of the NF-Crowd CC-OCR protocol. We then present the strategies of cutting costs at TYPE steps and TYPE steps in detail. We finally analyze the total cost and safety of the proposed protocol. We present the NF-Crowd CC-OCR protocol sketch in Fig. 3 and the formal description in Fig. IV-B.
Iv-a High-level ideas
The NF-Crowd CC-OCR protocol design includes the following novel mechanisms:
Enforceable off-chain execution: Similar to the strawman CC-OCR protocol, most existing decentralized crowdsourcing systems are designed to be executed in an on-chain mode, where both data storage and computation over stored data are performed on the blockchain to employ the decentralized trust in a simple but expensive way [duan2019aggregating, li2018crowdbc, wu2019bptm]. We believe that decentralized trust could be employed in a more cost-effective way. Specifically, we design the NF-Crowd CC-OCR protocol to be executed in an off-chain mode by default, involving no cost-intensive on-chain operations resulting in cost when all participants are honest. In case if any dishonest participant performs any fraudulent behavior that violates the protocol and aborts the off-chain execution, any honest participant reserves the ability to switch the off-chain mode to an on-chain mode similar to the strawman CC-OCR protocol so that the execution of the protocol could always get enforced.
Punishable protocol violation: In order to incentivize participants to stay honest so that the protocol can end successfully in its off-chain mode, we require each participant to lock an amount of ether in smart contracts as a security deposit to penalize potential misbehaviors that violate the protocol by confiscating the security deposit paid by the corresponding dishonest participant. Specifically, anyone who wishes to join the community to engage in a contest needs to first send out a function invocation transaction to call a function called at CA . The transaction carries no arguments but is with a non-empty to transfer an amount of ether to the CA as . Likewise, during the initial phase, besides , the function also charges an amount of ether from client as .
Iv-B Type cost-cutting strategy
The proposed strategy for cutting the cost of TYPE steps consists of four components, illustrated as step 2.1 (4.1), step 2.2 (4.2), step 2.3 (4.3) and step 6 in Fig. IV-B, respectively.
Off-chain aggregation: The first component converts TYPE on-chain aggregation into TYPE off-chain aggregation. The on-chain aggregation employed in the strawman protocol requires participants (i.e., designers or reviewers) to separately submit data to the blockchain, resulting in transactions. In the NF-Crowd protocol, participants transmit data to a single uploader (i.e., client ), who then submits aggregated data via a single transaction to the blockchain. In Fig. IV-B, at step 2.1 (4.1), each EO (VO) additionally carries a signature () and is then transmitted to client via off-chain channels.
On-chain Merkle root: The second component leverages a Merkle tree to reduce the cost of TYPE off-chain aggregation from to . Instead of uploading raw aggregated data to the blockchain, client here can group the aggregated data as a number of chunks222 Aggregated data is grouped into chunks for reducing the cost of reloading aggregated data onto the chain. We discuss more details on this in Section V. , create a Merkle tree that takes the chunks as elements, make chunks public via off-chain channels and upload the Merkle root to the blockchain. This strategy has the following properties:
O(1) cost: The only data that needs to be uploaded is the Merkle root and the cost of uploading the 32-byte root is constant.
Transparency: All chunks are available to all participants via off-chain channels.
Tamper resistance: The integrity of each chunk could be verified via the on-chain Merkle root and the integrity of each EO (VO) could be verified via ().
Traceability: Signatures () could be used to obtain the addresses of the designers (reviewers).
Thus, as long as all the participants are honest, this proposed strategy can achieve cost with the same level of security as in the on-chain aggregation strategy. At step 2.2 (4.2) in Fig. IV-B, client creates a Merkle tree for received EOs (VOs), upload Merkle root via function () and reveal the IPFS link of () via off-chain channels.
Countermeasure against intentional exclusion: In on-chain aggregation, data is uploaded by participants themselves and it is hard to keep any participant from getting involved in a contest. In off-chain aggregation, for the purpose of cutting costs, data from all participants is uploaded together by a single uploader, hence a dishonest uploader here has the ability to intentionally exclude data belonging to certain participants from the Merkle tree. As a countermeasure strategy, when a designer (reviewer) fails to verify her data via the on-chain Merkle root, the designer (reviewer) could re-upload the data onto the chain by herself as in the on-chain aggregation before the end of entry (review.e2) phase. The additional cost is shared between the participant and the uploader. In other words, we design the off-chain aggregation strategy to be backed up by the on-chain aggregation strategy and hence a dishonest uploader can hardly block any participant. Meanwhile, both the uploader and participants are incentivized to honestly follow the off-chain strategy to avoid additional charges in the on-chain strategy. This component is illustrated at step 2.3 (4.3) in Fig. IV-B, where function () would refund half of the transaction fee to the designer (reviewer) from paid by client .
Countermeasure against double votes: It is possible that a dishonest participant uploads data twice, first time via off-chain aggregation and second time via on-chain aggregation. For instance, a dishonest reviewer can first submit a VO to client at step 4.1 and later submit the same VO at step 4.3, even if the VO could be correctly verified through the on-chain Merkle root. Without taking care of this, the vote may be counted twice, which violates the ‘per vote per reviewer’ rule. It is hard for contract to detect the double-vote misbehavior because the contract has no knowledge of the off-chain chunks. Even if knows all chunks, it would be quite expensive to verify the double-vote misbehavior on the chain. Therefore, we design a function that could be called by either client or any reviewer to detect such a misbehavior off the chain and report it with a proof to during epoch-1 of the audit phase that follows the review phase so that the misbehavior could be efficiently verified by on the chain. We show the pseudo-code of at Algorithm 1. The function first verifies the timestamp of the transaction is within epoch-1 of the audit phase (line 1). Then, it computes the hash of the input chunk (i.e., chunk) that contains the repeated VO, retrieves the Merkle root uploaded by client at step 4.2 and verifies the input Merkle proof (i.e., proof) (line 2-4). After that, function would retrieve the repeated VO from the chunk (i.e., VO in the chunk) and decompose that VO to and (line 5), from which the contract gets the address of who voted via off-chain aggregation (line 6). Finally, if the contract finds that has also voted at step 4.3, it will mark as dishonest and record the address of the reporter and remove the votes cast by from the poll. Later, the reporter could withdraw an award confiscated from paid by the violator.
Iv-C Type cost-cutting strategy
Our strategy of cutting the cost of TYPE steps consists of two components, illustrated as step 5 and step 7 in Fig. IV-B, respectively.
Off-chain computation: By default, the NF-Crowd protocol encourages computations to be performed off the chain. Therefore at step 5 in Fig. IV-B, client could compute the winners off the chain after combining VOs received at step 4.2 and VOs uploaded at step 4.3 (if any) and simply upload the addresses of winning designers onto the chain via function offChainVerdict(). If no one challenges the results in a certain period of time, the contract would take the results as the final verdict. In this way, the expensive on-chain computation could be offloaded to the off-chain side and the cost of uploading computation results is usually a small constant value.
Countermeasure against incorrect off-chain computation
: It is important that the offloaded computation could be reloaded onto the blockchain so that off-chain computation results could be replaced with trustworthy on-chain computation results at any moment. By always backing up off-chain computation with on-chain computation, incorrect off-chain computation results would never be adopted in the final verdict. The procedure of reloading off-chain computation consists of two steps. First, as presented in Theorem 1, a TYPEstep is associated with at least one TYPE step, so that any data offloaded via the TYPE cost-cutting strategy presented in the Section 4.2 needs to be reloaded onto the chain. After that, computations could be performed on the reloaded data just as in the strawman protocol. For instance, at step 7 in Fig. IV-B, a reviewer decides to challenge uploaded by client at step 5. To do this, the reporter should first reload all onto the chain via function . We show the pseudo-code of at Algorithm 2. The function first verifies the timestamp (line 1) and Merkle proof (line 2-4). Then, the function runs a for loop to traverse VOs included in the input chunk (line 5-9), recover the address of reviewer signing each VO (line 6-7) and reload all the votes inside the chunk onto the chain (line 8). Then, the computation could be re-done on the chain via function . If the results of are different from the ones in offchainVerdict(), the final verdict would take the results of and a part of paid by client would be transferred to the reporter as an award. Otherwise, the final verdict would still take the results of offchainVerdict().
Iv-D Cost and Security analysis
We analyze the cost and security of the proposed NF-Crowd CC-OCR protocol as follows:
The NF-Crowd CC-OCR protocol has a total cost in the range of and the lower bound could be reached as long as the participants are rational.
The total cost would reach the upper bound when (1) all EOs are uploaded via onto the blockchain and (2) all VOs are uploaded via or onto the blockchain and (3) function is called for each VO and (4) function onchainVerdict() is invoked, namely,
where and denotes cost of uploading per EO and VO, and denotes percentage of designers and reviewers in the crowd that engage in the contest, and express cost of and onchainVerdict() and finally represents the total cost of calling other functions inside . In contrast, the total cost would reach the lower bound when none of ,, , and onchainVerdict() have been invoked, namely . All the five functions are countermeasures against dishonest participants by fixing problems made by them and confiscating paid by them, so the violators would gain no positive benefit but only a negative payoff. Considering that rational adversaries choose to violate protocols only when doing so brings them a positive payoff [dong2017betrayal, groce2012fair, guo2016rational, nguyen2013analyzing], rational participants would not choose to lose , which in turn would push the total cost to reach its lower bound. ∎
We define the abortion of protocols to be the case that neither offchainVerdict() nor onchainVerdict() has been executed by the end of audit phase. We define the correction of results to be the case that the effect of all the three identified protocol violations (intentional exclusion, double vote and incorrect off-chain computation) to the results has been eliminated by the end of audit phase.
The NF-Crowd CC-OCR protocol cannot be aborted and the results are guaranteed to be correct as long as at least one reviewer is honest.
We have assumed that at least one reviewer is honest in Seection II-E. Therefore, in the worst case, the client and all but one reviewers are dishonest. We first prove the never-abort property. During the review phase, the client may refuse to call offchainVerdict(), resulting in incorrect off-chain computation. As a countermeasure, the honest reviewer could always reload all votes via reloadChunkVO() and pick the winners via onchainVerdict(). Therefore, with the existence of a single honest reviewer, it is guaranteed that either offChainVerdict() or onchainVerdict() will be called by the end of audit phase. Next, we prove the always-correct property. As long as there is a single honest reviewer, this reviewer always reserves the ability to include her vote into the pool via submitVO() (intentional exclusion), to eliminate all illegal on-chain votes via doubleVotes() (double vote), to reload all legal off-chain votes onto the chain via reloadChunkVO() and finally to pick the winners from the pool formed by all legal votes via onchainVerdict() (incorrect off-chain computation). Therefore, the effect of all the three identified protocol violations to the results can be eliminated before end of audit phase. ∎
V Implementation and evaluation
In this section, we present the evaluation of the two protocols: strawman CC-OCR and NF-Crowd CC-OCR. We programmed a smart contract for each of the protocols using Solidity and evaluated the protocols over the Ethereum official test network kovan [kovan]. Similar to recent work on blockchain-based platforms [das2018yoda, dziembowski2019perun], the key focus of our evaluation is on measuring gas consumption as the execution complexity and monetary cost in Ethereum are measured via gas consumption. In addition, we compare the cost of the proposed protocols with that of two existing solutions.
V-a Gas consumption of proposed protocols
In Table II, we list the key functions in the programmed contracts that interact with protocol participants during different phases and the cost of these functions in both Gas and USD. For ease of expression, a function marked , or represents the function is included in strawman CC-OCR, NF-Crowd CC-OCR or both the two protocols, respectively. The cost in USD is computed through =, where GasToEther and EtherToUSD are taken as their mean value during the first half of the year 2019 recorded in Etherscan [etherscan], which are Ether/Gas and 175 USD/Ether, respectively. We next evaluate the cost of the two protocols:
Strawman CC-OCR protocol: The cost in Strawman CC-OCR includes the following components: (1) a client to set up a new contest via ($0.53) during CC-OCR.initial; (2) each interested designer to submit an entry via ($0.42) during CC-OCR.entry; (3) each reviewer to cast a vote via ($0.18) during CC-OCR.review and (4) client to pick winners via . It thus costs about for completing a contest that involves designers and reviewers, where is the average cost of picking winners among designers in .
|initial||creat a contest||
|entry||submit EO root||45322||$0.13|
|submit one EO||143978||$0.42|
|review||submit VO root||65956||$0.19|
|submit one VO||62267||$0.18|
|submit results of off-chain verdict||44967||$0.13|
|directly execute on-chain verdict||
|audit||report double vote||65844||$0.19|
|reload VO chunks to blockchain||
|redo verdict with smart contract||
NF-Crowd CC-OCR protocol: The lower bound of the cost in NF-Crowd CC-OCR also consists of four parts: (1) a client to set up a new contest via ($0.71) during CC-OCR.initial, which is more expensive because of the additionally transferred ; (2) the client to upload the Merkle root of via ($0.13) during CC-OCR.entry; (3) the client to upload the Merkle root of via ($0.19) during CC-OCR.review and finally (4) the client to upload winners via offChainVerdict () ($0.13). The lower bound of the cost is then . If any misbehavior occurs, the countermeasure functions (i.e., , , , , ) can be invoked and the cost for calling them will be mainly deducted from paid by protocol violators. It is worth noting that the cost of function , , increases along with the number of carried by a (i.e., ). Grouping VOs into could effectively reduce the cost of reloading VOs. For instance, when , the cost of reloading 100 VOs would be , which is almost twice of with .
V-B Comparison between NF-Crowd and existing solutions
Next, we compare the cost of strawman CC-OCR and NF-Crowd CC-OCR with both the platform fee charged by centralized 99designs platform and the cost of a recent decentralized protocol in [duan2019aggregating]. We assume that the design contest has a $500 reward. Also, for the purpose of evaluating the scalability of the solutions, we changed the scale of the crowd and displayed the results in Fig. 5. As can be seen, when is increased from 10 to 1000, the cost of strawman CC-OCR linearly increases from to over . In contrast, the cost of NF-Crowd CC-OCR stays at constantly regardless of the scale of the crowd. The cost of 99designs stays at $75, namely 15% of $500 reward. In fact, from reward onwards it is more economical to use NF-Crowd CC-OCR than 99designs. The cost of the decentralized protocol in [duan2019aggregating] increases from to when the crowd gets scaled up. Therefore, we see that NF-Crowd protocols minimize the cost by decoupling the amount of service fees from both the scale of the crowd and the amount of reward.
Vi Related work
Vi-a Centralized crowdsourcing
Crowdsourcing has been emerging as a successful business model and has driven the rise of centralized crowdsourcing platforms such as Upwork [Upwork], Amazon Mechanical Turk [AMT], 99designs [99designs] and designContest [designContest]. Via these platforms, clients publish human intelligence tasks (HIT) that are challenging for computers but easy for human to complete with rewards and interested workers (or freelancers or designers) accomplish the tasks to earn rewards. As the primary revenue stream, service fees are charged from the rewards by most of these platforms. Upwork [Upwork] charges workers a sliding service fee based on the lifetime billings with a specific client, which consists of 20% of the first $500, 10% of the billings between $500 and $10000 and 5% of the billings that exceed $10000. For instance, Amazon Mechanical Turk (AMT) [AMT] charges clients 20% fee on the reward and bonus amount (if any) clients pay workers. Currently, Clients and designers are used to accepting the high service fees as they need a relatively trustworthy intermediary to exclude dishonest behaviors [zhang2012reputation]. The NF-Crowd protocols proposed in this paper minimize the fees for purchasing trust in crowdsourcing. On the other hand, it makes the off-chain workload heavier. In other words, the NF-Crowd protocols transform the mandatory charge of monetary fees into non-monetary off-chain workload, offering a new option to participants in crowdsourcing.
Vi-B Decentralized crowdsourcing
Recent advancements in blockchain technologies [nakamoto2008bitcoin] and smart contract platforms like Ethereum [wood2014ethereum] are driving the rise of decentralized crowdsourcing systems [calado2018tamper, duan2019aggregating, feng2019mcs, li2018crowdbc, lu2018zebralancer, wang2018blockchain, wu2019bptm, xu2019blockchain]. In contrast to centralized crowdsourcing platforms, decentralized crowdsourcing systems leverage the distributed miners to decentralize both data storage and computation in a tamper-resilient manner. Thus they eliminate the need for a trusted third party. However, existing designs of decentralized crowdsourcing systems can hardly handle transaction fees in a scalable manner, resulting in total costs of decentralizing crowdsourcing even higher than service fees charged by centralized crowdsourcing platforms. For instance, decentralized CrowdBC [li2018crowdbc] charges 0.011 ether (i.e., $1.93 by taking the average gas and ether prices in first half of year 2019) to tag 100 images while the same task only spends about $0.45 in centralized AMT [AMT]. In [duan2019aggregating], aggregating data from 1,000 data providers costs $140. In [wu2019bptm], adding each task to the task matching smart contract cost about 210,000 gas (i.e., $0.61), namely $610 for adding 1,000 tasks. The NF-Crowd protocols proposed in this paper reduce the cost of running decentralized projects on top of Ethereum to a small constant value regardless of the scale of the crowd, which for the first time demonstrates a significant economic advantage in decentralizing crowdsourcing. In addition, we consider NF-Crowd a generic feature that is orthogonal to other design goals of decentralized crowdsourcing systems, allowing existing protocols to also become NF-Crowd by simply identifying TYPE and TYPE steps and reducing their cost with our strategies.
Vi-C Scaling blockchain with off-chain execution
Off-chain execution of smart contracts is a promising solution for improving blockchain scalability [cheng2019ekiden, das2019fastkitten, dziembowski2018general]. However, recent works in this line have to either assume one honest manager for off-chain execution [cheng2019ekiden] or allow the execution to get aborted when the manager is dishonest [das2019fastkitten]. The state channel network (SCN) [dziembowski2018general] could achieve the never abort property when at least one participant is honest but it only supports two-participant contracts. The NF-Crowd protocols proposed in this paper extend the objective to support complex multi-participant multi-round smart contracts without losing the never abort property. In addition, the NF-Crowd protocols for CC-OCR projects have been implemented in Ethereum, so they are ready-to-use.
Vi-D Using cryptocurrency as security deposits
There have been many recent efforts on blockchain-based protocol design that leverage cryptocurrency as security deposits to penalize unexpected behaviors and improve security [andrychowicz2014secure, dong2017betrayal, kiayias2015traitor, matsumoto2017ikp]. In [dong2017betrayal], ether is used as security deposits to provide verifiable cloud computing. In [matsumoto2017ikp], ether is used as security deposits to enforce certificate authorities to be honest. Inspired by these previous efforts, NF-Crowd demands each participant to lock ether in smart contracts as security deposits to penalize potential misbehaviors violating the protocol and thereby enforces participants to stay honest.
This paper proposes a new suite of protocols called NF-Crowd that reliably resolves the scalability issues faced by decentralized crowdsourcing projects. The proposed approach reduces the lower bound of the total cost to . We prove that as long as participants of a project powered by NF-Crowd are rational, the lower bound of the cost could be reached regardless of the scale of the crowd. We also demonstrate that as long as at least one participant of a project powered by NF-Crowd is honest, the project cannot be aborted and the results are guaranteed to be correct. We design NF-Crowd protocols for a representative type of project named crowdsourcing contest with open community review (CC-OCR). We implement the protocols over the Ethereum official test network. Our results demonstrate that NF-Crowd protocols can reduce the cost of running a CC-OCR project to less than $2 regardless of the scale of the crowd, providing a significant cost benefit in adopting decentralized crowdsourcing solutions.
We thank our shepherd, Alysson Bessani and the anonymous reviewers for their feedback and comments. Chao Li acknowledges the partial support by Fundamental Research Funds for the Central Universities (No. 2019RC038).