Real-time gross settlement systems (RTGS) [rtgs-2] constitute the most common approach to financial payments in closed banking networks, that is, between reputable institutions. In contrast, blockchain platforms have proposed a radically different paradigm, allowing account holders to interact directly with an online, yet highly secure, distributed ledger. Blockchain approaches aim to enable new use cases such as personal e-wallets or private transactions, and generally provide ecosystems more favorable to users. However, until now, such open, distributed settlement solutions have come at a high performance cost and questionable scalability compared to traditional, closed RTGS systems.
FastPay is a Byzantine Fault Tolerant (BFT) real-time gross settlement (RTGS) system. It enables authorities to jointly maintain account balances and settle pre-funded retail payments between accounts. It supports extremely low-latency confirmation (sub-second) of eventual transaction finality, appropriate for physical point-of-sale payments. It also provides extremely high capacity, comparable with peak retail card network volumes, while ensuring gross settlement in real-time. FastPay eliminates counterparty and credit risks of net settlement and removes the need for intermediate banks, and complex financial contracts between them, to absorb these risks. FastPay can accommodate arbitrary capacities through efficient sharding architectures at each authority. Unlike any traditional RTGS, and more like permissioned blockchains, FastPay can tolerate up to Byzantine failures out of a total of authorities, and retain both safety, liveness, and high-performance.
FastPay can be deployed in a number of settings. First, it may be used as a settlement layer for a native token and crypto-currency, in a standalone fashion. Second, it may be deployed as a side-chain of another crypto-currency, or as a high performance settlement layer on the side of an established RTGS to settle fiat retail payments. In this paper we present this second functionality in detail, since it exercises all features of the system, both payments between FastPay accounts, as well as payments into and out of the system.
Contributions. We make the following contributions:
The FastPay design is novel in that if forgoes full consensus; it leverages the semantics of payments to minimize shared state between accounts and to increase the concurrency of asynchronous operations; and it supports sharded authorities.
We provide proofs of safety and liveness in a Byzantine and fully asynchronous network setting. We show that FastPay keeps all its properties despite the lack of total ordering, or asynchrony of updates to recipient accounts.
We experimentally demonstrate comparatively very high throughput and low latency, as well as the robustness of the system under conditions of extremely high concurrency and load.
Outline. This paper is organized as follows: Section 2 introduces real-time gross settlement systems, and permissioned blockchains. Section 3 introduces the entities within FastPay, their interactions, and the security properties and threat model. Section 4 details the design of FastPay both as a standalone system, and operated in conjunction with a Primary. Section 5 discusses safety and liveness. Section 6 briefly describes the implementation of the FastPay prototype. Section 7 provides a full performance evaluation of FastPay as we modulate its security parameters and load. Section 8 discusses key open issues such as privacy, governance mechanisms and economics of the platform. Section 9 covers the related work, both in terms of traditional financial systems and crypto-currencies. Section 10 concludes.
Real-time gross settlement systems (RTGS) [rtgs-2] are the backbone of modern financial systems. Commercial banks use them to maintain an account with central banks and settle large value payments.
RTGS systems are limited in their capacity111For example, the relatively recent European Central Bank TARGET2 system has a maximum capacity of 500 transactions per second [target2], making them unsuitable for settling low-value high-volume retail payments directly. Such retail payments are deferred: banks exchange information bilaterally about pending payments (often through SWIFT [swift, swift-iso]), they aggregate and net payments, and only settle net balances through an RTGS, often daily. The often quoted volume figure of around 80,000 transactions per second for retail card networks [mastercard-performance, visa-performance] represents the rate at which ‘promises’ for payments are exchanged between banks, and not settled payments.
Traditional RTGS systems are implemented as monolithic centralized services operated by a single authority, and must employ a number of technical and organizational internal controls to ensure they are reliable (through a primary-replica architecture with manual switch over) and correct—namely ensuring availability and integrity. Traditionally only regulated entities have accounts in those systems. This result in a Balkanized global financial system where financial institutions connect to multiple RTGS, directly or indirectly through corresponding banks, to execute international payments.
Blockchain-based technologies, starting with Bitcoin [bitcoin] in 2009, provide more open settlement systems often combined with their own digital tokens to represent value. Permissionless blockchains have been criticized for their low performance [croman2016scaling] in terms of capacity and finality times. However, a comparison with established settlement systems leads to a more nuanced assessment. Currently, Ethereum [wood2014ethereum] can process 15 transactions per second. The actual average daily load on the EU ECB TARGET2 system is about 10 transactions per second [target2] (in 2018) which is a comparable figure (and lower than the peak advertised capacity of 500 transaction per second). However, it falls very short of the advertised transaction rate of 80,000 transaction per second peak for retail payment networks—even though this figure does not represents settled transactions. The stated ambitions of permissionless projects is to be able to settle transactions at this rate on an open and permissionless network, which remains an open research challenge [vukolic2015quest].
Permissioned blockchains [permissioned-blockchains, sok-consensus] provide a degree of decentralization—allowing multiple authorities to jointly operate a ledger—at the cost of some off-chain governance to control who maintains the blockchain. The most prominent of such proposals is the Libra network [libra], developed by the Libra association. Other technical efforts include Hyperledger [hyperledger], Corda [corda] and Tendermint [tendermint]. Those systems are based on traditional notions of Byzantine Fault Tolerant state machine replication (or sometimes consensus with crash-failures only), which presupposes an atomic commit channel (often referred as ‘consensus’) that sequences all transactions. Such architectures allow for higher capacities than Bitcoin and Ethereum. LibraBFT, for example, aims for 1,000 transactions per second at peak capacity [libraTPS-1, libraTPS-2]; this exceeds many RTGS systems, but is still below the peak volumes for retail payment systems. A latency of multiple seconds when it comes to transaction finality is competitive with RTGS, but is unusable for retail payment at physical points of sale.
To illustrate its full capabilities, we describe FastPay as a side chain of a primary RTGS holding the primary records of accounts. We call such a primary ledger the Primary for short, and its accounts the Primary accounts. The Primary can be instantiated in two ways: (i) as a programmable blockchain, through smart contracts, like Ethereum [wood2014ethereum] or Libra [libra]. The Primary can also be instantiated (ii) as a traditional monolithic RTGS operated by a central bank. In this case the components interfacing with FastPay are implemented as database transactions within the Primary. In both cases FastPay acts as a side infrastructure to enable pre-funded retail payments. FastPay can also operate with a native asset, without a primary ledger. In this case sub-protocols involving the Primary are superfluous, since all value is held within FastPay accounts and never transferred out or into the system.
FastPay involves two types of participants: (i) authorities, and (ii) account owners (users, for short). All participants generate a key pair consisting of a private signature key and the corresponding public verification key. As a side-chain, FastPay requires a smart contract on the main blockchain, or a software component on an RTGS system that can authorize payments based on the signatures of a threshold of authorities from a committee with fixed membership.
By definition, an honest authority always follows the FastPay protocol, while a faulty (or Byzantine) one may deviate arbitrarily. We present the FastPay protocol for equally-trusted authorities, assuming a fixed (but unknown) subset of at most Byzantine authorities. In this setting, a quorum is defined as any subset of authorities. (As for many BFT protocols, our proofs only use the classical properties of quorums thus apply to all Byzantine quorum systems [malkhi1998byzantine].)
When a protocol message is signed by a quorum of authorities, it is said to be certified: we call such a jointly signed message a certificate.
3.2 Accounts and Actions
A FastPay account is identified by its address, which we instantiate as the cryptographic hash of its public verification key. The state of a FastPay account is affected by four main high-level actions:
Receiving funds from a Primary account.
Transferring funds to a Primary account.
Receiving funds from a FastPay account.
Transferring funds to a FastPay account.
FastPay also supports two read-only actions that are necessary to ensure liveness despite faults: reading the state of an account at a FastPay authority, and obtaining a certificate for any action executed by an authority.
3.3 Protocol Messages
The FastPay protocol consists of transactions on the Primary, denoted with letter , and network requests that users send to FastPay authorities, which we call orders, and denote with letter . Users are responsible for broadcasting their orders to authorities and for processing the corresponding responses. The authorities are passive and do not communicate directly with each other.
Transfer orders. All transfers initiated by a FastPay account start with a transfer order including the following fields:
The sender’s FastPay address, written .
The recipient, either a FastPay or a Primary address, written .
A non-negative amount to transfer, written .
A sequence number .
Optional user-provided data.
A signature by the sender over the above data.
Authorities respond to valid transfer orders by counter-signing them (see next section for validity checks). A quorum of such signatures is meant to be aggregated into a transfer certificate, noted .
Notations. We write for the original transfer order certified by . For simplicity, we omit the operator value when the meaning is clear, e.g. . FastPay addresses are denoted with letters and . We use for authorities and by extension for the shards of authorities.
3.4 Security Properties and Threat Model
FastPay guarantees the following security properties:
Safety: No units of value are ever created or destroyed; they are only transferred between accounts.
Authenticity: Only the owner of an account may transfer value out of the account.
Availability: Correct users can always transfer funds from their account.
Redeemability: A transfer to FastPay or Primary is guaranteed to eventually succeed whenever a valid transfer certificate has already been produced.
Public Auditability: There is sufficient public cryptographic evidence for the state of FastPay to be audited for correctness by any party.
Worst-case Efficiency: Byzantine authorities (or users) cannot significantly delay operations from correct users.
The above properties are maintained under a number of security assumptions: (i) there are at most Byzantine authorities out of total authorities. (ii) The network is fully asynchronous, and the adversary may arbitrarily delay and reorder messages [dwork1988consensus]. However, messages are eventually delivered. (iii) Users may behave arbitrarily but availability only holds for correct users (defined in Section 4.5). (iv) The Primary provides safety and liveness (when FastPay is used in conjunction with it). We further discuss the security properties of FastPay in Section 5.
4 The FastPay Protocol
FastPay authorities hold and persist the following information.
Authorities. The state of an authority consists of the following information:
The authority name, signature and verification keys.
The committee, represented as a set of authorities and their verification keys.
A map tracking the current account state of each FastPay address in use (see below).
An integer value, noted , referring to the last transaction that paid funds into the Primary. This is used by authorities to synchronize FastPay accounts with funds from the Primary (see Section 4.3).
FastPay accounts. The state of a FastPay account within the authority consists of the following:
The public verification key of , used to authenticate spending actions.
An integer value representing the balance of payment, written .
An integer value tracking the expected sequence number for the next spending action to be created, written . This value starts at .
A field tracking the last transfer order signed by such that the authority considers as pending confirmation, if any; and absent otherwise.
A list of certificates, written , tracking all the transfer certificates that have been confirmed by and such that . One such certificate is available for each sequence number ().
A list of synchronization orders, , having transferred funds from the Primary to account . (See Section 4.3.)
We also define as the list of confirmed certificates for transfers received by . Formally, .
We assume arbitrary size integers. Although FastPay does not let users overspend, (temporary) negative balances for account states are allowed for technical reasons discussed in Section 5. When present, a pending (signed) transfer order effectively locks the sequence number of the account and prevents from accepting new transfer orders from until confirmation, that is, until a valid transfer certificate such that is received. This mechanism can be seen as the ‘Signed Echo Broadcast’ implementation of a Byzantine consistent broadcast on the label (account, next sequence number) [cachinBook].
Storage considerations. The information contained in the lists of certificates and and in the synchronization orders is self-authenticated—being signed by a quorum of authorities and by the Primary, respectively. Remarkably, this means that authorities may safely outsource these lists to an external high-availability data store. Therefore, FastPay authorities only require a constant amount of local storage per account, rather than a linear amount in the number of transactions.
4.1 Transferring Funds within FastPay
FastPay operates by implementing a Byzantine consistent broadcast channel per account, specifically using a ‘Signed Echo Broadcast’ variant (Algorithm 3.17 in [cachinBook]). It operates in two phases and all messages are relayed by the initiating user. Consistent Broadcast ensures Validity, No duplication, Integrity, and Consistency. It always terminates when initiated by a correct user. However, if a FastPay user equivocates, current operations may fail, and the funds present on the users account may become inaccessible.
Transferring funds. Figure 1 illustrates a transfer of funds within FastPay. To transfers funds to another FastPay account, the sender creates a FastPay transfer order () with the next sequence number in their account, and signs it. They then send the FastPay transfer order to all authorities. Each authority checks (➊) (i) that the signature is valid, (ii) that no previous transfer is pending (or is for the same transfer), (iii) that the amount is positive, (iv) that the sequence number matches the expected next one (), and (iv) that the balance () is sufficient (➋). Then, it records the new transfer as pending and sends back a signature on the transfer order (➌) which is also stored. The authority algorithm to handle transfer orders, corresponding to step ➋, is presented in Figure 2.
The user collects the signatures from a quorum of authorities, and uses them along the FastPay transfer order to form a transfer certificate. The sender provides this transfer certificate to the recipient as proof that the payment will proceed (➎). To conclude the transaction, the sender (➍) or the recipient (➏) must broadcast the transfer certificate () to the authorities (we call this a confirmation order)222Aggregating signed transfer orders into a transfer certificate does not requires knowledge of any secret; therefore, anyone (and not only the sender or the recipient) can broadcast the transfer certificate to the authorities to conclude the transaction..
Upon reception of a confirmation order for the current sequence number, each authority (➐) (i) checks that a quorum of signatures was reached, (ii) decreases the balance of the sender, (iii) increments the sequence number () to ensure ‘deliver once’ semantics, and (iv) sets the pending order to None (). Each authority also (v) adds the certificate to the list , and (vi) increases the balance of the recipient account asynchronously (i.e. without sequencing this write in relation to any specific payments from this account across authorities). The authority algorithm to handle confirmation orders (as in step ➐) is presented in Figure 2.
In Section 5 and Appendix A, we show that the FastPay protocol is safe thanks to the semantics of payments into an account and their commutative properties. FastPay is a significant simplification and deviation from an orthodox application of Guerraroui et al. [consensus-number], where accounts are single-writer objects and all write actions are mediated by the account owner. FastPay allows payments to be executed after a single consistent broadcast, rather than requiring recipients to sequence payments into their accounts separately. This reduces both latency and the state necessary to prevent replays.
Payment finality. Once a transfer certificate could be formed, namely authorities signed a transfer order, no other order can be processed for an account until the corresponding confirmation order is submitted. Technically, the payment is final: it cannot be canceled, and will proceed eventually. As a result, showing a transfer certificate to a recipient convinces them that the payment will proceed. We call the showing of a transfer certificate to a recipient a confirmation, and then subsequently submitting the confirmation order, to move funds, settlement. Confirmation requires only a single round trip to a quorum of authorities resulting in very low-latency (see Section 7), and giving the system its name.
Proxies, gateways and crash recovery. The protocols as presented involve the sender being on-line and mediating all communications. However, the only action that the sender must perform personally is forming a transfer order, requiring their signature. All subsequent operations, including sending the transfer order to the authorities, forming a certificate, and submitting a confirmation order can be securely off-loaded to a proxy trusted only for liveness. Alternatively, a transfer order may be given to a merchant (or payment gateway) that drives the protocol to conclusion. In fact, any party in possession of a signed transfer order may attempt to make a payment progress concurrently. And as long as the sender is correct the protocol will conclude (and if not may only lock the account of the faulty sender).
This provides significant deployment and implementation flexibility. A sender client may be implemented in hardware (in a NFC smart card) that only signs transfer orders. These are then provided to a gateway that drives the rest of the protocol. Once the transfer order is signed and handed over to the gateway, the sender may go off-line or crash. Authorities may also attempt to complete the protocol upon receiving a valid transfer order. Finally, the protocol recovers from user crash failures: anyone may request a transfer order that is partially confirmed from any authority, proceed to form a certificate, and submit a confirmation order to complete the protocol.
4.2 Sharding authorities
FastPay requires minimal state sharing between accounts, and allows for a very efficient sharding at each authority by account. The consistent broadcast channel is executed on a per-account basis. Therefore, the protocol does not require any state sharing between accounts (and shards) up to the point where a valid confirmation order has to be settled to transfer funds between FastPay accounts. On settlement, the sender account is decremented and the funds are deposited into the account of the recipient, requiring interaction between at most two shards (second algorithm of Figure 2).
Paying into an account can be performed asynchronously, and is an operation that cannot fail (if the account does not exist it is created on the spot). Therefore, the shard managing the recipient account only needs to be notified of the confirmed payment through a reliable, deliver once, authenticated, point to point channel (that can be implemented using a message authentication code, inter-shard sequence number, re-transmission, and acknowledgments) from the sender shard. This is a greatly simplified variant of a two-phase commit protocol coordinated by the sender shard (for details see the Presume Nothing and Last Agent Commit optimizations [DBLP:journals/dpd/SamarasBCM95, DBLP:conf/vldb/LampsonL93]). Modifying the validity condition of the consistent broadcast to ensure the recipient account exists (or any other precondition on the recipient account) would require a full two-phase commit before an authority signs a transfer order, and can be implemented while still allowing for (slightly less) efficient sharding.
The algorithms in fig. 2 implement sharding. An authority shard checks that the transfer order () or certificate () is to be handled by a specific shard and otherwise rejects it without mutating its state. Handling confirmation orders depends on whether a recipient account is on the same shard. If so, the recipient account is updated locally. Otherwise, a cross shard message is created for the recipient shard to update the account (see code in the Appendix for this operation).
The ability to shard each authority has profound implications: increasing the number of shards at each authority increases the theoretical throughput linearly, while latency remains constant. Our experimental evaluation confirms this experimentally (see Section 7).
4.3 Interfacing with the Primary
We describe the protocols required to couple FastPay with the Primary, namely transferring funds from the Primary to a FastPay account, and conversely from a FastPay to a Primary account. We refer throughout to the logic on the Primary as a smart contract, and the primary store of information as the blockchain. A traditional RTGS would record this state and manage it in conventional ways using databases and stored procedures, rather than a blockchain and smart contracts. We write for the state of the ‘blockchain’ at a given time, and for the set of FastPay transactions already processed by the blockchain.
Smart contract. The smart contract mediating interactions with the Primary requires the following data to be persisted in the blockchain:
The FastPay committee composition: a set of authority names and their verification keys.
A map of accounts where each FastPay address is mapped to its current Primary state (see below).
The total balance of funds in the smart contract, written .
The transaction index of the last transaction that added funds to the smart contract, written .
Accounts. The Primary state of a FastPay account consists of the set of sequence numbers of transfers already executed from this account to the Primary. This set is called the redeem log of and written .
Adding funds from the Primary to FastPay. Figure 3 shows a transfer of funds from the Primary to FastPay. The owner of the FastPay account (or anyone else) starts by sending a payment to the FastPay smart contract using a Primary transaction (➊). This transaction is called a funding transaction, and includes the recipient FastPay address for the funds and the amount of value to transfer.
When the Primary transaction is executed, the FastPay smart contract generates a Primary event that instructs authorities of a change in the state of the FastPay smart contract. We assume that each authority runs a full Primary client to authenticate such events. For simplicity, we model such an event as a (Primary) synchronization order (➋). The smart contract ensures this event and the synchronization order contain a unique, always increasing, sequential transaction index.
When receiving a synchronization order, each authority (i) checks that the transaction index follows the previously recorded one, (ii) increments the last transaction index in their global state, (iii) creates a new FastPay account if needed, and (iv) increases the account balance of the target account by the amount of value specified (➌). Appendix B presents the authority algorithm for handling funding transactions.
Transferring funds from FastPay to the Primary. Figure 4 shows a transfer of funds from FastPay to the Primary. The FastPay sender signs a Primary transfer order using their account key and broadcasts it to the authorities (➊). This is simply a transfer order with a Primary address as the recipient.
Once a quorum of signatures is reached (➋ and ➌), the sender creates a certified (Primary) transfer order, also called a transfer certificate for short. The sender broadcasts this certificate to the authorities to confirm the transaction (➍) and unlock future spending from this account. When an authority receives a confirmation order containing a certificate of transfer (➎), it must check (i) that a quorum of signatures was reached, and (ii) that the account sequence number matches the expected one; they (iii) then set the pending order to None, (iv) increment the sequence number, and (v) decrease the account balance.
Finally, the recipient of the transfer should send a redeem transaction to the FastPay smart contract on the Primary blockchain (➏). When the FastPay smart contract receives a valid redeem transaction (➐), it must (i) check that the sequence number is not in the Primary redeem log of the sender, to prevent reuse; (ii) update this redeem log; (iv) transfer the amount of value specified from the smart contract into the recipient’s Primary account.
4.4 State Recovery and Auditing
For every account , each authority must make available the pending order , the sequence number , the synchronization orders , and the certificates confirmed so far, indexed by senders (i.e. ) and receivers (). Sharing these data fulfills two important roles: (i) this lets anyone read the state of any incomplete transfer and drive the protocol all the way to settlement; (ii) it enables auditing authority states and detecting Byzantine faults (e.g. incorrect balance checks).
4.5 Correct Users and Client Implementation
A correct user owning a FastPay account follows the correctness rules below:
The user sets the sequence number of a new transfer order to be the next expected integer after the previous transfer (starting with ); i.e. they sign exactly one transfer order per sequence number;
They broadcast the new transfer order to enough authorities until they (eventually) obtain a certificate ;
They successfully broadcast the certificate to a quorum of authorities.
FastPay Client. To address the correctness rules above, our reference implementation of a FastPay client holds and persists the following minimal state:
The address and the secret key of the account;
The FastPay committee;
The sequence number to be used in the next transfer;
The transfer order that it signed last, in case it is still pending.
In this setting, the available balance of a user account is not tracked explicitly but rather evaluated (conservatively) from the Primary transactions and the available logs for incoming transfers and outgoing transfers (Section 4.4). Evaluating the balance before starting a transfer is recommended, as signing a transfer order with an excessive amount will block (correct) client implementations from initiating further transfers until the desired amount is available.
5 Security Analysis
Let denote the current state of the Primary. We define as the sum of all the amounts transferred to a FastPay address from the Primary:
For simplicity, we write when we mean to sum over certified transfer orders: .
The results presented in this section are proven in Appendix A. We start with the main safety invariant of FastPay.
Theorem 1 (Solvency of FastPay).
At any time, the sum of the amounts of all existing certified transfers from FastPay to the Primary cannot exceed the funds collected by all transactions on the Primary smart contract:
Next, we describe how receivers of valid transfer certificates can finalize transactions and make funds available on their own accounts (Primary and FastPay).
Proposition 1 (Redeemability of valid transfer certificates to Primary).
A new valid Primary transfer certificate can always be redeemed by sending a new redeem transaction to the smart contract.
Proposition 2 (Redeemability of valid transfer certificates to FastPay).
Any user can eventually have a valid FastPay transfer certificate confirmed by any honest authority.
Specifically, in Proposition 2, the confirmation order for is guaranteed to succeed for every honest authority , provided that the user first recovers and transfers to all the missing certificates required by , defined as the sequence such that , , , . The fact that no other certificates need to be confirmed (e.g. to credit the balance of itself) is closely related to the possibility of (temporary) negative balances for authorities, and justified by the proof of safety in Appendix A.
Note that having a FastPay certificate confirmed by an authority only affects ’s recipient and the sender’s balances (i.e. redeems the certificate) the first time it is confirmed.
Finally, we state that FastPay funds credited on an account can always be spent. We write for the set of incoming transfer certificates such that and is known to the owner of the account .
Proposition 3 (Availability of transfer certificates).
Let be an account owned by a correct user, be the next available sequence number after the last signed transfer order (if any, otherwise ), and be a new transfer order signed by with and .
Assume that the owner of has secured enough funds for a new order based on their knowledge of the chain , the history of outgoing transfers, and the set . That is, formally:
Then, for any honest authority , the user will always eventually obtain a valid signature of from after sending the following orders to :
A synchronization order from the Primary based on the known state ;
A confirmation order for every , preceded by all the missing certificates required by (if any) for the sender of ;
Then, the transfer order .
Worst-case efficiency of FastPay clients. To initiate a transfer (Proposition 3) or receive funds (Proposition 2) from a sender account , a FastPay client must address a quorum of authorities. During the exchange, each authority may require missing certificates , where is provided by . In an attempt to slow down the client, a Byzantine authority could return and/or fail to respond at some point. To address this, a client should query each authority in parallel. After retrieving the sequence number , the required missing certificates should be downloaded sequentially, in reverse order, then forwarded to . Given that FastPay client operations succeed as soon as a quorum of authorities completes their exchanges, this strategy ensures client efficiency despite Byzantine authorities.
We implemented both a FastPay client and a networked multi-core multi-shard FastPay authority in Rust, using Tokio333https://tokio.rs for networking and ed25519-dalek444https://github.com/dalek-cryptography/ed25519-dalek for signatures. For the verification of the multiple signatures composing a certificate we use ed25519 batch verification. To reduce latency we use UDP for FastPay requests and replies, and make the core of FastPay idempotent to tolerate retries in case of packet loss; we also provide an experimental FastPay implementation using exclusively TCP. Currently, data-structures are held in memory rather than persistent storage.
We implement an authority shard as a separate operating system process with its own networking and Tokio reactor core, to validate the low overhead of intra-shard coordination (through message passing rather than shared memory). We experimented with manually pinning processes to physical cores without a noticeable increase in performance through the Linux taskset feature. It seems the Linux OS does a good job in distributing processes and keeping them on inactive cores. We also experimented with a single process multi-threaded implementation of FastPay, using a single Tokio reactor for all shards on multi-core machines. However, this led to significantly lower performance, and therefore we opted for using separate processes even on a single machine for each shard. The exact bottleneck justifying this lower performance—whether at the level of Tokio multi-threading or OS resource management—still eludes us.
The implementation of both server and client is less than 4,000 LOC (of which half are for the networking), and a further 1,375 LOC of unit tests. It required about 2.5 months of work for 3 engineers, and a bit over 1,500 git commits. Keeping the core small required constant re-factoring and its simplicity is a significant advantage of the proposed FastPay design.
We evaluate the throughput and latency of our implementation of FastPay through experiments on AWS. We particularly aim to demonstrate that (i) sharding is effective, in that it increases throughput linearly as expected; (ii) latency is not overly affected by the number of authorities or shards, and remains near-constant, even when some authorities fail; and (iii) that the system is robust under extremely high concurrency and transaction loads.
We report on microbenchmarks of the single-CPU core time required to process transfer orders, authority signed partial certificates, and certificates. Table 1 displays the cost of each operation in micro seconds () assuming 10 authorities (recall ); each measurement is the result of 500 runs on an Apple laptop (MacBook Pro) with a 2.9 GHz Intel Core i9 (6 physical and 12 logical cores), and 32 GB 2400 MHz DDR4 RAM. The first 3 rows respectively indicate the time to create and serialize (i) a transfer order, (ii) a partial certificate signed by a single authority, and (iii) a transfer certificate as part of a confirmation order. The last 3 rows indicate the time to deserialize them and check their validity. The dominant CPU cost involves the deserialization and signature check on certificates (), which includes the batch verification of the 8 signatures (7 from authorities and 1 from sender). However, deserializing orders () and votes () is also expensive: it involves 1 signature verification (no batching) and creating 1 signature. Those results indicate that a single core shard implementation may only settle just over 4,000 transactions per second—highlighting the importance of sharding to achieve high-throughput.
In terms of networking costs, a transfer order is 146 bytes, and the signed response is 293 bytes. This could be reduced by only responding with a signature (64 bytes) rather than the full signed order, but we chose to echo back the order to simplify client implementations. A full certificate for an order is 819 bytes, and the response—consisting of an update on the state of the FastPay account—is 51 bytes. For deployments using many authorities we can compress certificates by using an aggregate signature scheme (such as BLS [bls]). However, verification CPU costs of BLS only make this competitive for committees larger than 50-100 authorities. We note that all FastPay message types fit within the common maximum transmission unit of commodity IP networks, allowing requests and replies to be executed using a single UDP packet (assuming no packets loss and 10 authorities).
|Measure||Mean ()||Std. ()|
|Create & Serialize Order||27||1|
|Create & Serialize Partial Cert.||27||2|
|Create & Serialize Certificate||4||0|
|Deserialize & Check Order||58||1|
|Deserialize & Check Partial Cert.||60||1|
|Deserialize & Check Certificate||236||10|
Microbenchmark of single core CPU costs of FastPay operations; average and standard deviation of 500 measurements for 10 authorities.
We deploy a FastPay multi-shard authority on Amazon Web Services (Stockholm, eu-north-1 zone), on a m5d.metal instance. This class of instance guarantees 96 virtual CPUs (48 physical cores), on a 2.5 GHz, Intel Xeon Platinum 8175, and 384 GB memory. The operating system is Linux Ubuntu server 18.04, where we increase the network buffer to about 96MB. In all graphs, each measurement is the average of 9 runs, and the error bars represent one standard deviation; all experiments use our UDP implementation. We measure the variation of throughput with the number of shards. Our baseline experiment parameters are: 4 authorities (for confirmation orders), a load of 1M transactions, and applying back-pressure to allow a maximum of 1000 concurrent transactions at the time into the system (i.e. the in-flight parameter). We then vary those baseline parameters through our experiments to illustrate their impact on performance.
Robustness and performance under high concurrency. Figures 5 and 6 respectively show the variation of the throughput of processing transfer and confirmation orders as we increase the number of shards per authority, from 15 to 85. We measure those by processing 1M transactions, across 4 authorities. Figure 5 shows that the throughout of transfer orders slowly increases with the number of shards. The in-flight parameter—the maximum number of transactions that is allowed into the system at any time—influences the throughput by about 10%, and setting it to 1,000 seems optimal for performance. The degree of concurrency in a system depends on the number of concurrent client requests, and we observe that FastPay is stable and performant even under extremely high concurrency peaks of 50,000 concurrent requests. Afterwards, the Operating System UDP network buffers fill up, and the authority network stacks simply drop the requests.
Figure 6 shows that the throughput of confirmation orders initially increases linearly with the number of shards, and then reaches a plateau at around 48 shards. This happens because our experiments are run on machines with 48 physical cores, running at full speed, and 48 logical cores. The in-flight parameter of concurrent requests does not influence the throughput much, but setting it too low (e.g. at 100) does not saturate our CPUs. These figures show that FastPay can support up to 160,000 transactions per second on 48 shards (about 7x the peak transaction rate of the Visa payments network [visa-performance]) while running on commodity computers that cost less than 4,000 USD/month per authority555AWS reports a price of 5.424 USD/hour for their m5d.metal instances. https://aws.amazon.com/ec2/pricing/on-demand (January 2020).
Robustness and performance under total system load. Figures 11 and 12 (see Appendix C) show the variation of the throughput of transfer and confirmation orders with the number of shards, for various total system loads—namely the total number of transactions in the test; they show that the throughput is not affected by the system load. The tests were performed with 4 authorities, and the client concurrency in-flight parameter set to 1,000. These figures illustrate that FastPay can process about 160,000 transactions per second even under a total load of 1.5M transactions, and that the total load does not significantly affect performance. These supplement figures 5 and 6 that illustrate the concurrent transaction rate (in-flight parameter) also does not influence performance significantly (except when it is too low by under-utilizing the system).
Readers may be surprised those measurements are key. The key measurement work by Han et al. [han2019performance] compares a number of permissioned systems under a high load, and shows that for all of Hyperledger Fabric (v0.6 with PBFT) [hyperledger-v06], Hyperledger Fabric (v1.0 with BFT-Smart) [hyperledger-v10], Ripple [ripple-documentation] and R3 Corda v3.2 [corda-v32] the successful requests per second drops to zero as the transaction rate increases to more than a few thousands transactions per second (notably for Corda only a few hundred). An important exception is Tendermint [tendermint], that maintains a processed transaction rate of about 4,000 to 6,000 transactions per second at a high concurrency rate. Those findings were confirmed for Hyperledger Fabric that reportedly starts saturating at a rate of 10,000 transactions per second [nasir2018performance]. Our results demonstrate that FastPay continues to be very performant even under the influence of extremely high rates of concurrent transactions (in-flight parameter) and overall work load (total number of transactions processed), as expected. This is apparently not the norm.
Influence of the number of authorities. As discussed in Section 4, we expect that increasing the number of authorities only impacts the throughput of confirmation orders (that need to transfer and check transfer certificates signed by authorities), and not the throughput of transfer orders. Figure 7 confirms that the the throughput of confirmation orders decreases as the number of authorities increases. FastPay can still process about 80,000 transactions per second with 20 authorities (for 75 shards). The measurements are taken with an in-flight concurrency parameter set to 1,000, and under a load of 1M total transactions. We note that for higher number of authorities, using an aggregate signature scheme (e.g. BLS [bls]) would be preferable since it would result in constant time verification and near-constant size certificates. However, due to the use of batch verification of signatures, the break even point may be after 100 authorities in terms of verification time.
We measure the variation of the client-perceived latency with the number of authorities. We deploy several FastPay multi-shard authorities on Amazon Web Services (all in Stockholm, eu-north-1 zone), each on a m5d.8xlarge instance. This class of instance guarantees 10Gbit network capacity, on a 3.1 GHz, Intel Xeon Platinum 8175 with 32 cores, and 128 GB memory. The operating system is Linux Ubuntu server 16.04. Each instance is configured to run 15 shards. The client is run on an Apple laptop (MacBook Pro) with a 2.9 GHz Intel Core i9 (6 physical and 12 logical cores), and 32 GB 2400 MHz DDR4 RAM; and connected to a reliable WIFI network. We run experiments with the client in two different locations; (i) in the U.K. (geographically close to the authorities, same continent), and (ii) in the U.S. West Coast (geographically far from the authorities, different continent). Each measurement is the average of 300 runs, and the error bars represent one standard deviation; all experiments use our UDP implementation.
We observe that the client-authority WAN latency is low for both transfer and confirmation orders; the latency is under 200ms when the client is in the U.S. West Coast, and about 50ms when the client is in the U.K. Figure 8 illustrates the latency between a client creating and sending a transfer order to all authorities, and receiving sufficient signatures to form a transfer certificate (in our experiment we wait for all authorities to reply to measure the worse case where authorities are Byzantine). The latency is virtually constant as we increase the number of authorities, due to the client emitting orders asynchronously to all authorities and waiting for responses in parallel.
Figure 9 illustrates the latency to submit a confirmation order, and wait for all authorities to respond with a success message. It shows latency is virtually constant when increasing the number of authorities. This indicates that the latency is largely dominated by the network (and not by the verification of certificates). However, since even for 10 authorities a FastPay message fits within a network MTU, the variation is very small. Due to our choice of using UDP as a transport there is no connection initiation delay (as for TCP), but we may observe packet loss under very high congestion conditions. Authority commands are idempotent to allow clients to re-transmit to overcome loss without sacrificing safety.
Performance under failures. Research literature suggests permissioned blockchains based on (often leader-based) consensus suffer an enormous performance drop when some authorities fail [DBLP:conf/icdcs/LeeSHKN14]. We measure the effect of authority failure in FastPay and show that latency is not affected when or fewer authorities are unavailable.
We run our baseline experimental setup (10 authorities distributed over 10 different AWS instances), when a different number of authorities are not available for . We measure the latency experienced by a client on the same continent (Europe), sending a transfer order until it forms a valid transfer certificate. Table 2 summarizes the mean latency and standard deviation for different . There is no statistically significant difference in latency, no matter how many tolerable failures FastPay experiences (up to for 10 authorities). We also experimented with killing authorities one by one with similar results, up to when the system did observably lose liveness as expected. The underlying reason for the steady performance under failures is FastPay’s lack of reliance on a leader to drive the protocol.
8 Limitations and Future Work
Threats to validity of experiments. Our experiments represent the best case performance, for a set number of authorities and shards, as they are performed in laboratory conditions. In particular, real-wold transactions may have the same sender account, which would prevent them from being executed in parallel. Further, the throughput evaluation places transaction load on an authority through the local network interface, and therefore does not take fully into account the operating system networking costs of a full WAN stack. Further, our WAN latency experiments were performed against authorities with very low-load. Finally, the costs of persisting databases to storage are not taken into account when measuring latency and throughput (we leave the implementation of low-latency persistent storage to future work).
Integrating privacy. As presented, FastPay exposes information about all transactions, namely the sender-recipient accounts and the amounts transferred, as well as the timings of those transfers. Fully integrating stronger privacy protections is a separate research project. However, we want to highlight that the architecture of FastPay is highly compatible with threshold issuance selective disclosure credential designs, such as Coconut [coconut]. In those schemes a threshold of authorities can jointly sign a credential that the user can subsequently randomize and present to execute a payment. Implementing hidden balances, like MinbleWinble [mimblewimble] and combining them with credentials should be possible—but beyond the scope of the present work.
Checkpointing, authority, and key rotation. The important enabler for the good performance of FastPay, but also an important limitation, is the fact that authorities do not need to reach consensus on the state of their databases. We demonstrate that payments are secure in this context, but various system maintenance operations are harder to implement. For example, checkpointing the state of all accounts in the systems, to compress the list of stored certificates would be beneficial, but cannot be straightforwardly implemented without consensus. Similarly, it would be beneficial for authorities to be able to rotate in and out of the committee, as well as to update their cryptographic signature keys. Due to the lack of tight synchronization between authorities there is no natural point that guarantees they all update their committees at the same logical time. Further, our proofs of liveness under asynchrony presume that transfer orders and certificates that were once valid, will always be valid. Integrating such governance features into FastPay will require careful design to safely leverage either some timing (synchrony) assumptions or use a more capable (but maybe lower performance) consensus layer, such as one facilitated by the Primary.
Economics and fees. Some cost to insert transactions into a system (like fees in Bitcoin), allows for sound accounting and prevents Denial of Service attacks by clients over-using an open system. The horizontal scalability of FastPay alleviates somehow the need to integrate such a scheme, since issues of capacity can be resolved by increasing its capacity through more shards (as well as deploying network level defenses). However, if there was a need to implement fees for using FastPay we would not recommend using micro-payments associated with each payment like in Bitcoin. We would rather recommend allowing a client to deposit some payment into a service account with all authorities, and then allow them to deduct locally some of this fee for any services rendered (namely any signed transfer order or confirmation order processed). In practical terms, the variable costs of processing transactions in FastPay is low. There is no artificial shortage due to lack of scalability, and a flat periodic fee on either senders or recipients might be sufficient to support operations (rather than a charge per transaction).
9 Related Works
We compare FastPay with traditional payment systems and some relevant crypto-currencies.
Traditional payment systems. In the context of traditional payment systems FastPay is a real-time gross settlement system (RTGS) [rtgs-1, rtgs-2]—payments are executed in close to real-time, there is no netting between participants, and the transfer of funds is final upon the full payment protocol terminating. All payments are pre-funded so there is no need to keep track of credit or liquidity, which makes the design vastly simpler.
FastPay, from an assurance and performance perspective is significantly superior to deployed RTGS systems: it (i) implements a fully Byzantine fault tolerant architecture (established systems rely on master-slave configurations to only recover from few crash failures), (ii) has higher throughput (as compared, for example with the TARGET2 [target2] European Central Bank RTGS systems that has a target throughput of 500 tx/sec), and (iii) has faster finality (as compared to TARGET2 providing finality of a few seconds). Since FastPay allows for fast gross settlement, participants are not exposed to credit risk, as in the case of retail payment systems such as VISA and Mastercard (that use daily netting, and have complex financial arrangements to mitigate credit risk in case of bank default). Furthermore, it does achieve both throughput and latency, comparable to those systems combined—about 80,000 tx/sec at peak times, when adding up the throughput of Visa and Mastercard together [visa-performance, mastercard-performance].
On the downside, FastPay lacks certain features of mature RTGS systems: in particular it does not support Delivery-on-Payment transactions that atomically swap securities when payment is provided, or Payment-versus-Payment, that atomically swap amounts in different currencies to minimize the risk of foreign exchange transactions. These require atomic operations across accounts controlled by different users, and would therefore require extending FastPay to support them (namely operations with consensus number of 2 per Herlihy [herlihy1991wait]), which we leave for future work.
Crypto-currencies. FastPay provides high assurance in the context of Byzantine failures within its infrastructure. In that respect it is comparable with systems encountered in the space of permissioned blockchains and crypto-currencies, as well as their eco-system of payment channels. FastPay is permissioned in that the set of authorities managing the system is closed—in fact we do not even propose a way to rotate those authorities and leave this to future work. Qualitatively, FastPay differs from other permissioned (or permissionless) crypto-currencies in a number of ways: it is secure under full network asynchrony (since it does not require or rely on atomic broadcast channels or consensus, but only consistent broadcast)—leading to higher performance. This direction was explored in the past in relation to central bank crypto-currency systems [RSCoin] and high performance permissionless systems [rocket2018snowflake]. It was recently put on a formal footing by Guerraroui et al. [consensus-number]. Our work extends this theory to allow increased concurrency, correctness under sharding, and rigorous interfacing with external settlement mechanisms. FastPay achieves auditability through a set of certificates signed by authorities rather than a sequential log of actions (blockchain), which would require authorities to reach agreement on a common sequence.
Quantitatively, compared with other permissioned systems FastPay is extremely performant. HyperLedger Fabric [hyperledger] running with 10 nodes achieves about 1,000 transactions per second and a latency of about 10 seconds [nasir2018performance]; and Libra [libra] and Corda [corda, corda-performance] achieve similar performance. JP Morgan developed a digital coin built from the Ethereum codebase, which can achieve about 1,500 transactions per second with four nodes, and imposing a block time of 1 second [baliga2018performance]. Tendermint [tendermint] reportedly achieves 10,000 transactions per second with 4 nodes, with a few seconds latency [tendermint-performance]. However, as we discussed in Section 7, many of those systems see their performance degrading dramatically under heavy load—whereas FastPay performs as expected.
FastPay can be used as a side chain of any crypto-currency with reasonable finality guarantees, and sufficient programmability. As compared to bilateral payment channels it is superior in that it allows users to pay anyone in the system without locking liquidity into the bilateral channel, and is fully asynchronous. However, FastPay does rely on an assumption of threshold non-Byzantine authorities for safety and liveness, whereas payment channel designs only rely on network synchrony for safety and liveness (safety may be lost under conditions of asynchrony). As compared to payment channel networks (such as the lighting network [lightning]) FastPay is simpler and does not require complex path finding algorithms [lightning, flare, grunspan2018ant, sivaraman2018routing].
FastPay is a settlement layer based on consistent broadcast channels, rather than full consensus. The FastPay design leverages the nature of payments to allow for asynchronous payments into accounts, and optional interactions with an external Primary to build a practical system, while providing proofs of both safety and liveness; it also proposes and evaluates a design for sharded implementation of authorities to horizontally scale and match any throughput need.
The performance and robustness of FastPay is beyond and above the state of the art, and validates that moving away from both centralized solutions and full consensus to manage pre-funded retail payments has significant advantages. Authorities can jointly process tens of thousands of transactions per second (we observed a peak of 160,000 tx/sec) using merely commodity hardware and lean software. A payment confirmation latency of less than 200ms between continents make FastPay practical for point of sale payments—where goods and services need to be delivered fast and in person. Pretty much instant settlement enables retail payments to be freed from intermediaries, such as banks payment networks, since they eliminate any credit risk inherent in deferred netted end-of-day payments, that underpin today most national Fast Payment systems [bolt2014fast]. Further, FastPay can tolerate up to one-third of authorities crashing or even becoming Byzantine without losing either safety or liveness (or performance). This is in sharp contrast with existing centralized settlement layers operating on specialized mainframes with a primary / backup crash fail strategy (and no documented technical strategy to handle Byzantine operators). Surprisingly, it is also in contrast with permissioned blockchains, which have not achieved similar levels of performance and robustness yet, due to the complexity of engineering and scaling full Byzantine Fault-Tolerant consensus protocols.
This work is funded by Facebook Calibra. The authors would like to thank Dahlia Malkhi for her constant feedback on this project, and Kostas Chalkias for feedback on late manuscript. We also thank the Calibra Research and Engineering teams for valuable feedback.
Appendix A Proofs of Security
This section proofs the results presented in Section 5.
a.1 Additional Notations
We define as the sum of all the amounts received from the Primary by a FastPay address , as seen at a given time by an authority :
Lemma 1 (Transfer certificate uniqueness).
then and certify the same transfer order:
Both certificates and are signed by a quorum of authorities. By construction, any two quorums intersect on at least one honest authority. Let be an honest authority in both quorums. signs at most one transfer order per sequence number, thus and certify the same transfer order. ∎
Lemma 2 (FastPay invariant).
For every honest authority , for every account , it holds that
Besides, if , we have that for some certificates such that and .
By construction of the FastPay authorities (Figure 2 and Figure 10): Whenever a confirmed certificate is added to , is decreased by , and is incremented into . Any new synchronization order equally increases and . Whenever a confirmed certificate is added to , possibly later (due to cross-shard updates), is increased once by . ∎
Lemma 3 (Primary invariant).
The total balance of all FastPay accounts on Primary is such that
By construction of the smart contract handling funding and redeeming transactions (Section 4.3): whenever a funding transaction is executed by the smart contract, both and increase by . Conversely, decreases by whenever a Primary transfer certificate is redeemed on-chain and added to . ∎
Lemma 4 (Funding log synchronization).
For every honest authority and every account , it holds that
By definition of the synchronization with the Primary (Section 4.3), and by security of the Primary and its client, only increases after a funding transaction has already increased by the same amount. ∎
Lemma 5 (Balance check).
For every honest authority , whenever holds, then we also have
By construction of the FastPay authorities (Figure 2), if , then was successfully processed by as a new transfer order from account . At the time of the request, did not exceed the current balance . Since is still pending, in the meantime, no other transfer certificates from account have been confirmed by . (A confirmation would reset the field pending and prevent from being pending again due to the increased sequence number.) Therefore, the balance did not decrease, and . ∎
Proposition 4 (Account safety).
For every account , at any given time, we have that
Let be the highest sequence number of a transfer certificate from . Let an honest authority whose signature is included in the certificate. At the same of signature, we had , therefore by Lemma 2 and Lemma 5:
Given that amounts are non-negative, for every honest node , we have
Finally, by Lemma 4. ∎
Proof of Proposition 1 (Redeeming to Primary).
Proof of Proposition 2 (Redeeming certificates to FastPay)..
If a certificate exists for account and sequence number , this means at least honest authorities contributed signatures to the transfer order . By construction of FastPay, these authorities have received (Figure 2) and will keep available (Section 4.4) all the previous confirmation orders with , . Therefore, any client can retrieve them and eventually bring any other honest authority up to date with . ∎
Proof of Proposition 3 (Availability of certificates).
Let be the following value evaluated at the time of the creation of the new transfer order :
By a case analysis similar to the proof of Lemma 2, provided that the owner of is communicating the information described in Proposition 3 to the authority , it will hold eventually that and . We deduce that eventually will accept the transfer order and make the value of its signed (pending) order available. ∎
a.4 Performance under Byzantine Failures
The FastPay protocol does not rely on any designated leader (like PBFT [castro1999practical]) to make progress or create proposals; FastPay authorities do not directly communicate with each other, and their actions are symmetric. Clients create certificates by gathering the first responses to a valid transfer order, and no action of a Byzantine authority may delay the creation of a certificate. A Byzantine authority may not even present a signature on a different order as a response to confuse a correct client, since it would have to be signed by the correct payer. Subsequently, a correct client submits the confirmation order to all authorities. Again, Byzantine authorities cannot in any way delay honest authorities from processing the payment locally in their databases, and enabling a subsequent payment for the sending account.
Byzantine clients may attempt denial of service attacks by over-using the system, and for example creating a very large number of receiving accounts (this could be disincentivized by charging some fee for an account creation). However, an attempt to equivocate by sending two transfer orders for a single sequence number could either result in their own account being locked (no single transfer order can achieve signatures to form a certificate and move to the next sequence number), or one of them succeeding—neither of which degrade performance. Transfer orders with insufficient funds or incorrect sequence numbers are simply rejected, which does not significantly affect performance (if anything they do not result in confirmation orders that are more costly to process than transfer orders, see Section 7).
Appendix B Code Listings for Core Operations
Algorithms for the core authority operations are simplified directly from the Rust implementation. We omit explicit typing, details of error messages returned, de-referencing, and managing variable ownership. The macro ensure, returns with an error unless the condition is fulfilled, and bail always returns with an error.