Ever since bitcoin [nakamoto2008bitcoin] has been proposed, blockchain technology has been widely studied for years. Extensive adoptions of blockchain technologies was seen in real world applications such as financial services with potential regulation challenges [michael2018blockchain, tapscott2017blockchain], supply chains [korpela2017digital, tian2016agri, abeyratne2016blockchain], health cares [azaria2016medrec, yue2016healthcare] and IOT devices [christidis2016blockchains]. The core of blockchain technology depends on the consensus algorithms applying to the open distributed computing world. Where computers can join and leave the network and these computers can cheat.
As the one of the most famous protocol that can solve the so called Byzantine general’s problem, bitcoin system suffers from the problem of low transaction rate with a transaction per second (TPS) of approximately , and long confirmation time (about an hour). As more and more machines joined the network, they are competing for the privileges to attach the block (miners) which result in huge waste of electric power. While high fees are payed to make sure the transfers of money will be placed in the chain. On par, there are multiple proposals to solve the low transaction speed issue. One method intends to solve the speed problem without changing the chain data structure, for instance segregated witness [lombrozo2015segregated] or off chain technologies such as lightning network [poon2016bitcoin] or plasma [poon2017plasma]. Another hard fork way changed the bitcoin protocol such as the bitcoin cash tries to improve the throughput of the system by enlarging the data size of each block from Mb to Mb.
To minimize the computational cost of POW, a series of proof of stake method POS [duffield2018dash, tron2018, david2017ouroboros, wood2014ethereum, goodman2014tezos]
are proposed to make sure that those who have the privilege to attach the block proportional to their token shares. Another idea targeting at utilizing the power in POW to do useful and meaningful tasks such as training machine learning models are also proposed[matthew2017aion]. In addition, inspired by the PBFT algorithm [castro1999practical] and a set of its related variations, so called hybrid (or consortium) chain was proposed. The general idea is to use two step algorithm, the first step is to elect a committee, the second step is collecting committee power to employ PBFT for consensus. Bitcoin-NG [eyal2016bitcoin] is the early adopter of this idea, which splits the blocks of bitcoin into two groups, one is for master election and another for regular transaction blocks. Honey-badger [miller2016honey] is the system that firstly introduced the consensus committee, it uses a predefined members to perform PBFT algorithm to reach consensus. The Byzcoin system [kogias2016enhancing] brought forth the idea of POW for the committee election, and uses a variation of PBFT called collective signing for speed purposes. The Algorand [gilad2017algorand] utilizes a random function to anonymously elect committee and use this committee to commit blocks, and the member of the committee only have one chance to commit block. Other popular systems include Ripple [schwartz2014ripple], Stellar [mazieres2015stellar] and COSMOS [kwon2016cosmos] etc. All these systems have one common feature, the split of layers of players in the network, which results in the complexity of the implementation of the system.
While aforementioned methods are aiming to avoid side chains, another thread of effort is put on using direct acyclic graph(DAG) to merge side chains. The first ever idea comes with growing the blockchain with trees instead of chains [sompolinsky2013accelerating], which results in the well known GHOST protocol [sompolinsky2015secure]. If one block links to previous blocks, then the data structure grows like a DAG instead of tree [lewenberg2015inclusive], SPECTRE [sompolinsky2016spectre] and PHANTOM [sompolinskyphantom] are such type of systems. Byteball [churyumov2016byteball] is the system that construct a main chain, and leverage this main chain to help infer the total order, nontheless, the selection of main chain is dependent on a role called witness, which is purely centralized. Conflux is a improvement of the GHOST based DAG algorithm which also utilises pivotal (main) chain without the introduction of witness and claim to achieve of TPS in reality [li2018scaling]. IOTA tried to avoid finality of constructing a linear total order by introducing the probabilistic confirmation in the network [popov2016tangle]. Aforementioned systems are permissionless chains, in the permissioned chains, DAG technology is also applied. HashGraph [baird2016swirlds] is the system that utilises the algorithm to propagete the block graph structure, and achieve the consensus by link analysis in the DAG, this method is proved to be Byzantine fault tolerant and is not relied on voting. Blockmainia [danezis2018blockmania] is based on the original PBFT design, but it’s underlying log is DAG based. Some of the side chain methods also borrows the idea of DAG, such as nano [lemahieu2018nano] and vite [liuvite]. These system in reality are rely on centralized methods to maintain their stability.
Social network analysis has widely adopted the method of streaming graph computing [ediger2011tracking, green2012fast, ediger2012stinger] which deals with how to quickly maintain information on a temporally or spatially changing graph without traversing the whole graph. We view the DAG based method as a streaming graph problem which is about how to compute the total order and achieve consensus without consuming more computing power. In distributed database systems, the problem of replicating data across machines is a well studied topic [demers1988epidemic]. Due to the low efficiency of the bitcoin network, there are multiple ways to accelerate the message passing efficiency [klarmanbloxroute], however, they did not deal with the network complexity. We view the problem of scaling DAG system in the network of growing size and topological complexity as another challenging issue, and proposed our own gossip solution. The main contribution of this paper is how to utilize the streaming graph analysis methods and new gossip protocol to enable real decentralized, and stabilized growing DAG system.
Ii Basic design
Ii-a Data structure
The local state of a node in the StreamNet protocol is a direct acyclic graph (DAG) . is the set of blocks in . is the genesis block. For instance, vertex in Figure 1 represents the Genesis block. is a function that maps a block to its parent block . Specially, . In Figure 1, parent relationships are denoted by solid edges. Note that there is always a parent edge from a block to its parent block (i.e., , ). is the set of directly reference edges and parent edges in this graph. is an edge from the block to the block , which means that happens before . For example in Figure 1, vertex represents the first block, which is the parent for the subsequent block , and . Vertex has two edges, one is the parent edge pointing to , another is reference edge pointing to . When a new block is not referenced, it is called a tip. For example, in Figure 1, block is a tip. All blocks in the StreamNet protocol share a predefined deterministic hash function Hash that maps each block in to a unique integer id . It satisfies that , Hash() Hash().
Ii-B StreamNet Architecture
Figure 2 presents the architecture of StreamNet, it’s consists of multiple StreamNet machines. Each StreamNet machine will grow its DAG locally, and will broadcast the changes using gossip protocol. Eventually, every machine will have a unified view of DAG. By calling total ordering algorithm, every machine can sort the DAG into a total order, and the data in each block can have a relative order regardless of their local upload time. Figure 3 shows the local architecture of StreamNet. In each StreamNet node, there will be a transaction pool accepting the transactions from the HTTP API. And there will be a block generator to pack a certain amount of transactions into a block, it firstly find a parent and reference block to attach the new block to, based on the hash information of these two blocks and the meta data of the block itself, it will then perform the proof of work (POW) to calculate the nonce for the new block. Algorithm 1 summarize the server logic for a StreamNet node. In the algorithm, the way to find parent block is by . And the way to find reference block is by calling which is the Markov Chain Monte Carlo (MCMC) random walk algorithm [popov2016tangle]. The two algorithms will be described in the later section.
Ii-C Consensus protocol
Based on predefined data structure, to present the StreamNet consensus algorithm, we firstly define several utility functions and notations, which is a variation from the definition in the Conflux paper [li2018scaling]. Chain() returns the chain from the genesis block to a given block following only parent edges. returns all blocks except those in the chain. Child() returns the set of child blocks of a given block. Sibling() returns the set of siblings of a given block. Subtree() returns the sub-tree of a given block in the parental tree. Before() returns the set of blocks that are immediately generated before a given block. Past() returns the set of blocks that are generated before a given block (but including the block itself). After() returns the set of blocks that are immediately generated after a given block. Later() returns the set of blocks that are generated after a given block (but including the block itself). SubGraph() returns the sub graph by removing blocks and edges except the initial set of blocks. ParentScore() presents the weight of blocks, each block have a score when referenced as parent. Score() presents the weight of blocks, each block achieves a score when attaching to the graph. TotalOrder() returns the ‘flatten’ order inferred from the consensus algorithm. Figure 4 represents the definition of these utility functions.
Ii-C1 Parent tip Selection by pivotal chain
The algorithm Algorithm 3 presents our pivot chain selection algorithm(i.e., the definition of ). Given a StreamNet state , Pivot(,) returns the last block in the pivot chain starting from the genesis block . The algorithm recursively advances to the child block whose corresponding sub-tree has the largest number of children. Which is calculated by When there are multiple child blocks with the same score, the algorithm selects the child block with the largest block hash. The algorithm terminates until it reaches a tip. Each block in the pivot chain defines a epoch in the DAG, the nodes in DAG that satisfy Past(,) - Past(,) will belong to the epoch of block . For example, in Figure 5, the pivot chain is , and the epoch of block contains two blocks and .
Ii-C2 Reference tip selection by MCMC
The tip selection method by using Monte Carlo Random Walk (MCMC) is as Algorithm 2
shows. Starting from the genesis, each random walk step will choose a child to jump to, and the probability of jumping from one block to the next block will be calculated using the formula in the algorithm.in the formula is an constant that is used to scale the randomness of the MCMC function, the smaller it is, the more randomness will be in the MCMC function. The algorithm returns until it finds a tip.
Ii-C3 Total Order
The algorithm Algorithm 4 defines StreamNetOrder(), which corresponds to our block ordering algorithm. Given the local state and a block in the pivot chain, StreamNetOrder(, ) returns the ordered list of all blocks that appear in or before the epoch of . Using StreamNetOrder(), the total order of a local state is defined as TotalOrder(). The algorithm recursively orders all blocks in previous epochs(i.e., the epoch of and before). It then computes all blocks in the epoch of as . It topologically sorts all blocks in and appends it into the result list. The algorithm utilizes the unique hash to break ties. In Figure 5, the final total order is .
Ii-D The UTXO model
In StreamNet, the transactions utilizes the unspent transaction out (UTXO) model, which is exactly the same as in Bitcoin. In the confirmation process, the user will call to get the relative order of different blocks, and the conflict content of the block will be eliminated if the order of the block is later than the one conflicting with it in the total order. Figure 6 shows the example of storage of UTXO in StreamNet and how conflict is resolved. Two blocks both referenced the same block with Alice having 5 tokens, and construct the new transaction out which representing the transfer of token to Bob and Jack respectively. However, after calling , the Bob transfer block precedes the Jack transfer block, thus the later block will be discarded.
Ii-E Gossip Network
In the bitcoin and IOTA network, the block information is disseminated in a direct mail way [demers1988epidemic]. Suppose there are nodes and links in the network, for a block of size , to spread the information of it, the direct mail algorithm will have a total complexity of . And the average complexity for a node will be In the chain based system, this is fine, because the design of the system already assume that the transaction rate will be low. However, in the DAG based system, this type of gossip manner will result in low scalability due to high throughput of the block generation rate and will result in network flooding. What’s worse, consider the heterogeneously and long diameters of network topology, the convergence of DAG will take long time which will cause the delay of confirmation time of blocks.
Ii-F Differences with other DAG protocols
Here, we mainly compare the difference of our protocol with two mainstream DAG based protocols, one is IOTA, another is Conflux.
The major difference with IOTA are in three points:
Firstly, the IOTA tip selection algorithm’s two tips are all randomly chosen, and ours is one deterministic which is for the total ordering purposes and one by random which is for maintaining the DAG property;
Secondly, the IOTA consensus algorithm is not purely decentralized, it relies on a central coordinator to issue milestones for multiple purposes, and our algorithm does not dependent on such facility.
Lastly, in IOTA, there is no concept of total order, and there are 3 ways to judge if a transaction is confirmed:
The first way is that the common nodes covered by all the tips are considered to be fully confirmed;
All transactions referenced by the milestone tip are confirmed.
The third way is to use MCMC. Call times to select a tip using the tip selection algorithm. If a block is referenced by this tip, its credibility is increased by 1. After selections have been cited times, then the credibility is .
The major difference with Conflux are in two points:
Firstly, Conflux will approve all tips in the DAG along with parent, which is much more complicated than our MCMC based two tip method. And when the width of DAG is high, there will be much more space needed to maintain such data structure.
Secondly, Conflux total ordering algorithm advances from genesis block to the end while StreamNet advances in the reverse direction. This method is one of the major contribution to our streaming graph based optimizations, which will be discussed in the next chapter. In Conflux paper, there is no description of how to deal with the complexity paired with the growing graph.
Ii-G1 Safety & Liveness
Because StreamNet utilises the GHOST rule to select pivot chain, which is the same as in Conflux. Thus, it shares the same safety and correctness property as Conflux. Although the choice of reference chain in StreamNet is didfferent from Conflux, it only affects the inclusion rate, which is the probability of a block to be included in the total order.
According to Theorem 10 in [sompolinsky2015secure] and the deduction in [li2018scaling], given a period of , and block in pivot chain in this period, the chance of kicked out by its sibling is no more than in Figure 7. Which is the same as in Conflux.
Followed by the definitions in Conflux paper [li2018scaling], in (1), is the number of blocks in the subtree before , is the number of blocks in the subtree of before . is honest node’s block generation rate. is the attacker’s block generation ratio with respect to . From the equation, we can conclude that with the time goes by, the chance of a block in the pivot chain to be reverted is decreased exponetially.
Iii Optimization Methods
One of the biggest challenges to maintain the stability of DAG system is that, as the local data structure grows, the graph algorithms (, , ), relies on some of the graph operators that need to be recalculated for every newly generated block, which are very expensive. Table I list all the expensive graph operators that are called. Suppose the depth of the pivot chain is , then we give the analysis of complexity in the following way. and rely on the breath first search (), and the average complexity would be , and for each MCMC() and Pivot() called the complexity would be in total in both of these two cases. The calculation of also relies on the operator, in the StreamNetOrder() algorithm, the complexity would be accrued to . TopOrder() is used in sub-order ranking the blocks in the same epoch. It’s the classical topological sorting problem, and the complexity in the StreamNetOrder() would be .
|Graph Property||Algorithm used||Complexity||Tot|
|Past(G,b) - Past(G,p)||StreamNetOrder()|
Considering new blocks are generated and merged into the local data structure in a streaming way. The expensive graph properties could be maintained dynamically as the DAG grows. Such that the complexity of calculating these properties would be amortized to each time a new block is generated or merged. In the following sections, we will discuss how to design streaming algorithms to achieve this goal.
Iii-a Optimization of Score() and ParentScore()
In the optimized version, the DAG will have a map that keeps the score of each block. Once there is a new generated/merged block, it will trigger the BFS based UpdateScore() algorithm to update the scores of the block in the map that are referenced by the new block. The skeleton of the UpdateScore() algorithm is as Algorithm 5 shows.
Iii-B Optimization of Past(G,b) - Past(G,p)
We abbreviate the Past(G,b) - Past(G,p) to calculate as GetDiffSet(G,b,C) which is shown in the Algorithm 6. This algorithm is in essence a dual direction algorithm. Starting from the block , it will traverse all its referenced blocks. Every time a new reference block is discovered, it will perform a backward to ‘look back’ to see if itself is already covered by the ’s parent block . If yes, would not be added to the forward queue. To avoid the complexity of the backward , the previous calculated diff set will be added to the covered set , which will be passed to GetDiffSet() as a parameter. To be more specific, when a backward BFS is performed, the blocks in will not be added to the search queue. This backward search algorithm is denoted as IsCovered() and described in detail in Algorithm 7.
Figure 8 shows the example of the GetDiffSet() method for block . It first perform forward BFS to find block which does not have children, then it will be added to the diff set. then move forward to , which have three children, if it detect which is the parent of , it will stop searching promptly. If it continue searching on or , these two blocks would not be added to the search queue, because they are already in the covered set.
Iii-C Optimization of TopOrder()
The topological order is used in sorting the blocks in the same epoch. To get the topological order, every time, there needs a top sort of the whole DAG from the scratch. However we can easily update the topological order when a new block is added or merged. The update rule is, when a new block is added, it’s topological position will be as Figure 9 shows. This step can be done in
To summarize, the optimized streaming operators can achieve the performance improvement as Table II shows.
|Graph Property||Algorithm used||Complexity||Tot|
|Past(G,b) - Past(G,p)||StreamNetOrder()|
Iii-D Genesis Forwarding
The above algorithm solved the problem of how to dynamically maintaining the information needed for graph computation. However, it still needs to update the information until genesis block. With the size of the graph growing, the updating process will become harder to compute. With the grwoth of DAG size, the old historical confirmed blocks are being confirmed by more and more blocks, which are hard to be mutated. And the exact probability can be computed in formula (1). Hence, we can design a strategy to forward the genesis periodically and fix the historical blocks into a total ordered chain. The criteria to forward the genesis are based on the threshold of ParentScore(). Suppose we define this threshold as , then we only forward the genesis if:
In Figure 10, we set , and there are three side chains with , . And in pivot chain, there are multiple blocks that has , they are condidates for the new genesis, we choose the block with minimumn as the new genesis.
In addition, after the new genesis has been chosen, we will induce a new DAG in memory from this genesis, and persist the ‘snapshot’ total order (Conflux paper has the same definition, but it does not show the technical detail, we do not view it trivial) in the local data base. Once the total order is queried, a total order based on the current DAG will be appended to the end of the historical snapshot total order and be returned. In addition, the vertices in UTXO graph that belongs to the fixed blocks will be eliminated from the memory and be persisted to disk as well. The algorithm is as Algorithm 8 shows.
Iii-E The Direct Signal Gossip Protocol
To minimize the message passing in the gossip network, there are solutions in [demers1988epidemic]. And in Hyperledger [androulaki2018hyperledger] they have adopted the PUSH and PULL model for the gossip message propagation. However, their system is aiming at permissioned chain. Suppose the size of the hash of a block is , we designed the direct signal algorithm. The algorithm is divided into two steps, once a node generate or receive a block, it firstly broadcast the hash of the block, this is the PUSH step. Once a node receive a hash or a set of a hash, it will pick one source of the hash for the block content, this is the PULL step. The direct signal algorithm’s complexity will be and for a node averaged to The algorithm is as Algorithm 9 shows.
Iv Experimental Results
We have implemented the StreamNet based on the IOTA JAVA reference code (IRI) v1.5.5 [IOTACode]. We forked the code and made our own implementation, the code is freely available at [StreamNet]. In this paper we use version v0.1.4-streamnet in v0.1-streamnet beta branch.
The features we have adopted from the IRI are:
The block header format, as shown in Figure 11. Some of the data segments are not used in StreamNet which are marked grey.
Gossip network, the network is a bi-directional network in which every node will send and receive data from its peers;
Transaction bundle, because of the existence of the bundle hash feature, StreamNet can support both the single transaction for a block and batched transactions as a bundle.
Sponge hash functions which is claimed to be quantum immune, in our experiment, the POW hardness is set to 8 which is the same as the testnet for IOTA.
The features we have abandoned from the IRI are:
The iota’s transaction logic including the ledger validation part;
The milestone issued by coordinators, which is a centralized set up.
The features we have modified based on the IRI is:
The tip selection method based on MCMC, since the tip selection on IRI has to find a milestone to start searching, we replace this with a block in the pivotal chain instead.
The features we have added into the StreamNet are:
The consensus algorithms, and we have applied the streaming method directly in the algorithms;
The UTXO logic which is stored in the signature part of the block header, we used the graph data structure to store UTXO as well.
In IOTA’s implementation, the blocks are stored in the RocksDB [RocksDB] as the persistence layer, which makes it inefficient to infer the relationships between blocks and calculate graph features. In our implementation, we introduced an in-memory layer to store the relationships between blocks, such that the tip selection and total ordering algorithm will be accelerated.
Iv-B Environment Set Up
We have used the AWS cloud services with 7 virtual machines, for each node, it includes a four core AMD EPYC 7571, with 16 Gb of memory size and 296Gb of disk size. The JAVA version is 1.8, we have deployed our service using docker and the docker version is 18.02.0-ce.
We have 7 topologies set up of nodes, which are shown in Figure 12, these configurations are aiming to test:
The performance when the cluster connectivity is high (congestion of communications, like 3-clique, 4-clique, 7-clique and 7-star);
The performance when the cluster diameter is high (long hops to pass message, like 4-circle, 7-circle, 7-bridge);
As for the data, we have created 1,000 accounts, with the genesis account having 1,000,000,000 tokens in the coinbase block. We divided the accounts into two groups (each group will have 500 accounts), the first group will participate in the ramp up step, which means the genesis account will distribute the tokens to these accounts. And for comparison we have issued four set of different size transactions (5000, 10000, 15000 and 20000) respectively. In the execution step, the first group of accounts will issue transactions to the second group of accounts, which constructs a bipartite spending graph. Since there are more transactions than the number of accounts, there will be double spend manners in this step. The number of threads in this procedure is equal to the number of nodes for each configuration. Jmeter [halili2008apache] is utilized as the driver to issue the transactions and Nginx [nedelcu2010nginx] is used to evenly and randomly distribute the requests to different nodes.
Iv-C Results and Discussions
Iv-C1 Block generation rate test
To test the block generation rate, we set each block in StreamNet to have only one transaction. And the performance on this configuration is as Figure 13
shows. To begin with, as the size of the cluster grows, the network as a whole will witness little performance loss on all of the data scales. In the experiment, we can also see that with the growth of the data, the average TPS on most of the configurations have grown a little bit (there are definitely outliers that need our time to triage), this is because the genesis forwarding algorithm needs some ramp up time to get to the stable growth stage. Considering the system is dealing a growing graph instead of a chain, and the complexity analysis in the previous section, the experiment clearly shows that our streaming algorithm shed the light on how to deal with the growing DAG.
Iv-C2 Bundle transaction test
By default, each block in StreamNet will support bundle transaction. We set each bundle to contain 20 transactions and for each block there are approximately 3 transactions included. The performance on this configuration is as Figure 14 shows. In this experiment, we can see that the performance (TPS) comparing with the block test improved more than twice. This is because there will be less POW works to be done. In addition, with growth of the data we do not witness obvious performance down turn. But there are some performance thrashing in the experiment, which needs more study.
In this paper, we proposed a way to compute how to grow the blocks in the growing DAG based blockchain systems. And how to maintain the total order as the DAG structure is dynamically turning larger. We referred one of the earliest DAG implementation IRI to conduct our own experiments on clusters of different size and topology. Despite the network inefficiency in the IRI implementation, our method is proven to be able to tolerate the increasing complexity of the graph computation problems involved. This is due to the streaming graph computing techniques we have introduced in this paper.