Ethereum  is a mature decentralized web infrastructure that has proved reliability and had led the growth of decentralized applications. It features an ecosystem for decentralized applications based on a general purpose virtual machine and its dedicated programming language. This ecosystem holds certain characteristics, which has generated a large community of developers that have proposed new methods to tackle some of the limitations affecting the Ethereum protocol.
The Ethereum Foundation together with the Ethereum community have been working on the development of the second generation of Ethereum, Ethereum 2.0 (Eth2). The approach proposed on the Eth2 protocol tackles both: the scalability challenge by introducing the sharding, and the sustainability challenge by changing the consensus protocol to Proof-of-Stake (PoS). This ambitious project is divided into different phases because of its complexity . Taking place, with the launch of the Eth2 main net, the beginning of the project or phase 0, the Beacon Chain .
Eth2 aims to create a major network of clients that will provide the infrastructure for sharding. The different developer teams have been working over the last two years to generate Eth2 client implementations, each client targeting a specific type of user. This variety of clients, together with the low computing-power required by the Eth2 protocol, offers to most of the users the possibility to join the network.
In contrast to the Ethereum1 client, the Eth2 one is divided in two main parts, the Beacon Node and the Validator. The beacon node, or node, performs all kind of interactions with the Eth2 network. It establishes the connections with other nodes, exchanges the messages, downloads the blocks and maintains the view of the Beacon Chain as the view of the Shards that the validator is performing at, building among them the entire network. On the other hand, the validator is in charge of the logical part, generating the attestations on the received blocks, as well as proposing the blocks when needed. To perform its duties, the validator needs to follow the current state of the Chain, and to be in constant communication with a Beacon Node that provides such information.
A project with this complexity level completely relies on the healthiness of the network. To propagate the messages between the nodes of the network, Eth2 relies on the GossipSub p2p protocol . The protocol is based on the exchange of messages and metadata between peers distributed in meshes, achieving the message propagation within the targeted time-margins and minimizing wasteful bandwidth consumption. Given the Eth2 network requirements and the interaction between the clients, it is important to monitor critical points for possible attacks. In this paper, we introduce a network monitoring tool that we call Armiarma, used to generate a complete analysis of the p2p network of the Eth2 main net. he tool is capable of showing a general overview of the composition of the network, and is also capable of doing a detailed analysis of a specific node.
The remainder of this paper is organized as follows. Section II discusses the background and related work. Section III explains the methodology used for our study and evaluation. In Section IV we show and analyze the results obtained by our tool. Finally, Section V concludes this work and presents some possible future directions.
Ii State Of The Art
On a large decentralized blockchain that involves so many nodes and clients, the security and integrity of the protocol relies on the healthiness of the network. Being able to debug from inside all the occurring events, offers the possibility to get an in-depth view of the network status and might even help to prevent performance issues as well as more serious network reliability problems or even security vulnerabilities. Escaping from the complex and tedious work that would mean generate specific test cases for each of the clients, the community have tried to develop tools to interact easily with the network, without having to operate as a full client.
One of the those tools is Rumor. Rumor appears as an interactive shell script tool for debugging and testing the interaction between the Eth2 clients. Originally started as a Eth2 networking tool written in Go, Rumor was designed as an alternative to become the common platform for testing the different Eth2 nodes in a less effort and time-consuming way. By offering a set of commands that can replicate most of the Eth2 network protocol interactions, Rumor offers the abstraction from the code implementation that makes, setting a custom node, possible by simply tipping a few commands.
The platform is based on an environment close to the Bash shell that eases the test building labour, where some of the Bash syntax (e.g., variables and control flow statements) are accepted. Along with the scripting capabilities, Rumor includes a complex system of sub-environments (known in Rumor as actors) that allows to generate different nodes with different properties simultaneously. The different nodes distributed along the different actors share the common environment, which perform as link between them. This environment settles the possibility for the peers to exchange information, needed to perform a synchronization of actions based on constantly occurring events.
Due to its debugging purposes, most of the commands of the platform offer the debugging and testing related information in a log format. This logs, configurable most of the time in different levels (error, debug, info, etc..), are the ones that provide the insights of the processes that are taking place on the node. The governance over the node specs and its conduct makes Rumor the appropriate tool for testing and debugging the Eth2 clients and their behaviour under different conditions. The variety of commands that Rumor offers is still on a developing stage, but it already includes most of Eth2 networking specs such:
Generate a ENR identity for the node needed to participate on some of the Eth2 protocols.
The Eth peer discovery protocol dv5.1.
Generate a peer database or peerstore of the connected and discovered peers.
Establish a connection with a given peer.
Exchange Status and Metadata of the Beacon State with the connected peers.
Listen and publish messages on the GossipSub p2p protocol used in Eth2.
Send and respond to RPCs from and to the peers.
Import blocks and states from the Beacon Chain.
The different hosts configurations and behaviour strategies can be deployed in different ways. If the desired test is not too long, it can be launched from the rumor shell by typing the commands in the desired order. For more repetitive tasks or tests, the platform offers the possibility to read the list of commands from a given file (e.g., script.rumor). For a more complex solutions, Rumor also offers the option to add the given test-plan into a service, allowing the test-case to behave as a server.
Ii-B Node Discovery Protocol v5
As a decentralized platform, Eth2 tries to avoid any point of centralization inside the proposed protocol. The networking area, and more specifically the peer discovery, is not an exception to this rule. The implemented GossipSub protocol, despite being a protocol oriented to a message propagation on decentralized applications, does not offer any kind of peer discovery service. Leaving that task in charge of the application. Eth2 developed its own node discovery protocol, Discovery 5 (dv5), currently on its 5.1 version .
Dv5 focuses on the Ethereum Node Records (ENR) exchange along the peers. This node records, including the IP and ports needed to establish a connection, are recorded in a Distributed Hash Table (DHT). This DTH offers the possibility to sample and search along the generated database, allowing to easily update the node record of a specific peer if modifications are detected. As entry point to the dv5 protocol, the nodes can start searching for others that publicly advertise themselves on specific topics, or as an alternative that improves the performance, a first connection with boot-nodes is possible.
The peer discovery service is essential for the network in general. New peers joining the network need connections from where synchronize the chain. Peers that are already on the network also need from time to time to actualize their information source, ensuring that they are properly following the head of the chain. Furthermore, to increase the resilience of the network to Sybil attacks, dv5 serves a record of nodes on the network that the node could connect at any point.
The approach proposed on this paper tackles the lack of information about the performance of the GossipSub protocol and the peers on the Eth2 main net. Armiarma is built on top of Rumor and it offers a simple yet powerful method that provides meaningful data about the p2p network and the Eth2 clients. The approach is divided into two parts, the Armiarma crawler and the Armiarma analyzer, as described bellow.
Iii-a Armiarma Crawler
Armiarma Crawler is the data gatherer part of the tool. The crawler has been based on the Eth2 client debugging tool Rumor, and its performance is linked to its ability to discover and peer with clients from the Eth2 network. Launching the crawler has been summarized in a chain of commands that execute the initialization from a bunch of files. Having a head file that initializes all the entire crawling process, the rest of the files get called from the main one performing the host initialization. This process starts with the host set up, by assigning the IP and ports that will be used by the host. Followed by the ENR generation, the Eth2 ENR identity gets generated from the host info and by specifying the network that we want to belong to, that in this case is the main net . Once we advertise on the ENR of the crawler that we participate on the Eth2 main net, a Beacon Node needs to be faked. Claiming to be at the genesis state prevents the crawler from being penalized for not offering services from the beacon chain as blocks synchronization. For successfully faking an Eth2 client and being accepted by the rest of the peers, the custom host has been set up to be at the genesis state of the Beacon Chain. After setting the host, the GossipSub p2p protocol is initialized, joining the main topics from the Eth2 Specs. The Rumor GossipSub implementation has been modified  to store all the data gathered from the peers we get connected to, getting from each of them:
Information about the peer:
Client Type and Client Version
The connection/disconnection events with the timestamp of each event.
A Counter of every message we have received from each peer on the five main GossipSub topics of Eth2:
Once the host is fully initialized, the peer discovering protocol dv5  and the connectall Rumor services are launched. This way, all the peers that we are able to find through the dv5 protocol are recorded into a peerstore and attempted to be connected. From all of those peers that we establish a connection, the crawler would start to get the messages from the GossipSub topics previously joined, generating as well the metrics previously mentioned. To export the metrics, a new command has been added to Rumor, exporting the peerstore and the metrics every given time interval. As the Armiarma Implementation is constantly improving and adding new features or gathering new metrics from the network, its synergy with Rumor also leads Rumor to be continuously improving its implementation with new features.
Iii-B Armiarma Analyzer
While Armiarma crawler is focused on the interaction with the p2p network, recording into a json file all the events and the peer information, Armiarma analyzer gets in charge of performing an analysis from the raw information gathered. At the moment, the analysis gets performed in a subsequent moment after the metrics get exported from the crawler. In a combination between the read peerstore and the metrics from the GossipSub implementation, the analyzer parses the peer information individually, ending up in a large panda database of the crawled peers. During the parsing process, the format of several items gets modified, aiming to ease the data analysis.
One of those format modifications are the recorded connections and disconnections events that get counted. From there, the total connection time to each peer is obtained. As explained in Section III-A, the crawler supports the main five topics from the Eth2 p2p network. Because of this, the crawler usually reports connections and disconnections in batches of five, mostly at the same exact time. To prevent an over-counting of the events, and most importantly, to get an accurate connection time per peer, the events of the same type (i.e., connection or disconnection) and almost the same time (i.e., inside of a time-window of 500ms) are counted only once. This helps to get a closer representation of the actual number of connections and their duration.
Once the parsing process has been successfully achieved, the analyzer offers several tools for filtering and plotting from pandas. That eases the chart representation of desired metrics from the database. The obtained compilation of graphs and their insights will be discussed on the following section IV.
|Initial Date||2020-12-12 16:25:54|
|Finishing Date||2020-12-14 13:34:26|
In order to test the capabilities of the Armiarma tool we performed an experiment in which we let the crawler run for several days collecting as much data as possible. Following the low processing-power oriented philosophy on the Eth2 project, the crawler has been set on a ARM based Raspberry Pi 4 with 4GB of RAM and with a standard 1 Gbit Ethernet connection. The exact running period of the experiment is shown in table I. The discussion of the results obtained from the analysis have been divided in the three subsections described bellow.
Iv-a Eth2 Network Analysis
In blockchain protocols as Eth2, the security of the network relays on a high number of nodes working together. To have a stable performance, the network has a margin of peers that can go offline, without necessarily leading to the crash on the protocol. In this case, Eth2 needs at least 66% of the clients actively attesting on the generated blocks to finalize.
The first part of the network analysis consists on having the crawler to get a list of peers and attempt connecting to them. We can observe on figure 1 that from the amount of peers we obtained from the dv5 protocol, the crawler wasn’t able to connect with all of them. That is normal considering that not all the peers are connected all the time and most clients have restrictions in the number of peers they can connect with, therefore rejecting additional connections. Also, this could be related to a non port forwarding configuration by those peers, preventing us to initiate the peering handshake.
Blockchain networks are geographically distributed all around the world. Having the peers distributed, means that the peers are less likely to get affected by the same type of regional problems or events (e.g., National censorship). On figure 2, we can appreciate that despite having a large concentration of peers in two countries, the US and Germany, the peers distribution is mostly homogeneous.
Taking a closer look, on Figures 3 and 4, we can observe that not all the peers connected have had the same interaction with the crawler. Note that the number of connections and disconnections registered by the crawler are based on the number of GossipSub topics supported by the connected peer, in this case all the clients supports all the five main Eth2 protocols. The difference on the time we have been connected to the peers, shown on Figure 5 can be related to the pruning process that takes place on the GossipSub protocol. This process aims to keep the predefined range of peer connections. Note that to keep the computing-power and network bandwidth low, several client implementations reduce the maximum number of peers the node is allowed to be connected to.
In a decentralized network such as Eth2, low latency connections are essential for tracking every occurring event as fast as posible. In this case, on the Eth2 Beacon Chain protocol, there is a new Slot event every 12 seconds. Meaning that every 12 seconds a randomly chosen validator will have the chance to propose a Beacon Block and therefore receive the token reward. On figure 6
, we can observe the latency of the connections that the crawler recorded while being connected to the peers. We can observe that the majority stays with a latency below 3 seconds, having an outlier with a 31 seconds latency. There are two things we should take into account when analyzing the latency:
The crawler is not a fully functional client. Its purpose is to provide insights about the network and therefore, it tries to stay connected with as many peers as possible. Meanwhile, a client would focus on keeping a low range of peers (30-60) that would be chosen based on their specs (latency, message sharing, etc.).
The crawling process was done on a raspberry Pi located in Spain. Thus, peers located on the other side of the world would usually report a higher latency, e.g. the outlier peer with an IP from USA (See Section IV-C).
Iv-B Eth2 Clients Interaction
Currently, there are five Eth2 clients available for participating on the network. With Armiarma, we were able to detect which client is used by each connected peer, and it is even possible to get the version they are using, as shown in Figure 7 and Table II. Note that we did not received any connection from any Lodestar client, but got connected to 26 peers from unknown origin. This Unknown peers could participate on the network as bootnodes, other crawlers, possible attackers or just clients that did not set the user agent field.
As we can see in Figure 7, there is a large number of peers from Lighthouse. In contrast, on the peerstore we observe that 55.17% of the peers use the TCP port used by Prysm clients (), showing that the majority of nodes in the Eth2 network use Prysm. 111Although it is possible to change the dialing port of the host in all the clients, the percentage of users that change it to this one specifically is low. There are several reasons for this disparity. First, the figure does not show all the peers the crawler discovered, but only those it managed to established a connection with. Lighthouse and Teku clients usually accept higher number of peers and are more flexible in the number of peers they keep. Prysm on the other hand, has a more strict limit on the number of peers they accept. Although they have flags to change the maximum number of peers, most nodes usually run with the default parameters. Finally, issues related to port-forwarding could have also played a role and prevented some connections from Prysm nodes. Nevertheless, the distribution of Eth2 clients is not the focus of this paper, but the capabilities of the network crawler. With Armiarma, we could see how many version of each client are out there in the wild, as shown in Table II. This is useful as we can observe over a long period of time, how client teams deploy new versions and how fast users update their clients.
On the comparison of the behaviour of the different clients with the crawler, we can observe several differences. First we compare the average number of connections and disconnections for each one of the clients, as shown in Figure 8 and 9. Lighthouse, Teku, Nimbus and the Unknown clients obtain an relatively close number of connection events that goes from 620 to 840 on average per node, being Teku the client that obtains the highest number of dialing events. Once again, the low number of events from the Prysm nodes stands out. Despite connecting to the same number of peer as Nimbus, Prysm nodes have dialed x23 times less to our crawler.
The unexpected number of dialing events seen from Prysm nodes, brings up again the different peering strategies adopted by the clients while implementing the GossipSub protocol. Among the possible strategies, Prysm seems to have the most rigid one, dismissing connections that goes beyond the selected peer-limit. Nonetheless, for the same reasons Prysm nodes also tend to not disconnect often, making those connections more stable, hence the lower number of connections attempts.
Teku, on the other hand, implements one of the most flexible peering strategies, allowing for a large number of connections but also dropping connections relatively often. This leads, to many re-connection attempts and therefore we observe this high number of connection events. Shortly after the main net launched, Prysm has tried to remove the previously mentioned limitation by adding a range buffer for incoming connections. Therefore, it is expected that new Prysm nodes joining the network or old nodes with the recent Prism client version could show more flexibility in their peering strategy.
However, number of connections does not equate to connected time, because clients could attempt multiple connections while still disconnecting shortly after the connection has been established. Therefore, we keep track of both connections and disconnections which allows us to have accurate total and average connection times. Thus, we plot in Figure 10 the average amount of time (in minutes), that each type of client was connected to the Armiarma network monitoring tool. As we can see, the results between Teku and Prysm reverse, Teku with a high number of connections, spends little amount of time connected to our tool, while on the other hand Prysm with a low number of connection does remain long periods of time connected to the tool. As mentioned above, this reflects once again the different peering strategies of the clients. Meanwhile, Lighthouse, Nimbus and the unclassified peers follow similar proportions between connections and time connected.
The time connected to a peer does provide some insight into the stability of the client. Some clients might be more aggressive than others pruning peers to avoid bad actors. However this does not say much about the level of participation of the different clients in the network. For that, we measure the total number of Beacon Block, Aggregate and Proof messages received from each peer, grouping them by type of client.
As observed in Figure 11, while Prysm clients have more stable connections, Nimbus peers relayed about 8 times more Beacon Block messages than Prysm peers. Also, we notice that Lighthouse and Teku relayed a very low number of messages. This behaviour also repeats with the Beacon Aggregation and Proof messages, plotted on figure 12.
The reason behind this behaviour is directly related to the GossipSub protocol. In GossipSub, nodes have two types of peers, a few ones inside the mesh and those outside the mesh. For those inside the mesh, all messages are sent directly. However, GossipSub saves network bandwidth consumption by only sending the metadata of a message with nodes outside the mesh, before sending the whole message. If the receiver has not seen the message before, then it asks for the whole message to be sent, if it has seen it before then it does not ask for it. In addition to that, even for peers inside the mesh if the message has already been seen, GossipSub does not forward this to the client application. Thus, future messages that have been seen from different peers will not be received.
Similar results were obtained when checking the average number of received messages on the GossipSub Topics as observed on Figures 13 and 14. We wanted to understand a little better, why Nimbus and Prysm are the most chatty of the clients. Given that we are measuring who talks first to the crawler, we suspected that perhaps in this case, network latency could have an important impact. So we studied the average latency for each client. As we can observe in Figure 15, both Prysm and Nimbus have the lowest latency rate on average. This fits well with the clients that have the highest average number of messages, explaining the phenomena we observe. Overall, we can still see that, despite having a higher average latency rate, Lighthouse and Teku are still able to provide a significant number of messages, showing contribution to the network.
Leaving the type of client aside, we also wanted to see if all peers communicate in relatively similar amounts or if there was any node communicating significantly more than others. Thus, we plot in Figure 16 the number of beacon Block messages received from each peer. As we can see, there are five peers that communicated over a thousand Beacon blocks, while the large majority of the peers did not communicate more than ten. More precisely, 75% of Beacon Blocks were received from only ten peers, while 91.3% of the peers transmitted less than ten Beacon Blocks. Keep in mind that due to the way the GossipSub protocol works, this does not mean that the large majority of peers do not communicate at all, simply that a few ones are systematically the first ones to transmit new unseen Beacon Blocks to the Armiarma crawler.
In addition, we analyzed how many messages, peers send with respect to the time they keep connected. A node transmitting too many messages in a very short connection time could possibly signal a deny of service (DoS) attack. While a node with a very long connection but very few or no messages at all, could signal an attempt of eclipse attack.
The distribution is shown in Figure 17. Note that in this figure we count Beacon Blocks as well as Proof Aggregations messages. The distribution shows a dense cloud of peers that stayed connected between 5 and 8 hours and transmitted between 5 and 1000 messages. Particularly interesting, is the case of the dot in the top right corner of the figure. This peer transmitted almost 100,000 messages and stayed connected around 36 hours. We could call this a super-peer, because it shows an extremely stable connection and it also transmits a huge amount of Beacon data to its peers in a timely manner.
|City||New Jersey, North Bergen|
|Beacon Block messages||0|
|Beacon A. And P. messages||0|
|Voluntary Exits messages||0|
|Proposer Slashing messages||0|
|Attester Slashing messages||0|
Iv-C Individual Peer Analysis
The large analysis performed on the Eth2 network has been done from peer related information and it shows a global view of the network. But Armiarma also has the ability of analyzing peers individually. Armiarma can extract a large amount of information regarding specific peers. To showcase an example we show on the Table III the information extracted from one of the peers in the network. For security and privacy reasons, we have omitted sensible information as peer Id, node Id, public key and IP related to the peer. This peer showed an unusually high latency which caught our attention. Thus, Armiarma can help us monitor and keep track of nodes suspected of malicious behaviour. This demonstrate that Armiarma could be used in security analysis, in addition to the network status study.
In this paper we have presented some of the information that can be gathered transparently with our network monitoring tool, Armiarma, directly showing a perception of the network, allowing us to get a better overview of the Eth2 network state. The presented analysis shows the status of the recently launched Eth2 main net, that, as on the date of writing this paper, is experiencing a healthy behaviour.
Overall, we can see the large amount of information that the Armiarma analyser can produce while checking the internal behaviour of different clients in the network. While in this case we mostly focus on different clients, we could also search for differences between versions of the same client and therefore track long term improvements in the clients development.
Both Rumor and Armiarma are tools with a huge potential that are still on a development stage. As future work, we would like to keep contributing to the development of Rumor and keep improving Armiarma as well. In particular, we plan to increase the customization level of the host, increase the Eth2 functionalities, develop a GossipSub protocol modification customized for Armiarma analyzer and increase the range of metrics we are gathering.
This work has been supported by the Ethereum Foundation under Grant FY20-0198. We would like to thank the researchers of the Ethereum Foundation for their feedback and suggestions. In particular, we would like to thank @Protolambda as the main Rumor maintainer for his help setting up our experiments and his constructive feedback on this study.
-  Armiarma-crawler repository:. Note: https://github.com/Cortze/rumor/tree/armiarma/alpha0 Cited by: §III-A.
-  Eth2 main net info. Note: https:https://github.com/eth2-clients/eth2-mainnet Cited by: §III-A.
-  Ethereum whitepaper. Note: https://ethereum.org/en/whitepaper/ Cited by: §I.
-  General specs of the dv5 protocol. Note: https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md Cited by: §II-B, §III-A.
-  Phase-0. Note: https://notes.ethereum.org/@djrtwo/Bkn3zpwxB Cited by: §I.
-  (2019) GossipSub: a secure pubsub protocol for unstructured, decentralised p2p overlays. Cited by: §I.