SToN: A New Fundamental Trade-off for Distributed Data Storage Systems

01/23/2022
by   Bastien Confais, et al.
0

Locating data efficiently is a key process in every distributed data storage solution and particularly those deployed in multi-site environments, such as found in Cloud and Fog computing. Nevertheless, the existing protocols dedicated to this task are not compatible with the requirements of the infrastructures that underlie such computing paradigms. In this paper, we initially review three fundamental mechanisms from which the existing protocols are used to locate data. We will demonstrate that these mechanisms all face the same set of limitations and seem to have a trade-off in three distinct domains of interest, namely, i) the scalability, ii) the ability to deal with the network topology changes and iii) the constraints on the data naming process. After laying out our motivation and identifying the related trade-offs in existing systems, we finally propose a conjecture (and provide a proof for this conjecture) stating that these three properties cannot be met simultaneously, which we believe is a new fundamental trade-off the distributed storage systems using the three fundamental mechanisms have to face. We conclude by discussing some of the implications of this novel result.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

page 5

page 6

page 9

page 12

page 16

08/10/2020

Cloud Fog Architectures in 6G Networks

Prior to the advent of the cloud, storage and processing services were a...
07/26/2019

ServerMix: Tradeoffs and Challenges of Serverless Data Analytics

Serverless computing has become very popular today since it largely simp...
02/21/2018

RStore: A Distributed Multi-version Document Store

We address the problem of compactly storing a large number of versions (...
02/21/2018

Managing and Querying Multi-versioned Documents using a Distributed Key-Value Store

We address the problem of compactly storing a large number of versions (...
04/09/2019

Cold Storage Data Archives: More Than Just a Bunch of Tapes

The abundance of available sensor and derived data from large scientific...
11/18/2019

GPT Conjecture: Understanding the Trade-offs between Granularity, Performance and Timeliness in Control-Flow Integrity

Performance/security trade-off is widely noticed in CFI research, howeve...
01/07/2022

Microeconomic Foundations of Decentralised Organisations

In this article, we analyse how decentralised digital infrastructures ca...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Fog computing infrastructures have been proposed as an alternative to cloud computing to provide computation abilities with minimal latency guarantees which makes it particularly suited for the Internet of Things (IoT) era [2]. It is clear that low latency response times for heavy-load computing tasks should come with an efficient data storage system backend. In particular, this infrastructure is to be spread geographically in several sites for endurance and reliability. On the other hand, designing a scalable storage solution that constrains the network traffic while at the same time places the data at an appropriate location to provide the lowest access times possible is a major challenge [48, 33]. As of today, no consensus exists to our knowledge on the means and ways to be able to address this issue. One of the important aspects of a distributed data storage solution is the ability to locate data in an efficient manner. In a previous work [7], it has been argued that an ideal storage solution should possess five important properties: (i) data locality, (ii) network containment, (iii) disconnected mode of operation, (iv) mobility and (v) scalability.

The main role of a data location protocol is to access the target server storing a specific piece of data through the use of a data identifier known by the user (“Data Identifier” block on Figure 1). Depending on the application, this can be an Uniform Resource Identifier (URI), a filename, or any string that can be uniquely associated with each piece of data. Many protocols exist for locating data from the data identifier. Protocols like a Distributed Hash Table (DHT) or approaches like Information-Centric Networks (ICN) take the identifier and route the request directly to the node storing the piece of data (arrow between the “Data identifier” block and the storage node on Figure 1), while others primarily identify a server address before routing the request through an IP network (arrow between the “Data identifer” block and the “Location identifier” block on Figure 1 and the arrow between the “Location identifier” block and the storage node). Figure 1 illustrates these two scenarios in a generic environment with protocols locating data directly from the Data identifier and other that needs to use an intermediate location identifier.

For instance, protocols such as DNS act like a mapper between the identifier of the data and the identifier of the node storing the data. Similar functionality can be observed in protocols such as Distributed Hash Table (DHT) or an Information Centric Network (ICN) routing. Finally, the last category consists of protocols that take as input both the data identifier and the location identifier.

All of these protocols can be implemented in different ways and yet they all have to satisfy at most two of the three properties dictated by the Brewer’s theorem (CAP theorem) [14]: Consistency, Availability and a failure tolerance mechanism against network Partitioning.

Figure 1: A generic process including request and response to access a data object in a distributed environment.

In this study, the set of contributions can be listed as follows:

  • We clearly demonstrate the limitations of the existing protocols for three mechanisms used to locate the data in a distributed environment;

  • For the existing protocols considered, we establish an important trade-off that no distributed data storage solution can be Scalable (S), Tolerant to the Dynamic Network topology changes (To) and allow No Constraints (cannot impose any limitations) on the data Naming (N) or on the data placement;

  • We finally provide a formal proof for this conjecture that we believe characterizes a new fundamental trade-off in distributed data storage systems using the three fundamental mechanisms described earlier.

The provided proof applies to the existing protocols we could have identified in the literature. However, it is not general proof, rather proof that covers the three types (Directory-based, Hash-based, Exploration-based) of protocols which we believe are the only protocols presently used by all existing systems.

The rest of the article is organised as follows. In Section 2, we present the properties we expect from an ideal data location protocol in a Fog computing environment. Then, in Section 3

, we propose to classify the data location protocols in three different categories. For each category, we present some existing protocols and show why they do not meet the properties presented in the previous section. We then propose a fundamental conjecture that every data access protocol of a given distributed data storage system has a trade-off between i) scalability, ii) tolerance to physical network topology changes and iii) constraints on the data placement and naming. We finally formulate a proof of this conjecture in Section 

4, and point out limitations and implications of this new result in Section 5. We finally conclude our paper in Section 6.

2 Properties of an ideal data location protocol

We initially present the properties we desire in a data location protocol designed for a multi-site environment, and in particular for a Fog computing environment.

Scalability (S)

Scalability is the ability of a process, a network, or a software program to grow and meet the increased demand without any observable performance degradation. Two conclusions can be drawn from scalability: There is no node contacted in each request that would create a system bottleneck and there is no database that is growing boundlessly which stores a record for each piece of data stored in the system.

To our point of view, the scalability is an important performance measure because a protocol adapted in a Fog computing environment should be able to work with a large number of nodes geographically spread and store large volumes of data at the same time. Jogalekar et al. [19] proposed to measure the scalability of a distributed system through Quality of Service (QoS) metrics. We instead propose in this article to measure scalability using:

  1. the size of the table used to store the data location;

  2. the number of nodes storing such tables;

  3. the number of requests to determine the location of data;

  4. the number of network links used during this process.

To our point of view, a system is scalable as long as the execution of a request does not require to contact all the nodes in the network or it does not require each node to store a record per data fragment.

Tolerance to Dynamicity of the Network (To)

The Tolerance (To) to the physical network topology changes is the ability for a system to automatically reconfigure when the network topology is modified, i.e., some network links or nodes are added or removed, having its location identifier (i.e., its IP address) changed. In other words, changing the topology does not require any additional action of the system’s administrator. Note that such changes in the physical network topology also require to update the network overlay to reflect the newly added or removed nodes.

The difference between To and the Partition tolerance (P) in the conventional CAP context is that in To any network topology change is permanent and not the consequence of a failure that would likely be fixed in the near future. This property can be evaluated/quantified with the network traffic needed to reconfigure the data storage solution and the associated number of reconfiguration steps.

No Constraints on data Naming/Placement (N)

The absence of any constraint on data naming is the ability for the user to choose the name (data identifier) used to access the contents of the data. In the same way, the absence of constraint on the data placement enables the user to choose the node on which they want to store their data. This enables the user to make a choice regarding the policy in terms of privacy, durability or access rights proposed by different servers. Moreover, it provides a "stable" name that does not change regarding the placement of the data fragments if the data has to move for technical reasons, without the direct consent of the user.

Notation: In the following, we use as the number of nodes in the network and to represent the number of data chunks stored. We also denote to be the path with the highest number of hops between any two hosts in the network. If the topology is well-balanced, the equality usually holds but if all the nodes are interconnected within a chain topology, we shall have .

3 A comparative study of the existing protocols

In previous sections, we have presented the properties that an ideal data location protocol must possess. Let us now explore how the existing solutions behave within the context of these properties, namely the scalability, ability to tolerate physical network topology changes, and the constraints on the data placement/naming.

To simplify the presentation, we propose a classification of the existing protocols that are used to locate data into three distinct categories:

  1. Protocols based on a directory which can be implemented in centralized or distributed fashion;

  2. Protocols based on some form of computation;

  3. Protocols based on an exploration of the network.

As we shall see, this crude classification enable us to simplify the presentation of the concepts used by different protocols. We also believe that any other existing protocol can be classified under either one of these categories.

3.1 Directory-based protocols

This category of protocols relies on a database which can reither be centralised or distributed. These databases store the link between the data identifier and the location of a replica (the original content of the data).

3.1.1 Central server

The first approach is to deploy this database on a central server which is then requested each time a client needs to locate the data. This approach is used in many distributed data storage solutions such as High Performance Computing (HPC) (for instance, in PVFS [5], Lustre [12]) or I/O intensive Scale-Out NAS (for instance in RozoFS [34]).

In all of these solutions, clients primarily inquire about a central server to determine the right node storing the data fragment before any attempt is made to reach out that node. The main advantage is that adding or removing a storage node does not impact the central server and each data fragment can be localized in only a single request.

The main drawback of this approach is its centralized nature that degrades the scalability. Using a centralized server cannot handle the burden of the increased volume of simultaneous requests and store the location of the entire data in the system without noticeable performance degradation.

3.1.2 Domain Name System (DNS) protocol

In order to remove the bottleneck due to using a centralized server, the Domain Name System (DNS) protocol was proposed. DNS is a distributed directory [31] designed to provide scalability. It relies not only on a hierarchical namespace of identifiers but also on a hierarchy of servers. Each server stores the data fragment locations with an identifier ending with a specific suffix. When the client initiates a request, the root server replies and returns the address of the next server managing the data’s identifiers matching a given suffix. The client then requests from this new server which can either return the data location directly or the address of another server managing the data’s identifiers of a more specific suffix. For instance, in Figure 2, the client initially requests the servers responsible for the root zone which only returns the address of the server responsible for the com., then, it requests successively the servers specified in the response of the requests until reaching the searched record. In the example request to look for “data.k1.example.com”, the client then requests the server responsible for the com. zone before being able to request the server responsible for the zone example.com knowing that the data location identifier storing the data is k1.example.com.

Figure 2: Iterative process of the DNS look-up process.

The approach of DNS leads to better scalability properties because the table of names (addresses) stored in each server is reduced (due to the distributed nature of the processing) compared to a centralized approach. Nevertheless, this reduction is not homogeneous and depends on the balance of the tree and the balance of the data identifiers used. In the worst case, one centralized server stores on the order of records whereas, in the case of a balanced tree, each server only stores location records.

On the other hand, this approach is not resilient to physical network topology changes because if the address of a server changes, it must be reflected on the server managing the parent zone [32]. Similarly, if a server in the tree becomes unreachable, all the records stored in the corresponding subtree cannot be accessed For instance, in Figure 2, if the server managing the zone com. at addr2 becomes unreachable, all the servers in the coresponding subtree and thus all the records belonging to a subzone of com. cannot be reached.

Adding resiliency to dynamic physical network topology changes cannot be done without adding extra complexity. This is because servers do not necessarily know the entire network topology within a short time. Therefore, servers cannot join their parents to change the address specified for the zone they manage. For instance in Figure 2, if the server managing the zone example.com is moved from addr4 to a new address, the server managing the zone com. at addr2 has to be updated to account for this change in the network. From a limitation point of view, we identify that the suffix needed to aggregate the data identifier within different servers is a drawback because the users cannot really choose the data identifier they want to use.

3.1.3 Gossip protocols

An alternative way to overcome the bottleneck of a centralized server is to replicate the database on every node. This is what the Gossip protocols do in general. On a set of regular intervals, each node contacts its neighbors to exchange the list of identifiers for the data fragments they store. They also exchange the location identifiers learned from other nodes. This way, each node is able to construct its own table to locate every data fragment in the network. Such a protocol is illustrated in Figure 3. The main advantage of this approach is that the location identification process consists of using this local table only without exchanging network traffic and hence the bottleneck. However, the scalability issue of the centralized approach is not actually solved because when a large amount of data is stored, the table stored by each node also increases proportionally.

Figure 3: Example of a Gossip protocol where “n1” stores a green file and “n4” stores an orange file. By propagating the knowledge, all nodes know the location of these two files.

Cloud solutions like Dynamo [8] utilizes such an approach. In addition, all of the solutions relying on a blockchain technology [9] adapt this approach because in such solutions, the blockchain is replicated on all the nodes and stores the location of each data fragment.

3.1.4 Names embedding the location

Yet another solution is to embed the database inside the data identifiers themselves, as it is shown in Figure 1. For instance, accessing a data named http://<ip address>/data or even email addresses embed the name of the server inside the identifier. In order to reach a URL such as http://<ip address>/data, the network will first route the request to the IP address and then, once the server is reached, the “data” will be queried. The main drawback of such an approach is the addition of a constraint on the data naming process. Moving a data fragment over the network leads to a name change while it allows us to easily locate the data. Nevertheless, routing the request to the destination is still a major challenge.

In IP routing schemes, each router reads the entire routing table (though might be sizable) to determine where to forward the request. The selected route is the most specific one matching the destination address. For instance, if a router has two destination addresses: 2001:db8:1::/64 to routerA and 2001:db8::/32 to routerB, a packet destined to 2001:db8:1::1 would be forwarded to routerA due to a more specific matching with the first option.

In order to limit the size of such tables, two main mechanisms are utilized in IP networks. The first mechanism is the prefix aggregation which merges several routes pointing to the same next hop. For instance, if a router stores the routes 2001:db8:abcd:1234::/64 via 2001:db8:: and 2001:db8:abcd:12ab::/64 via 2001:db8::, the two routes can be aggregated into a more general single route, namely 2001:db8:abcd:1200::/56 via 2001:db8::. This process leads to a reduction in the number of routes stored in each router with the goal of increasing performance and scalability. Nevertheless, finding the minimal set of routes that do not create loops in the network is an NP-complete problem [26]

and a few heuristics’ have already been proposed 

[43].

The second approach is to use a default route. Routers only know the routes to few IP prefixes (generally those which are administered by the same party) and forwards all the packets to a default route. Although this leads to a reduction in the number of routes in the core routers, it does not limit the number of routes in edge routers which still have to store the full routing table. Today, the full routing table in IPv4 stores more than 780 K routes111https://bgp.he.net/report/netstats which clearly illustrates the limitation of both approaches.

3.2 Hash-based protocols

A second way to locate data is to rely on a predefined mathematical function (usually through utilization of a one-way hash function) applied to the data identifier.

3.2.1 Consistent Hashing-based approaches

Consistent Hashing (CH)-based approaches such as CRUSH [47] or SWIFT’s RING [1] propose to take into account the placement constraints in the corresponding mathematical function. Although such approaches provide partial control over the data placement process, they require each network node to be aware of the entire network topology to be able to map a hash value to the node address (size of the routing tables is on the order ). The advantage is that because all nodes know the entire network topology, they just have to hash the data identifier and map it on the table to be able to find which nodes are the destination for the data. The data locating process sends only one request on the network , even if this request has to go across all the network.

Sharing the entire topology with all the network nodes makes it hard for the system to face alterations in the topology while at the same time satisfying minimum network traffic. This is because it would require to redistribute the new topology information among all the nodes (). Finally, we would like to note that the constraints on data placement and naming are the same as in the DHT because these constraints are inherent to the use of hash functions. These characteristics are summarised in Table 1.

3.2.2 Distributed Hash Tables

Distributed Hash Tables (DHT) locate the node storing a specific data fragment by hashing the name of its identifier (for instance a cryptographic function SHA-1 in the Chord DHT [44]). Figure 4 shows an example of a DHT where the key key is stored on the node with the identifier immediately greater than 42. This location identifier is found by the client before sending any request. Note that the process of locating a data fragment is completely local as it only consists of computing an appropriate hash value.

Figure 4: Example of a DHT whereby a hash function is used to determine the key key which is stored on the node with the identifier of 42.

On the other hand, the DHT possesses a routing mechanism to reach the destination node storing the data. The routing table contains the address of a particular node’s identifier and the node forwards the request to the node with the closest identifier to the destination identifier. This way, since identifiers are sorted in the routing table, there is no need to browse all possible routes. In addition, limiting the number of routes helps build a scalable system. In a DHT, each node keeps at most routes where is the size of the keyspace. For instance, in Chord DHT, each route points to the node with the identifier immediately superior to where varies from 0 to . This is illustrated in Figure 5.

Figure 5: A typical routing table in a DHT.

The inherent design of DHT is scalable because each node of the network stores a routing table with at most entries. Nevertheless, having a small routing table leads to an increase in the number of hops and thus reducing the overall data access latency performance [27, 45]. Several DHT implementations have been proposed to reduce the number of hops such as Kademlia [30] or Pastry [39]. These solutions increase the number of neighbors in order to reduce the number of hops. In fact, we realize that this performance issue (access latency) can be tied to the Brewer’s CAP theorem and its trade-off between availability and consistency. To overcome this performance issue, many storage solutions have preferred to relax the strong consistency requirement. This is one of the reasons why implementations often rely on the eventual consistency model instead [24].

This approach also supports dynamic changes in the network topology as it is able to automatically reconfigure. Adding a node to the network leads to inform the neighbors and to re-balance the data. Because of the use of a hash function which is consistent, data balancing leads to move the minimum set of data on the order of where is the number of nodes and is the replication level.

However, the drawback of this approach is that it is not possible to choose the node storing the location record for a specific data fragment. In that sense, it is possible that the location record can be stored far away from where the data is originated. In other words, in order to store a data fragment on a specific node, you need to find a data identifier that has a hash value that fits within the range of keys stored by that node. It also means that moving a data fragment would lead to a compulsory change in the data identifier.

3.3 Exploration-based protocols

The exploration-based protocols are based on clients randomly exploring the network and trying to find the data they are looking for. The main advantage of this approach is that it does not require the storage of the data location. There is no stored record to indicate that a specific data fragment is stored on a specific node just like we typically have in a local database approach. In addition, exploration methods do not impose any constraints on the location, unlike computation-based strategies.

3.3.1 Flooding

In flooding approaches, when a user looks for a piece of data fragment, nodes of the network request their neighbors whether they store the data fragment the user is asking for. If not, nodes forward the request to their neighbors. The process continues until the requested data fragment has been found or until all nodes have received the request. A Time-To-Live (TTL) associated with the request limits the loops. An exploration method based on flooding is illustrated in Figure 6. The biggest advantage of this approach is the absence of maintenance for forwarding tables. Each node is aware of its neighbors only where network connections are available.

3.3.2 Random walk

Random walk approaches appeared in literature as an improvement to flooding approaches [15]. In a random walk scheme, instead of forwarding a request to all its neighbors, each node chooses one or several specific neighbors which are the most promising. The ultimate goal is to locate the data easily on time without overloading the network links.

To select the relevant neighbors, several algorithms and approaches have been proposed. Some of them rely on bloom filtering [28] while others use complex heuristics [21]. Random walks strategies are used in storage solutions like GNUTella [37]. In practice, this approach relieves the scalability issue of the flooding approach but in the worst case, it potentially requires to contact almost all the network nodes. This approach is used in different software implementations such as GNUTella  [3] or Spinneret [38].

Figure 6: Example of a flooding approach when the Time-To-Live (TTL) specified is too low to reach the needed data object.

3.4 Discussion about the existence of other approaches

In this article, we focused on the aforementioned three mechanisms used to locate data, because we keenly believe that all the existing approaches in literature either fall under one of them or are simply a mixture of two or more mechanisms. For instance Cassandra[23] relies on a one-hop DHT which is in fact a DHT where data is placed according to a hashing algorithm and where node identifiers are distributed with a gossip mechanism. Such a solution is intended to achieve a tradeoff between the benefits and the drawbacks of the two mechanisms involved. Let us detail two specific examples of a mixture of mechanisms:

3.4.1 Information Centric Network

Information Centric Network (ICN) is a new approach that is deemed to replace an IP network. In the ICN, users do not contact a specific node but instead, they send their requests for a specific data fragment they would like to retrieve. It is the network that is in charge of routing the request to the intended node.

Many implementations have been proposed and they all rely on a local database. For instance, in Data-Oriented Network Architecture [22] (DONA), routers store the association between a data identifier and the next hop in the network. This approach clearly relies on a local database concept that has the same scalability issues mentioned earlier. Similarly, Mobility First [42] proposes to store all the routes in a central server whereas COntent Mediator architecture for content-aware nETworks (COMET) [40] and Naming Data Network (NDN) propose to store the routes in a hierarchical, DNS-like infrastructure.

3.4.2 Location Identifier Separation Protocol

The second protocol we shall focus on is Locator Identifier Separation Protocol (LISP). Standard IP protocol uses IP addresses that represent not only the location of a node (the set of networks it is connected to) but also its identifier. This is exactly what we described in Section 3.1.4.

The LISP protocol separates these two roles by introducing an association between a node identifier and its location. A variety of implementations were proposed. For instance, LISP-DHT [29] relies on a DHT associated with a computation function. Another implementation called LISP-Tree [18] relies on a local database and more specifically on a DNS-like protocol. We also note that the implementation of Saucez et al. [41] is inspired by the Open Shortest Path First protocol (OSPF) which is quite similar to a gossip protocol in the sense that the entire database is replicated on all nodes.

Through these two examples, we demonstrated that the three proposed approaches are quite commonly used in different fields of networking and distributed computing. We realize that these two protocols do not have an implementation that uses an alternative way to locate data or to make the association between identifiers.

4 A proof of the conjecture

In this section, we formulate the conjecture that no distributed storage system can be scalable, tolerant to network partitioning and letting the users choose the name of data they store without constraints. We then introduce existing fundamental distributed system trade-offs that we shall rely on, before finally establishing the proof of our conjecture in a separate subsection.

4.1 Formulation of a fundamental conjecture

We have summarized the advantages and drawbacks of different solutions in Table 1 in terms of notation for the three classes of concepts we identified and presented previously. Directory-based approaches like using a central server or a gossip protocol face a limitation in the size of the routing table, which grows with the number of objects stored (). However, in case of a request, it is easy to locate a data fragment as all the information is centralized. It takes network requests to reach the central server and requests to a gossiped database because in a gossip protocol, the database was previously replicated on all nodes to enable local access. We also note, that the central server may be located at the other side of the network and therefore the only request needs to be forwarded by links (where is the number of nodes). These approaches also allow us to add more nodes easily as the nodes just need to know the address of the central server or a few neighbors to start gossip.

Approaches like DNS or IP routing try to face the scalability limitation by introducing a hierarchical approach. This hierarchical approach comes with a hierarchical naming system and thus constraints on naming. The only difference in DNS and in IP routing is that DNS is an iterative protocol, and IP routing is a recursive protocol. Therefore, the number of requests and the number of network links used vary.

In Hash-Based protocols, the constraint on naming and placement is very strong. In a DHT, it requires requests to locate a piece of data. Because the overlay network does not map the physical topology, each request can lead to cross the entire network. Thus, the number of network links used to be . In Consistent Hashing, each node needs to know all of the nodes composing the network, thus a routing table of size is required.

Finally, in exploration-based protocols, there is no database but it requires in the worst case that each of the nodes contacts each of its neighbors. In the worst case, the network is a complete graph, thus links are required to be used.

Table 1 clearly demonstrates that none of the presented solutions is able to (1) scale with the number of data objects stored (there is no single point in the network that needs to store as many records as data objects in the network), (2) possesses a form of tolerance to the physical network topology changes and (3) enable users to choose freely where to place their data and what the data identifier would be while storing it222Note that we do not strictly mean physical location here. In other words, the user might be interested in the properties of the location s/he chooses such as privacy, durability, access privileges etc. Especially in cloud settings, the user may require such properties to hold for the data and may wish to assign a specific name for the data for compatibility reasons. Here the user can be thought as the cloud admin as well, it does not need to be a third party or a customer receiving service. As the name of the file may change over the course of its lifetime as stored in the system. However, the question is about whether or not there would be any connection between the naming convention and the storage location. When we refer to the user having all freedom to choose a name, we strictly mean that this link is broken.. In other words, it shows that none of the solutions is able to satisfy the three properties we have presented in Section 2 all at the same time.

Based on this observation, we conjecture that it is impossible to design a distributed data storage solution that can simultaneously meet these three properties mentioned above. This claim is similar (but we believe orthogonal) to the popular Brewer’s CAP theorem, which establishes that it is not possible to have a sequential consistency, availability, and support for network partitioning all at the same time [14].

addcode=

Table 1: Table showing how existing solutions deal with scalability, tolerance to physical network topology changes and constraints on naming or placement of the data for a network of nodes.

,rotate=90,center angle=60,lap=0pt-(2em) angle=60,lap=0pt-(2em) angle=60,lap=0pt-(2em)Size of a routing table angle=60,lap=0pt-(2em)Number of routing tables angle=60,lap=0pt-(2em) Number of requests to locate a piece of data angle=60,lap=0pt-(2em) Number of network links used in the data location process angle=60,lap=0pt-(2em) Network traffic to add or remove a node angle=60,lap=0pt-(2em) How to name a piece of data that we want to be placed on a server “x”? angle=60,lap=0pt-(2em) Data identifier changes when the data is moved Solution Scalability Tolerance to Network Topology Changes Constraints on Naming or Data Placement Directory-based protocols
Central server “obj” no

Gossip “obj” no

Domain Name System does not support dynamicity “obj.suffix” no


IP Routing an additional protocol is required “destination/obj” yes

Hash-based protocols
Distributed Hash Tables yes
Consistent hashing yes

Exploration-based protocols
Flooding “obj” no
Random walk “obj” no


4.2 Background: Existing fundamental trade-offs for distributed systems

The conjecture we had in the previous section is not the first fundamental trade-off in distributed systems sharing these three properties. As shown in Figure 9, there exists known fundamentals trade-offs such as the Fisher Lynch and Paterson [13] trade-off or the well-known Brewer’s theorem [4]. These trade-offs state that any distributed system can at most satisfy two out of the three important distributed system features: Consistency, Availability, and Partition Tolerance. However, other important properties need to be considered together in a distributed system design.

(a)
(b)
Figure 9: An illustration of the impossibility results of the famous CAP (a) and the FLP (b) theorems.

4.2.1 Property definitions

In addition to the properties we defined in Section 2, we need to formally define some more additional properties for convenience.

Consistency (C)

In our context, we assume consistency to mean strong consistency (linearizability) just like in CAP. This means that all operations can be linearized in total ordering. In other words, any read operation that begins after a write operation completes must return that value, or the result of a later write operation.

Safety

The safety is a relaxation of consistency. In a safe system, nodes have to agree on a consensus of the value returned. But, contrary to Consistency (C), Safety does not guarantee the value returned to be the last written one.

Availability (A) and Liveness

An available system is a system in which any request must eventually be addressed. However this notion of availability referred in the Brewer’s theorem [14] is not very clear because it is not specified what amount of delay is tolerable. For instance, it may be possible to wait for an unlimited amount of time, turning the system not live. In order to clarify this, we define in the following the -availability (-A) to characterize systems with a response given within a threshold time . We note that if we obtain the liveness property which would be a special case of this availability definition.

Partition tolerance (P)

The network partition tolerance refers to the fact that due to unexpected node failures, the network can arbitrarily be split into several connected components called network partitions and messages sent between partitions are permanently lost.

Fault tolerance (F)

The fault tolerance is the ability of a system to continue its normal operation without interruptions when a subset of nodes fails. This can be implemented in the form of a hardware or a software fault mitigation system.

4.2.2 Fisher, Lynch and Paterson (FLP) theorem

One of the fundamental trade-offs in distributed systems is the impossibility theory of Fisher Lynch and Paterson (FLP) [13]. This theorem states that in a reliable network, it is impossible to have a deterministic algorithm to solve the consistency problem in case of any node failure or in case of an asynchronous system model. In other words, no distributed system can possess safety, liveness, and fault tolerance properties all at the same time.

4.2.3 Brewer’s CAP theorem

A second fundamental trade-off is the Brewer’s CAP theorem [4]. This theorem states that in an asynchronous network model, it is not possible to create a distributed solution that is tolerant to network partitioning, available, and sequentıally consistent, all at the same time.

The consequence of the CAP theorem is that all distributed data storage solutions have to make a choice. Either they guarantee strong consistency, they enable users to access the latest version of the data without guaranteeing a quick response in case of a network partition (CP) or they stay available (quick response) when some of the nodes fail without guaranteeing a strong consistency (AP). Under the light of this observation, we will state our theorem by considering these two kinds of systems separately.

While this theorem is criticised [36, 17], especially for its lack of definitions for the properties, it is still a viable and a fundamental reference to characterise distributed systems.

4.3 A Proof

In this subsection, we will prove that no distributed data storage solution can meet simultaneously all three properties i) Scalability (S), ii) Tolerance to Dynamicity of the Network topology (To) and iii) the absence of constraints on data placement/naming i.e., No Constraint on data placement/Naming (N). This is illustrated in Figure 10.

Figure 10: An illustration of the impossibility result we desire to prove.

To prove this conjecture, we primarily take in to account the implications of the CAP theorem. In other words, since partition tolerance is an inevitable requirement of any distributed system, we consider two potential cases. In the first case, we prove the impossibility result for Available (A) systems that can also deal with Network Partitioning (P). In the second case, we will establish the proof for the systems that possess Consistency (C) and the tolerance to Network Partitioning (P) properties at the same time.

We emphasize that we do not consider solutions that are Available (A) and strongly Consistent (C) without any tolerance for Network Partitioning (P) because this last property is essential in distributed systems as soon as the storage solution is spread among several geographical locations. Another argument for not considering solutions that are available and consistent has been exposed in an extension of the CAP theorem. Abadi et al. [16] considers that network partitioning is not a characteristic of the system but a characteristic of the environment. The system’s property is its reaction to this partitioning. They propose the PACELC model that can be summarised in the following way: When the network is partitioned (P), there is a trade-off between Availability (A) and Consistency (C). But, when the network forms a connected component, the trade-off is between Latency (L) and Consistency (C). Finally, Diack et al. showed that it is impossible to have Consistency and Availability simultaneously [10]. For these three reasons, we do not consider a non-practical solution that would be both available and consistent.

4.3.1 A Proof for Available and Network Partition Tolerant (AP) systems

In this case, we consider all the possible mechanisms to locate data in a -AP distributed system and state a hypothesis. More specifically, we assume that a system possesses all three features: S, To, and the N. In the following paragraphs, we consider different categories of protocols and we show that for each one of them, the hypothesis always leads to a contradiction.

Case 1: A Solution relying on a location database

Under the circumstances of the existence of a location database, a node needs to request the database to determine the location of data. Because the system is both -AP and (N), this request is performed at the time a user likes to access a specific data fragment.

However, there exist different ways of setting up the database. Using a centralized database leads all the network flows to concentrate on a specific machine resulting in a system bottleneck. On the other hand, replicating the database on a few nodes can easily be implemented (since strong consistency is not a must) and can solve the aforementioned bottleneck. In both scenarios, however, the database has to store the location of every data fragment which, according to the definition presented above, would violate the Scalability property (S). In addition, the address of the central database should be communicated to all the other nodes for data transfer. So changing the address of the database may be quite difficult without violating the tolerance to the dynamicity of the network topology property i.e., To.

Finally, hierarchical databases, like we have seen in DNS where a “tree” approach is adapted can be regarded as a potential solution. In that, a central node may be first contacted but this time instead of responding with the location of a specific data fragment, it would respond with the address of the next database to contact to. Nevertheless, the root node should still be contacted at each request and known to all the network nodes. Such a tree approach may alleviate the bottleneck but does not completely remove it.

Case 2: A Solution relying on a computational method

For a -AP solution, nodes may use a computational method to determine the location of a data fragment. Typically, these computational methods serve as functions and take the name of the data fragment as input and map it to the location identifier of the node storing a data replica. In this particular case, however, the location of each data fragment is completely dependent on its identifier and therefore the computational methods violate the (N) property.

Note that due to the dynamicity of the network, nodes randomly enter or leave the network. Therefore, a mechanism is required to update the mapping function accordingly and enable the solution to be tolerant to physical network topology changes. Such mechanisms should rely on a local database or an exploration of the network, and therefore will have the same line of limitations similar to the ones presented in this paper.

Finally, it is interesting to note that some computational methods allow users to define “constraints” on the data placement process, allowing them to have a computational method that satisfies the (N) property. For instance, in CRUSH [47], the placement rules enable users to choose the node a specific piece of data is stored on. However, rules are replicated among all the nodes. Therefore, if the placement of all data fragments is constrained, CRUSH algorithm would not act like a hashing mechanism but more like a local database.

Case 3: A Solution relying on an exploration of the network

The third approach to finding the location of the data fragment is the exploration of the network. In this case, a node can locate the data fragment location (destination) in a number of hops as long as the Time-To-Live (TTL) allows.

Let us first assume a system that possesses the following characteristics: -AP, Scalable (S), Tolerant To network changes (To) and has no constraints on the data placement or naming (N). Since the solution is -AP, a finite TTL must be used to guarantee a response in a finite time and hence the availability of the whole system. But at the same time, there is no guarantee to find the right data fragment location within the specified TTL, making the solution violating the AP characteristics. Therefore, this argument clearly illustrates the contradiction and a -AP-S-To-N solution cannot exist when the underlying protocol relies on the exploration of the network.

4.3.2 A Proof for Consistent and Network Partition Tolerant (CP) systems

In the previous subsection, we showed that meeting the properties S, To and N all at the same time is impossible for a partition-tolerant distributed data storage system which chooses availability over consistency. In this subsection, we argue that exact same conclusion can be made for a distributed data storage solution that overweights strong consistency over availability.

Case 1: A solution relying on a location database

Using a similar rationale of the previous subsection, let us assume a system that meets the following characteristics: Strong consistency (C), Scalability (S), Tolerance to network changes (To) and has no constraints on the data placement or naming (N). To achieve a strong consistency, there are two options: A local centralized database may be used or it would be required to get the location of all replicas and execute a consensus protocol. In the first option, the centralized server cannot scale and shall violate the scalability (S) property. In the second option, if multiple nodes have to reach a consensus, the properties S and To both would be required at the same time. For instance, the Paxos protocol [25] requires to have a stable network in a partition that contains more than half of the nodes. Therefore, it is not possible to achieve the tolerance to network dynamicity (To) property. To the best of our knowledge, no consensus protocol is able to scale in a dynamic environment. Protocols such as Practical Byzantine Fault Tolerance (PBFT) [6] have the same characteristics.

In blockchain architectures, the chain is replicated in all network nodes and each node keeps track of all the old records (blocks) due to the immutability guarantees. Although this leads to serious scalability issues [35], the process to achieve a consensus when data is added has also its own limitations. In blockchains, a transaction needs to be checked by several different nodes before its insertion into the chain. Even if the chain itself is not replicated in a consistent manner, the process to insert a new piece of data guarantees consistency. The number of nodes required to validate the transaction creates a trade-off between the performance and the security of the chain [49] (i.e., the consistency). Several solutions have been proposed to overcome these limitations [49, 46]. All these solutions propose to relax the consensus of the read value to maintain acceptable performance and therefore the system would have to violate the consistency (C) property.

As a consequence, a system that uses a location database is not plausible if we are to require S, To and N all meet at the same time.

Case 2: A solution relying on a computational method

In the case of a CP solution, nodes use a computational method to determine the location of a data fragment. Similarly, these computational methods serve as functions and take the name of the data fragment as input and return the location. Since the input is the name of the data, such methods obviously would violate the N property. In other words, the location of a data fragment is completely dependent on its assigned identifier. Similar to AP solution, these mapping functions require each node to know the list of all node identifiers in order to set the range of values the function has to return.

Finally, the function needs to be updated each time a node enters or leaves the network. This implies the nodes to be able to determine a consensus about the function to use. Therefore, the function (or a parameter used by the function) can be distributed using a mechanism relying on a local database or an exploration of the network. These mechanisms suffer from the same limitations as the ones presented in this paper.

Case 3: A solution relying on an exploration of the network

Finally, in the case of an exploration of the network, guaranteeing the strong consistency can be done at either the write or read levels. In both cases, it requires to retrieve or to write many data replicas. Depending on how replication is performed, achieving consistency may require to retrieve (or write) at least times, where is the number of replicas. Note that this would imply in the worst case to potentially reach all the nodes in the network, this approach cannot be used with a TTL lower than the diameter of the network. More specifically, we would need to reach at least nodes as required by consensus protocols which would typically lead to contact more nodes because not all network nodes store a data replica. This obviously leads to a scalability (S) issue as the network grows. Also, the use of the consensus algorithm implies that, after a sufficient amount of node failures, no consensus protocol would be able to operate accurately leading to a failure to simultaneously maintain consistency (C) and the required tolerance to physical network topology changes (To).

Figure 11: A summary of the proof depending on the system’s characteristics considered and the implementation.

In conclusion, all of these demonstrations are summarised in Figure 11. We demonstrated that none of the three studied mechanisms to locate data is able to satisfy S, To and N properties. Therefore combining our arguments for AP systems in the previous subsection, we proved that these three properties cannot be met simultaneously in both potentially viable distributed systems i.e., systems that guarantee strong consistency (CP) and systems that guarantee availability (AP).

5 Limitations and Implications

First of all, the presented conjecture takes in to account three fundamental mechanisms to locate data in a distributed environment, namely (1) the directory-based and (2) the computation-based methods as well as (3) the approaches based on the exploration of the network. To the best of our knowledge, these are all the mechanisms in literature to locate the data. On the other hand, the present proof does not take into account any other mechanisms (if they ever exist) that may be potentially used for locating data. This can be regarded as one of the important limitations of this study. We are also aware that as new mechanisms emerge in the distributed system research, the conjecture as well as its proof may need to be revised.

One of the consequences of our result is that classical approaches cannot be used to create location protocols in a Fog computing environment because, as defined in the introduction section, the ideal Fog storage solution requires (i) storing the data locally which would imply no constraints on the data placement (N), (ii) reducing the network traffic, (iii) supporting the mobility which is tolerant to the dynamicity of the network (To), and finally, which is (iv) scalable and is able to support a large number of Fog servers while being able to store a large amount of data.

One of the other consequences that we expect to come out of this result is that protocols such as Mobile IP [20] shall not be able to find a distributed method to store the locations of the nodes that would require all three properties at the same time. This observation also directly impacts the way we distribute information in multiple controllers of a Software Defined Network [11] which will bring forth many more implications of our result on the design of next generation communication protocols.

6 Conclusion and Future work

In this article, we initially reviewed different protocols for locating data fragments in a distributed environment and grouped them under three main categories. Some of these solutions rely on a local database. Others use either a computational method or are based on an exploration of the network at the data request time. We argued that no solutions from all three categories are able to meet the requirements for deployment in a distributed environment such as Fog infrastructures. In other words, these three solutions could not meet the scalability (S), the tolerance to dynamic physical network topology changes (To) without imposing constraints on the data placement or naming process (N). We finally proved our conjecture with regard to the CAP context by structuring our arguments orthogonally. We showed no protocol can satisfy the three properties at the same time for both available and partition tolerant and consistent and partition tolerant solutions. This implies that the ideal storage solution for some of the distributed environments such as Fog computing cannot satisfy the properties we expect. As future work, we plan to extend our proof for emerging data locator protocols if they ever appear in literature and look for methods to achieve the best compromise between these properties for a given distributed system application.

References

  • [1] J. Arnold (2014) OpenStack swift: using, administering, and developing for swift object storage. 1st edition, O’Reilly Media, Inc.. External Links: ISBN 1491900822, 9781491900826 Cited by: §3.2.1.
  • [2] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli (2012) Fog computing and its role in the internet of things. In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing, MCC ’12, New York, USA, pp. 13–16. External Links: ISBN 978-1-4503-1519-7, Document Cited by: §1.
  • [3] F. R. Bordignon and G. H. Tolosa (2001) Gnutella: distributed system for information storage and searching model description. J. Internet Technology 2 (2), pp. 171–184. Cited by: §3.3.2.
  • [4] E. Brewer (2010) A certain freedom: thoughts on the cap theorem. In Proceedings of the 29th ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, PODC ’10, New York, USA, pp. 335–335. External Links: ISBN 978-1-60558-888-9, Document Cited by: §4.2.3, §4.2.
  • [5] P. H. Carns, W. B. Ligon,III, R. B. Ross, and R. Thakur (2000) PVFS: a parallel file system for linux clusters. In Proceedings of the 4th Annual Linux Showcase & Conference - Volume 4, ALS’00, Berkeley, CA, USA, pp. 28–28. Cited by: §3.1.1.
  • [6] M. Castro and B. Liskov (2002-11) Practical byzantine fault tolerance and proactive recovery. ACM Trans. Comput. Syst. 20 (4), pp. 398?461. External Links: ISSN 0734-2071, Link, Document Cited by: §4.3.2.
  • [7] B. Confais, A. Lebre, and B. Parrein (2017-08) Performance Analysis of Object Store Systems in a Fog and Edge Computing Infrastructure. Transactions on Large-Scale Data- and Knowledge-Centered Systems. External Links: Document Cited by: §1.
  • [8] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels (2007-10) Dynamo: amazon?s highly available key-value store. SIGOPS Oper. Syst. Rev. 41 (6), pp. 205?220. External Links: ISSN 0163-5980, Link, Document Cited by: §3.1.3.
  • [9] D. Di Francesco Maesa, P. Mori, and L. Ricci (2017) Blockchain based access control. In Distributed Applications and Interoperable Systems, L. Y. Chen and H. P. Reiser (Eds.), Cham, pp. 206–220. External Links: ISBN 978-3-319-59665-5 Cited by: §3.1.3.
  • [10] B. W. Diack, S. Ndiaye, and Y. Slimani (2013-Apr.) CAP theorem between claims and misunderstandings: what is to be sacrificed?. International Journal of Advanced Science and Technology 13, pp. 01 – 12. External Links: Link Cited by: §4.3.
  • [11] A. Dixit, F. Hao, S. Mukherjee, T.V. Lakshman, and R. Kompella (2013) Towards an elastic distributed sdn controller. In Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking, HotSDN ?13, New York, NY, USA, pp. 7?12. External Links: ISBN 9781450321785, Link, Document Cited by: §5.
  • [12] S. Donovan, G. Huizenga, A. Hutton, C. Ross, M. Petersen, and P. Schwan (2003) Lustre: Building a file system for 1000-node clusters. In Proceedings of the Linux Symposium, Cited by: §3.1.1.
  • [13] M. J. Fischer, N. A. Lynch, and M. S. Paterson (1985-04) Impossibility of distributed consensus with one faulty process. J. ACM 32 (2), pp. 374–382. External Links: ISSN 0004-5411, Link, Document Cited by: §4.2.2, §4.2.
  • [14] S. Gilbert and N. Lynch (2002-06) Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. SIGACT News 33 (2), pp. 51–59. External Links: ISSN 0163-5700, Document Cited by: §1, §4.1, §4.2.1.
  • [15] C. Gkantsidis, M. Mihail, and A. Saberi (2004) Random walks in peer-to-peer networks. In IEEE INFOCOM 2004, Vol. 1, pp. 130. Cited by: §3.3.2.
  • [16] W. Golab (2018-03) Proving pacelc. SIGACT News 49 (1), pp. 73–81. External Links: ISSN 0163-5700, Link, Document Cited by: §4.3.
  • [17] Q. Huang, D. A. Freedman, P. Dowell, and K. P. Birman (2012-02) Overcoming cap with consistent soft-state replication. Computer 45 (02), pp. 50–58. External Links: ISSN 1558-0814, Document Cited by: §4.2.3.
  • [18] L. Jakab, A. Cabellos-Aparicio, F. Coras, D. Saucez, and O. Bonaventure (2010) LISP-tree: a dns hierarchy to support the lisp mapping system. IEEE Journal on Selected Areas in Communications 28 (8), pp. 1332–1343. Cited by: §3.4.2.
  • [19] P. Jogalekar and M. Woodside (2000) Evaluating the scalability of distributed systems. IEEE Transactions on Parallel and Distributed Systems 11 (6), pp. 589–603. Cited by: §2.
  • [20] D. B. Johnson, J. Arkko, and C. E. Perkins (2011-07) Mobility Support in IPv6. Request for Comments, RFC Editor. Note: RFC 6275 External Links: Document, Link Cited by: §5.
  • [21] A. W. Kazi (2010) Prefetching bloom filters to control flooding in content-centric networks. In Proceedings of the ACM CoNEXT Student Workshop, CoNEXT ?10 Student Workshop, New York, NY, USA. External Links: ISBN 9781450304689, Link, Document Cited by: §3.3.2.
  • [22] T. Koponen, M. Chawla, B. Chun, A. Ermolinskiy, K. H. Kim, S. Shenker, and I. Stoica (2007-08) A data-oriented (and beyond) network architecture. SIGCOMM Comput. Commun. Rev. 37 (4), pp. 181–192. External Links: ISSN 0146-4833, Document Cited by: §3.4.1.
  • [23] A. Lakshman and P. Malik (2010-04) Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44 (2), pp. 35–40. External Links: ISSN 0163-5980, Link, Document Cited by: §3.4.
  • [24] A. Lakshman and P. Malik (2010-04) Cassandra: a decentralized structured storage system. SIGOPS Oper. Syst. Rev. 44 (2), pp. 35–40. External Links: ISSN 0163-5980, Document Cited by: §3.2.2.
  • [25] L. Lamport (2002) Paxos Made Simple, Fast, and Byzantine. In Procedings of the 6th International Conference on Principles of Distributed Systems. OPODIS 2002, Reims, France, December 11-13, 2002, pp. 7–9. Cited by: §4.3.2.
  • [26] F. Le, G. G. Xie, and H. Zhang (2011) On route aggregation. In Proceedings of the Seventh COnference on Emerging Networking EXperiments and Technologies, CoNEXT ’11, New York, USA. External Links: ISBN 978-1-4503-1041-3, Document Cited by: §3.1.4.
  • [27] J. Li, J. Stribling, T. M. Gil, R. Morris, and M. F. Kaashoek (2005) Comparing the performance of distributed hash tables under churn. In Peer-to-Peer Systems III, G. M. Voelker and S. Shenker (Eds.), Berlin, Heidelberg, pp. 87–99. External Links: ISBN 978-3-540-30183-7 Cited by: §3.2.2.
  • [28] A. Marandi, T. Braun, K. Salamatian, and N. Thomos (2017) BFR: a bloom filter-based routing approach for information-centric networks. In 2017 IFIP Networking Conference (IFIP Networking) and Workshops, Vol. , pp. 1–9. Cited by: §3.3.2.
  • [29] L. Mathy and L. Iannone (2008) LISP-dht: towards a dht to map identifiers onto locators. In Proceedings of the 2008 ACM CoNEXT Conference, CoNEXT ’08, New York, NY, USA. External Links: ISBN 9781605582108, Link, Document Cited by: §3.4.2.
  • [30] P. Maymounkov and D. Mazieres (2002) Kademlia: A Peer-to-Peer information system based on the XOR metric. In International Workshop on Peer-to-Peer Systems, pp. 53–65. Cited by: §3.2.2.
  • [31] P. Mockapetris (1987-11) Domain names - concepts and facilities. Request for Comments, RFC Editor, Network Working Group. Note: RFC 1034 External Links: Document Cited by: §3.1.2.
  • [32] R. Morera and A. McAuley (2005-10) Adapting DNS to dynamic ad hoc networks. In MILCOM 2005 - IEEE Military Communications Conference, Vol. , pp. 1303–1308 Vol. 2. External Links: Document, ISSN 2155-7578 Cited by: §3.1.2.
  • [33] M. I. Naas, P. R. Parvedy, J. Boukhobza, and L. Lemarchand (2017) IFogStor: an iot data placement strategy for fog infrastructure. In 2017 IEEE 1st International Conference on Fog and Edge Computing (ICFEC), Vol. , pp. 97–104. Cited by: §1.
  • [34] D. Pertin, S. David, P. Évenou, B. Parrein, and N. Normand (2014-04) Distributed File System based on Erasure Coding for I/O Intensive Applications. In 4th International Conference on Cloud Computing and Service Science, Barcelone, Spain. Cited by: §3.1.1.
  • [35] S. Pongnumkul, C. Siripanpornchana, and S. Thajchayapong (2017) Performance analysis of private blockchain platforms in varying workloads. In 2017 26th International Conference on Computer Communication and Networks (ICCCN), Vol. , pp. 1–6. Cited by: §4.3.2.
  • [36] R. Ramakrishnan (2012) CAP and cloud data management. Computer 45 (2), pp. 43–49. Cited by: §4.2.3.
  • [37] M. Ripeanu (2001) Peer-to-peer architecture case study: gnutella network. In Proceedings First International Conference on Peer-to-Peer Computing, Vol. , pp. 99–100. Cited by: §3.3.2.
  • [38] J. Rose, C. Hall, and A. Carzaniga (2007) Spinneret: a log random substrate for p2p networks. In 2007 IEEE International Parallel and Distributed Processing Symposium, Vol. , pp. 1–8. External Links: Document Cited by: §3.3.2.
  • [39] A. I. T. Rowstron and P. Druschel (2001) Pastry: scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms Heidelberg, Middleware ’01, London, UK, pp. 329–350. External Links: ISBN 3-540-42800-3 Cited by: §3.2.2.
  • [40] R. Salguero (2010) COntent mediator architecture for content-aware networks. Technical report COMET European Union Framework Program 7. Cited by: §3.4.1.
  • [41] D. Saucez, L. Iannone, O. Bonaventure, and D. Farinacci (2012) Designing a deployable internet: the locator/identifier separation protocol. IEEE Internet Computing 16 (6), pp. 14–21. Cited by: §3.4.2.
  • [42] I. Seskar, K. Nagaraja, S. Nelson, and D. Raychaudhuri (2011) MobilityFirst future internet architecture project. In Proceedings of the 7th Asian Internet Engineering Conference, AINTEC ’11, New York, USA, pp. 1–3. External Links: ISBN 978-1-4503-1062-8, Document Cited by: §3.4.1.
  • [43] A. Sridharan, S. Bhattacharyya, C. Diot, R. Guérin, J. Jetcheva, and N. Taft (2001) On the impact of aggregation on the performance of traffic aware routing. In Teletraffic Engineering in the Internet Era, J. M. de Souza, N. L.S. da Fonseca, and E. A. de Souza e Silva (Eds.), Teletraffic Science and Engineering, Vol. 4, pp. 111 – 123. External Links: ISSN 1388-3437, Document Cited by: §3.1.4.
  • [44] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan (2001-08) Chord: a scalable peer-to-peer lookup service for internet applications. SIGCOMM Comput. Commun. Rev. 31 (4). External Links: ISSN 0146-4833, Document Cited by: §3.2.2.
  • [45] D. Stutzbach and R. Rejaie (2006-04) Improving lookup performance over a widely-deployed dht. In Proceedings IEEE INFOCOM 2006. 25TH International Conference on Computer Communications, Vol. . External Links: Document, ISSN 0743-166X Cited by: §3.2.2.
  • [46] M. Vukolić (2016) The quest for scalable blockchain fabric: proof-of-work vs. bft replication. In Open Problems in Network Security, J. Camenisch and D. Kesdoğan (Eds.), Cham, pp. 112–125. External Links: ISBN 978-3-319-39028-4 Cited by: §4.3.2.
  • [47] S. A. Weil, S. A. Brandt, E. L. Miller, and C. Maltzahn (2006) CRUSH: controlled, scalable, decentralized placement of replicated data. In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06, New York, USA. External Links: ISBN 0-7695-2700-0, Document Cited by: §3.2.1, §4.3.1.
  • [48] M. Yannuzzi, R. Milito, R. Serral-Gracià, D. Montero, and M. Nemirovsky (2014-12) Key ingredients in an iot recipe: fog computing, cloud computing, and more fog computing. In 2014 IEEE 19th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Vol. , pp. 325–329. External Links: Document, ISSN 2378-4865 Cited by: §1.
  • [49] Q. Zhou, H. Huang, Z. Zheng, and J. Bian (2020) Solutions to scalability of blockchain: a survey. IEEE Access 8 (), pp. 16440–16455. Cited by: §4.3.2.