An Integrated P2P Framework for E-Learning

03/13/2019 ∙ by Nikita Bhagatkar, et al. ∙ Indian Institute of Technology Kanpur 0

The paper is concerned with the design and development of a P2P Presentation System (P2P-PS) for live streaming of lectures coupled with a shareable whiteboard. Video streaming uses a data-driven modified mesh architecture which supports dynamic fanout. The whiteboard may be used in synchronization with an active video streaming session. It is shared by both the presenter and the participants in the audience for graphic illustrations during the lecture. P2P-PS provides an important feature called ask doubt that allows any participant to send a query (or a doubt) during an active presentation session. The query is disseminated to all other peers by preserving the causality between ask doubt and its resolution. A buffered approach is employed to boost the performance of the whiteboard. The simulations experiments were performed with over 1000 peer nodes on Emulab using 200 physical nodes. The results show that the proposed overlay eventually stabilizes even in the presence of a churning rate of up to 30%. The maximum path length being just six hops. The estimated throughput is found to be close to our theoretical results.



There are no comments yet.


page 1

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The cloud-based collaboration platforms are of three types, namely,

  1. Collaborative creation of forms, reports or documents (e.g. Overleaf [1], Google Drive [2]).

  2. Collaborative development of large computer programs (e.g. Github [3]).

  3. Peer to peer tutoring/reciprocal learning by teaching (e.g, Duolingo [4], Coursera [5], Kahoot [6], Brainly [7]).

These collaboration tools are inspired by the concept of social networking. The underlying idea is to combine online access to web-based interfaces that can some what mimic classroom lectures with a Facebook like secure interaction environment between the teachers and the students. A few of these collaboration tools include online tests, assignments or links to external web materials. In summary, social learning platforms provide a framework for a coordinated learning environment or a discussion forum.

Brainly [7]

follows a slightly different approach. It harnesses crowdsourcing of like-minded students to combine their problem-solving skills. It uses deep learning techniques based on historical data to predict special requirements of a learner. A student gets a personalized learning experience when he/she gets connected to an appropriate peer-mentor who helps the student in understanding the problem or develop the required skills.

However, the main focus of any social learning platform is essentially a passive discussion forum

with variations such as crowdsourcing or incorporating additional features like peer mentoring. The elements of an active learning as it happens in a live classroom or a physical meeting can be captured by combining a P2P Presentation System (P2P-PS) with the features of an video conferencing system such as Skype 

[8] or Hangout [9]. The major difficulty in the suggested approach is scalability, e.g., group call limit in Skype is only 25. Most video conference systems include screen sharing feature. However it is one-to-many as many-to-many sharing does not make sense in the context of a conferencing system. Our approach, therefore, is novel as it incorporates live video streaming aided by a shareable whiteboard. We propose an immersive solution by coupling the P2P-PS with a P2P file sharing an searching system at the back-end. The back-end depends on a conventional Chord-like DHT [10] but uses a de Bruijn graph  based implementation of DHT for file sharing, file searching. It provides a number of features such annotations, comment and discussion forum. So, one can create a mashable presentation.

The rest of the paper is organized into six sections. There are two possible organizations of peers for live streaming, namely, (i) tree and (ii) mesh. Section II deals with the inadequacies associated with the tree organization and then explains why a pure mesh organization is inappropriate for the live streaming in an implementation of our system due to an additional feature which allows non-generating peers to seek clarifications during the streaming. Section III describes the modified form of mesh organization which allows us to exploit the availability of spare capacities at peers so that dynamic data-driven fanouts at peers can support live streaming. Joining and leaving of peers, handling of ’ask doubt’ feature and memory utilization by modified mesh organization all discussed in this section. Section IV focuses on design and development of Live shareable whiteboard. The results of experiments on Emulab are discussed in section V and section VI concludes the paper.

Ii Live Streaming

There are two approaches for live media streaming on a P2P system, viz.

  • Tree-based: It maintains a logical tree overlay where the root generates the stream, and other peer nodes get it from their respective parents. The leaves are free-riders receiving the flows from the others.

  • Mesh-based: A mesh architecture allows each peer to connect to other peers. It also lets a peer combine sub-streams received from more than one peer.

Other live media streaming techniques are variations of the two basic approaches.

Ii-a Inadequacy of Tree-based Live Streaming Approach

The tree model follows a push-based technique. The data propagates level by level. A node may have multiple children but one parent. If the parent leaves a session, the entire sub-tree below it gets disconnected from the network. Hence, an internal node cannot leave the tree even if it is not interested in receiving the streaming contents any longer.

Another major disadvantage of a tree is that it cannot handle dynamicity in the network. A peer may leave the tree either gracefully or involuntarily. In either case, all the existing children of the leaving peer would have to search for a new parent. It results in interruption and loss of data streams. The entire sub-tree under the leaving peer would suffer data loss until all the orphaned children can find a new parent.

The depth of a tree decreases with increase in fanout. Constructing a bushy tree helps in reducing the latency. However, the fanout bound is determined by the upload bandwidth at a node. A few internal nodes may be available with spare capacity. Such internal nodes can accept additional requests for adopting new children. Reconstruction of the tree will not be required if the number of available peers with spare capacity is equal to the degree of the departed node. Otherwise, the entire sub-tree should be reconstructed. The reconstruction of a tree incurs a significant delay in the streaming process. One of the examples of single tree overlay architecture is SpreadIt [11]. It uses a single tree for media streaming. But, as stated earlier, it is vulnerable to the dynamic nature of nodes.

The learning platform proposed in this paper supports a feature for seeking clarifications called ”Ask doubt.” A peer can invoke it seeking clarification at any point of the live streaming. The question is propagated to the presenter as well as to the other participating peers. The simplest way to implement this feature in the tree model would be to send the question directly to the root peer (the initiator or the speaker of the session). The speaker would then take the responsibility of propagating the doubt to all the descendants in the tree. It guarantees a causality relationship in ’asking a question’ and ’resolving it.’ Considering the inadequacy of tree-based streaming, we used a mesh-based approach for supporting live streaming.

Ii-B Mesh Approach for Live Streaming

In the mesh-based P2P system, the nodes randomly connect forming an overlay mesh. These connections could be used to deliver data in unidirectional or in a bidirectional way. For example, CoolStreaming [12] maintains bidirectional connections while on the other hand, PRIME [13] maintains unidirectional connections. The mesh-based approach uses swarming content delivery mechanism [14]. A peer who has received data from a server could itself act as a server for other peers. Swarming is commonly used in BitTorrent [15]. A peer collects data parallelly from other peers and combines it in a single file thereby efficiently utilizing the bandwidth of its neighboring peers. It reduces the load on the primary server because many peers now share the distribution content.

Iii Modified Mesh Approach

We use an approach slightly different from the usual mesh. It creates a mesh based on the availability of spare capacity at a peer. The modified mesh approach is thus data-driven and supports a dynamic fanout. However, the fanout value cannot be unconstrained. The upload bandwidth and streaming rate determines the fanout. However, the feeder node of fanout cannot support a faster outflow rate than its inflow rate. A relationship should exist between the in-degree and out-degree of a node for receiving the data packets continuously. We would continue to abuse the terms ”parent” and ”children” of a node while referring respectively to a neighbor on an inflow path and a neighbor on an outflow path.

During a streaming session, all the nodes except for the ones connected directly to the source must maintain the relationship, ’in-degree out-degree.’ It does not preclude the relationship ’in-degree out-degree.’ So, a node may have a higher number of inflows than the number of outflows. It essentially implies that a node has more parents than the number of children. However, we discard such a case, because if a recipient gets more packets than it could send then eventually it will run into a buffer overflow problem leading to loss of multiple packets and re-transmissions.

We divide the media contents for streaming into chunks. These chunks are propagated to the peers where they could assemble the packets to a media file. The presenter is at the source node which streams the media contents to its neighboring peers. Being a multi-parent, multi-child architecture, a peer could ask for chunks from its parent peers and deliver these to its children. As stated earlier, whenever a source node joins the system, it authenticates itself from the bootstrap server. After authentication, the presenter could start streaming data. However, it will not send data to any of its children unless explicitly asked. Whenever a new peer joins the system, it will get a list of nodes from the bootstrap server. As soon as the node gets attached to its first parent, it will ask the parent about the latest (current) streaming packet id. Assume that a peer have parents , . Let receive its latest packet information from parent . Let the chunks ids are assigned in increasing or , , and so on. Further, let the latest packet be . Hence, in round 1, peer will request for from , from , from , and so on, from . Suppose, gets initial reply from , then from , then from , etc. So, peer will ask for from , from , from and so on. In other words, a peer will ask for the next id from the parent which replies first. It will create a stack of ids at the parent side. This process works parallelly. The above procedure is depicted below in a series of diagrams in Fig. 1-4. Figure 1 shows a partial view of a random mesh structure during execution of the algorithm. Consider the node and its parents only.

Fig. 1: Partial view of mesh overlay.

The initial requests for the latest packets are made to the parent in order from as depicted in Fig. 2. There is no need for a separate scheduler process. Each peer issues a request for the latest packet in a way akin to a reservation system.

Fig. 2: Initial request.

Now, suppose, gets packet id0 from as indicated by Fig. 3.

Fig. 3: Request reply sequence for latest packet.

Figure 4 depicts the process of issuing requests for pulling out subsequent packets after the parent peers reply to the previous requests placed by .

Fig. 4: Request for subsequent packets.

Whenver receives a packet from any of its parents; it updates the id for the next request. The process of updating id is performed in a mutually exclusive manner. It could be observed that the request for id is sent to a parent even when the id is not available. It is like making a reservation for a packet of a particular id. With this approach, there is no requirement for having a separate scheduling technique.

Iii-a Peer Joining

Initially, the root starts an active session of the mesh overlay. The root is synonymous to the presenter or the teacher. It is responsible for generating the streaming content. An ordinary peer has to authenticate itself before being is allowed to join the overlay. The session key serves as a means of authentication. A peer could join a session only if the root has already started the session. The bootstrap server doubles up as an authentication server in our implementation. Each peer (other than initiator) is authenticated from the session key it submits. The bootstrap server adds the requesting peer to the active peer list after successful. The active peer list consists of all the active peers in the system having spare capacity. The bootstrap server then returns a subset of active peers to the requesting peer. Let the total number of active peers be denoted by . A subset of nodes selected randomly from the active peers which joined the system before the newly admitted peer. The reason for choosing this parameter is to decrease the distance between the source and the newly joined peer. It results in improved startup time. Once the list is returned by the bootstrap server, the peer send ’Adopt me’ request to all the active peers in the list. A peer could accept or discard the request depending on its fanout value. Fanout of a node is its out-degree. Fanout of a node is determined by:

Hence, the fanout value of each peer is a dynamic value. Initially, we maintain the condition that in-degree = out-degree except for the nodes which are directly connected to the source node. Hence, if the number of parents (in-degree) of any nodes becomes less than its out-degree, then it requests the bootstrap server to send (in-degree - current_parent_count) number of active peers. The bootstrap server randomly selects a subset of active peers from the entire active peer list and returns it to the requesting node.

Iii-B Peer Leaving

When a node quits gracefully, it proactively informs the other children and the bootstrap server about its exit. However, involuntarily exit of a node is a bit difficult to handle. If the exiting node is not serving any other node at the point of leaving, then there is no disruption in the network. So, let us consider the leaving of a node which serves other nodes. At any point of time, a peer has to remember only the id which it has requested to its parents. Hence, if a parent fails abruptly, it will result in the loss of just one chunk. Assume that it takes units of time for transferring a chunk between a sender and a receiver. If there is no response for the time, then the peer compares the ’requested id’ with the id of the last packet it has received. If the requested id is less than the last packet id it has received, then requester assumes that the server node is either dead or does not have the packet. So, it creates another request id for the same packet to a different peer. It implies that unavailability of one chunk may lead to a delay of = time units. The missing chunk is received from one of the other parent nodes.

If a peer experience a delay of units for a response from any of the parent , then reports to the bootstrap server. The bootstrap server then initiates ’are you alive?’ message to . If there is no response for a time exceeding , then is assumed to be dead. The bootstrap server removes the non-responding node from the list of active peers list and informs the child node about the loss of its parent so that it could update its parent list. However, no parent list exists. A parent store the list its children. Hence, if a node exits involuntarily, then none of its children gets any information about the parent’s exit. Every child node has to figure out on its own about the failure of its parent.

Suppose a node is left with only one parent who also fails abruptly. The orphaned peer must then request the bootstrap server to assign a new set of parents. It might lead to a bit of delay. We used a scheme of proactive allocation of parents to reduce the delay for allocation of parents to the orphaned nodes. Once, a child notices that it has less than parents it requests the bootstrap server to allocate at least one extra parent. Seeking one additional parent is to restrict the effect a flash exit which occurs when a presentation (lecture) is about to end. Many peers start leaving the system at once. The count of the parents for each existing peer falls below at a faster rate. Therefore, at that point, it might not be possible to satisfy the requirement of parents. In the worst case, only a source node and a single peer may be available, and all other nodes may have departed. In this case, the source could be the only parent of the remaining peer.

Iii-C Handling ”Ask Doubt” Feature

Every peer is aware of the source node’s (presenter’s device) address. Whenever a participant of the audience wants to initiate a query, he/she clicks on ’ask doubt’ button provided in the UI of our application. It establishes a direct connection with the presenter’s device. The presenter gets a notification of the query and may send an acknowledgment. The peer device is allowed to send the query after receiving the acknowledgment. The doubt or the query should be in the form of an audio message like it happens during a physical presentation. The message is unicast on the link between the source and the peer initiated the query. The child peers of the requester would then pull the data while the source pushes the data to its other children. Thus, the query is propagated in a push-based manner as happens in the tree-based approach. It guarantees that no query is resolved by the speaker before it is asked, thereby preserving the causality relationship.

Iii-D Memory Utilization

A video packet consists of three key fields: Video data, the start of data, the length of the data. The system maintains a dictionary called PList for storing packets. It stores the data of recent packets. Each packet size is determined by an MTU of 1400 bytes. The streaming rate is assumed to be 2 Mbps.

  • The number of packets generated per second = = 179 packets.

  • The maximum amount of video data that can reside in PList is equal to . When size(PList), then for inserting a new packet in the data list, the oldest packet video data is removed from the PList.

  • The video data stored in PList could run a session for seconds.

  • Older data may be needed only if session exceeds this limit. Note that only video data is removed, start and length fields remains unchanged.

Consider the case where session is running for 8 continuous hours. The number of packets generated in 8 hours = packets. The total memory consumed by PList in 8 hours = .

Iv Live Shareable Whiteboard

A video packet consists of three key fields: Video data, the start of data, the length of the data. The system maintains a dictionary called PList for storing packets. It stores the data of recent packets. Each packet uses an MTU of 1400B. We have used a streaming rate of 2 Mbps.

  • The number of packets generated per second = = 179 packets.

  • The maximum amount of video data that can reside in PList is equal to . When size(PList), then for inserting a new packet in the data list, the oldest packet video data is removed from the PList.

  • The video data stored in PList could run a session for seconds.

  • Older data may be needed only if session exceeds this limit. Note that only video data is removed, start and length fields remains unchanged.

Consider the case where session is running for 8 continuous hours. The number of packets generated in 8 hours = packets. The total memory consumed by PList in 8 hours = .

Iv-a Design and Implementation

The live media streaming uses the pull-based approach to get the data stream packets from the parent peer. However, live shareable whiteboard follows a push-based approach for data propagation. There is no parent-child relationship between the nodes. All adjacent nodes are neighbors.

(a) Packet ID.
(b) Packet structure.
Fig. 5: White board packet.

A whiteboard packet consists of a packet-id, a label-id, the packet length and the data. The packet-id is composed of the client-id and the sequence number. The structure of a live board packet is shown in Fig. 5.

The packet-id field uniquely identifies a packet. The client-id is the first 8B of the SHA-1 hash of the IP address of a user. The client-id ensures that a packet is not delivered to the same peer from whom it is received. Multiple pages are supported in the shareable board. Each page is identified with a unique label-id. The operations performed on every page are stored separately. The label-id makes the canvas repainting task easier on the receiver side. It also allows maintaining a consistent view of both the sender and the receiver. Due to push-based approach, there may be data redundancy at the receiver side. So, the packets which are already seen are marked and discarded. Propagating text data is not an intensive operation. Therefore, even with the push-based approach, it does not contribute much to control overhead.

Iv-B Repainting Live Shareable Whiteboard

The user interface of the live board application provides many operations like free-pen, basic shapes, color, etc. Among these, free-pen is the most time consuming A single free-pen operation generates a lot of events. Sending one event over the network consumes a lot of bandwidth and degrades the system performance under the presence of moderate to heavy network traffic. To improve the performance, we used a buffered approach. A bunch of events is stored in a buffer. The entire buffer contents are sent if either the buffer is full or a mouse release event occurs. The operations are replayed at the receiver’s side. It is guaranteed that the events are repainted in a sequential manner only. The size of the buffer could be decided based on the packet size or MTU.

Iv-C Consistency of Board Views at Peers

Maintaining a consistent view of all peers is the most important thing. Our system maintains a separate list for every label-id. Whenever a new packet arrives, its label-id is extracted, and the corresponding packet is placed in that label-id list at the appropriate position according to its sequence number. The algorithms for packet generation, sender and receiver processes are provide in Algorithm 1, 2, and 3 respectively. A push-based approach causes packet losses and network delays. It may lead to an inconsistent view of the board among the peers if the replay at a receiver happens without handling of delayed packets. For handling delayed packets, at first each packet is placed at its correct relative position among other packets and later the repainting is done using the label-id and the sequence number carried in the packet. It is not a compute-intensive operation. The repainting is also very quick, as most computers can perform more than operations per sec. Furthermore, repainting performed only for one single page of the canvas at a time, not for the entire canvas. Therefore, repainting is quite fast.

while BufferNotFull OR MouseReleaseEventNotHappened do
       Create data packet; SendQueue.put(packet);
end while
Algorithm 1 Packet generation algorithm.
packet = SendQueue.get();
for  nodes in neighbor do
       Send to all except from whom it is received;
end for
Algorithm 2 Sender’s algorithm.
receive data packet;
extract packet-id;
if seen[packet-id]==True then
end if
       extract client-id, sequence no., label-id, packet length and data;
       if  Label[label-id].list.last-sequence no. sequence_no then
             last-sequence_no = sequence_no;
       end if
             insert (sequence no., data) at appropriate position in Label[label-id].list;
             repaint page with label-id from that sequence_number;
       end if
end if
Algorithm 3 Receiver’s algorithm.

A peer joining late may get an inconsistent view of the shareable board. We used a combination of push-pull to solving this problem. Initially, a comparison is done with label-id and if a conflict arises then another comparison is performed with the sequence numbers. If the current label-id or the sequence number is greater than zero, then it is safe to assume that data propagation has already started. Hence, the late joining peer explicitly requests (pulls) for the previous data from one of the neighbors. The number of operations performed on each page is stored with its corresponding page number of the canvas. A page may perhaps be left blank intentionally. In other words, the number of operations performed on a page may be zero or a positive number. In a pull request, a user first requests for the data of the page in the current view. It can be extracted with the help of label-id. Therefore, a pull request must include label-id — the sender replies with the number of operations performed on that page. The receiver side then verifies the reply. If the number of operations performed on that page is zero, then, no further request is made for that label-id. Otherwise, a request for remaining operations would be made. These operations are requested one at a time in a pull-based manner until the latest sequence number is reached. When the late joiner receives all the packets only then his/her session gets activated.

An explicit deferred joining process is necessary to avoid the inconsistent intervention of a late joiner. To understand why it happens, consider a user who joins after half an hour of the start of a session. Suppose the user immediately starts some operations on the first page of the canvas when the user’s device is still in the process of receiving data from other peers. Such an intervention by the user leads to an overwriting of the previous data. Therefore, we defer the activation time of the late joining peer until the data synching is complete. Another solution could be to enable the group-undo feature in the system, i.e., every undo command gets propagated to the entire group, and required changes will be done on every peer’s canvas. Even if the user joins the system before the sync is complete and starts scribbling, we could use a group-undo command to undo those operations. The inclusion of the group-undo feature is not available in the current work.

V De Bruijn Overlay for P2P File Sharing and Searching

The proposed P2P file sharing and searching system incorporates many additional features. The most important among these features is annotations of audio, video and PDF files. Therefore, it is a concomitant component to P2P-PS front-end platform. It helps in reflective learning and creating mash-up presentation using stored repositories. The file sharing system is implemented using de Bruijn graph overlays.

V-a De Bruijn Graphs

A de Bruijn graph is a labeled directed multigraph with fixed out-degree . Every node of the graph has an or a label of fixed length. Let the length of each be and alphabet of size be . De Bruijn graph of nodes can be constructed as follows. Each node is connected by an outgoing edge to other nodes. There is an outgoing edge from a node to a node if can be created by applying a left shift to the label and appending one symbol from alphabet of size to the rightmost position, i.e., where . If labels are considered as base numbers, then outgoing edges will be to all the nodes with label equal to where . Figure 6 illustrates an example of a de Bruijn graph .

Fig. 6: An example of de Bruin graph

Routing in a de Bruijn graph is specified in the form of a string. Let and denote the source and the destination nodes with labels and respectively. Then the routing path from to is obtained by looking at the maximum overlap between suffix of and prefix of , and calculated as appended by the non-overlapped part of . For concreteness, assume that a look-up for destination node 1011 is initiated by a source node with . The maximum overlap of the suffix of 1110 and prefix of 1011 is 10 as shown in Fig 7. Hence, the routing path would be specified by string 111011, i.e., 1110 appended by the non-overlapped part 11 of .

Fig. 7: An example of routing in de Bruin graph

The next hop is derived from a current hop as follows:

  1. is a neighbor of in the input de Bruijn graph.

  2. The length of longest suffix of that is same as prefix of is 1 more than the length of the longest suffix of that is equal to a prefix of .

This routing scheme is known as substring routing.

The structure and the routing method indicate that the diameter of de Bruijn graph is . De Bruijn graphs have low clustering and exhibit -node-connectivity [16]. -node-connectivity is not possible due to self-loops on nodes with IDs of the form for to . The nodes can be linked together to form a ring which makes the graph -regular and also achieves -node-connectivity [17]. -node-connectivity makes the graph more resilient to fault-tolerance, as even the failure of any nodes cannot disconnect the graph and diameter remains at most . Loguinov et al [16] showed that the expected congestion in de Bruijn graph is much less than the other counterparts under a similar rate of load, due to larger bisection width of the graph.

V-B Comparison with Other Graphs

De Bruijn graph possesses better asymptotic degree-diameter properties, compared to some of the widely used DHTs Chord [10], Trie [18], CAN [19], Pastry [20] and Butterfly [21]. Tables I and II provide a summary of comparisons in terms of degree and diameters from the analysis made by Loguinov et al. [16]. The blank cells in Table II indicate that corresponding node degrees are not supported.

Graph Degree Diameter D
de Bruijn
TABLE I: Asymptotic degree-diameter properties.
k de Bruijn Trie Chord CAN Pastry Butterfly

20 huge 31
3 13 40 20
4 10 26 1,000 16
10 6 13 40 10
20 5 10 20 20 20 8
50 4 8 7 7
100 3 6 5 5
TABLE II: Graph diameter for nodes
K Moore Graph de Bruijn

17.9 18.3
3 11.7 11.9
4 9.4 9.5
10 5.8 5.9
20 4.5 4.6
50 3.5 3.5
100 2.98 2.98
TABLE III: Average distance between pair of nodes for

Table III compares the average distance between the nodes in de Bruijn graph to the optimal Moore graph [16] with the same degree . In a de Bruijn graph it remains very close to optimal values even for the smaller values of .

Vi Overlay Network Based on De Bruijn Graphs

Loguinov et al. [16] proposed guidelines for incremental construction of a de Bruijn graph. However, the paper falls short of implementation as it does not address the problem of maintaining de Bruijn structure in presence of churning.

For implementation, we choose the parameters , and for a de Bruijn graph. It allows us around 16 million nodes inside network labeled ”00000000-77777777” in octal strings. However, the diameter of a de Bruijn graph remains O(1) (to be precise 8). The DHT overlay in our system would support efficient look-ups, and an epidemic dissemination-based protocols can be designed to infect all the nodes in just a few rounds.

Vi-a Information Structure at Each Node

We refer to the nodes in the underlying de Bruijn graph as virtual nodes. For maintaining the underlying graph, a physical node in our system is responsible for a range of virtual nodes with consecutive s. A range of a physical node in our system is referred to as its zone. Initially, when a single physical node joins the DHT, it is responsible for all virtual nodes from to . The virtual ID-space may be visualized as a ring, where each physical node is responsible for only a arc of the ring. Figure 8 illustrates an example of three physical nodes responsible for three different zones.

Fig. 8: Three physical nodes in a de Bruijn based ring with Zone ownership.

The structure is similar to Chords [10]. However, unlike the Chord overlay a physical node responsible for an ID space arc may be located somewhere randomly within it. A node has an outgoing edge to another node , if there is at least one virtual node in ’s zone having an outgoing edge to a virtual node in ’s zone. Each node keeps a list of its outgoing and incoming edges. Each node in these lists knows its address and zone ID. The details of the structure maintained at each node is given in Table IV.

Item Information
ID ID of node is a label lying in the Zone
Zone A range of virtual IDs for which node is responsible
External Address Address on which others should contact
Outgoing Edges List of nodes connected by outgoing edge, along with info about their Zone and Address

Incoming Edges
List of nodes connected by an incoming edge, along with info about their Zone and Address
TABLE IV: Structure maintained at each node

Vi-B Joining of Nodes

Suppose tries to join. It selects a random ID and forwards a join request to identify the owner of the zone in which the chosen random ID falls. For convenience in the description we use the following convention:

  • Any intermediate node receiving the join request initiated by is referred to as

  • The node whose zone contains the random ID chosen by , is node .

The problem of joining is split into three parts according to actions of , and as explained below.

Vi-B1 Actions of

Node requests the rendezvous server (whose public IP address is known to all) to provide the external address (IP address) of and those in a list of random peers. picks one peer randomly from the list supplied by rendezvous server and sends a join request to that peer. The request contains the following:

  1. ’s external address

  2. random virtual ID from the entire ID-space i.e. to .

The join request initiated by tries to identify , the owner of the random ID sent in the join request. When node sends such a request for the first time, its ID is chosen using SHA-1 hash value of its external address. But, upon retries, a random ID from the entire region is picked.

Vi-B2 Actions of

on receiving the join request forwards it to the next node on the routing path. If a routing path is not provided in the request, will create one using ’s ID and destination ID given in the join request. The correct path will be sent along with the join request.

Vi-B3 Actions of

On receiving the join request from (possibly through intermediate nodes) sends the information regarding its zone ID, the incoming and the outgoing links. does not accept further Join requests until joining of is complete, or the timeout of 10 seconds occurs.

The reply from contains a structure maintained at . It includes:

  • Virtual node label of

  • External address of

  • Outgoing and incoming edges of

waits for to complete joining process send information about ’s new zone. then updates its own zone as follows.

  • Changes its zone

  • Sends keys with values to be managed by

  • Notifies the neighbors about the change in zone

  • Drops the edges destroyed due to the shrinking of its zone

Vi-B4 Action of After Receiving the Reply

picks the half part of the zone not containing ’s ID and chooses a random ID (label) from the picked zone as own ID. sends information of its ID and its zone ID to , and the attachment of node in DHT overlay is complete. needs to perform the following actions to complete the joining process:

  • Disseminates its ID and zone information to all the links shared by

  • Identifies the links to be dropped and checks if there is any new link to . Then makes the corresponding changes to the incoming and the outgoing edges list.

willingly accepts load shared by . For the interest of space the algorithm has not been included here. The interested readers can review the algorithm appearing in appendices A and B.

Vi-B5 Join Example

For keeping the examples small, we show the Node Join process in de Bruijn graph with and . Figures 9(a) and 9(b) illustrate joining process of the first two nodes in the system.

(a) First node.
(b) First node.
Fig. 9: Joining of the first pair of nodes in the system.

The process of joining of the third node is shown in Fig. 10(a). Notice that the joining of the third node requires removal of link (dashed red line). For the joining of the fourth node two new links are to be inserted as shown by green lines in Fig. 10(b).

(a) link gets removed.
(b) & links get added.
Fig. 10: Joining of the next pair of nodes.

Vi-C Leaving of Nodes

Consider the leaving of a node from the system. A node is identified who can merge the zone of into its own Zone. There are two part to the leave process, as leaving node is aware of its both the predecessor and the successor in the DHT overlay.

Vi-C1 Actions of

identifies a successor zone and a predecessor zone. The first virtual ID belonging to the successor zone equal to ’s Zone.endID + 1. Similarly, the last virtual ID belonging to the predecessor Zone is equal to ’s Zone.startID - 1. picks one of the two IDs, finds the respective owners of the zones and sends a request to leave. The leave request contains the information regarding ID, Zone, incoming and outgoing links to the chosen neighbor, say . If agrees within timeout period of 5 seconds, sends the load to and the leave process is complete. Otherwise, tries the same with the other neighboring node. If the timeout occurs again, the entire procedure is repeated assuming chosen nodes were busy in other Leave procedure.

Vi-C2 Action of

Node receiving the request checks if it is going to leave the system. If not, it accepts the request and agrees to take over the load by merging the two zones. The merging process includes merging of the incoming and outgoing links as well. then notifies all the linked nodes regarding the update. Algorithm 7 in Appendix C gives a step-wise description of ’s operations.

Vi-D Zone Update

During join and leave, nodes whose zone changes notify the nodes linked to them by outgoing or incoming edges. When such notification is received, a node adds or drops links caused by the change.

Vi-E Keep-alive and Failure detection

Every two minutes, each node sends keep-alive message to all linked nodes. These nodes update the timestamp with the zone information corresponding to that node. If no update is received in last five minutes for a linked node, it is considered dead, and owner of the successor zone is notified to take the responsibility of orphaned zone.

Vi-F Links between Zones

If there is an outgoing edge from a virtual node in one zone to a virtual node in another zone, then there exists an outgoing edge between one node to the other. On many occasions, zones of the neighboring nodes change. A brute force way to find an edge in the underlying graph, would slow down join and leave operations.

Node receives zone update from another node . checks if the size of its zone is larger than or equal to . If so, then a link exists (because every possible suffix of length is present in ’s zone). Otherwise, it determines the range of the suffix of length lying in the and range of the prefix of length lying in ’s zone. If , or then there exists an edge from node to node . Similarly, there is an edge exists if or lies within .

Vi-F1 Examples

  • Consider two zones: node ’s zone [’00000000’, ’17777777’] and node ’s zone [’40000000’, ’77777777’]. Since the size of zone of node is larger than ’10000000’, it will have all possible suffix of length . Therefore, it will have link to any node in the system. Same is the case for and hence, both have outgoing edges to each other.

  • Consider another two zones: node ’s zone [’00000000’, ’00777777’] and node ’s zone [’01000000’, ’01777777’]. The range of suffixes of length , for node is [’0000000’, ’0777777’]. The range of prefixes of length , for node is [’0100000’, ’0177777’]. There is a non-empty intersection between the two ranges, and hence an edge exists from node to node . (In the underlying graph, ’00100000’ has an outgoing edge to ’01000000’.)

  • For the above example, range of suffixes of length , for is [’1000000’, ’1777777’]. The range of prefixes of length , for is [’0000000’, ’0077777’]. The intersection between two ranges is empty, and hence no edge can exist from to .

Vii Sharing, Searching, and Global Indexing

The implementation supports two types of searches (i) by ID/key, and (ii) by keywords. A user can share one or more directories which contain the file(s). By default, at least one directory is shared which may be empty. The said directory is located in the Downloads folder of the user’s system. Anything downloaded from the P2P system is stored in this directory and is automatically shared.

Vii-a Hashing and Key Generation

We have used a 160-bit SHA-1 value for the contents of the file and also use it as a key. Since, each node has 24-bit label, the first 24 bits of a key are used for the routing purpose. For example, first 24 bits of the SHA-1 hash value ”2fd4e1c67a2d28fced849ee1bb76e7391b93eb12” is ”2fd4e1” which is equal to ”13752341” in octal.

Vii-B Pre-processing of Files

At the start of the application and after every 30 minutes, the directories are scanned for any new or modified files. The new and modified files are queued for processing. SHA1 of a file is computed, and it is stored in the local database. Any deleted or renamed file is also identified. For deleted files, the entries related to them are deleted from the local database. A set of important keywords are associated with file to improve the chances of finding a document or file in the DHT. We provide support for extracting text from PDF and video files that come along with the subtitles. If a PDF is created using scanned copies, the text is extracted using an OCR tool. Once the text is extracted, TF-IDF scheme is used to identify top 100 keywords for a file using its content. In this scheme, the file is considered to be a set of documents of 1000 words each. In the TF-IDF scheme, a term is assigned a higher score if it has larger term frequency other terms, but appears in a less number of the documents. The scoring scheme is, therefore, based on Inverse Document Frequency. These keywords are stored in the local database along with file’s modification times, which can be used to avoid processing of the file again. For all keywords, SHA-1 is also computed to obtain the keys. The users can also manually add up to 10 keywords.

Vii-C Global Indexing of Files

A Key is a globally unique ID of an object (e.g., SHA-1 value), and the Value (the name of the object), the size of the object, the address of the object and the timestamp of the last refresh. The keywords are stored as meta data and maintained in the local database at the external nodes of an overlay. The purpose is to enable keyword based search for related document and media files. We need two basic operations, namely, Put and Get.

For each file, MultiKeyPut is used to insert key-value pairs of all keywords in the system. The keywords also include words in the file name. The associated value stores information such as SHA-1 hash of file, size of the file, name of the file and address of the file. After every 30 minutes, MultiKeyPut is called for keywords of each file, thus refreshing the timestamp for keys in corresponding nodes. A node will check for the keys’ timestamp every 10 minutes, if a particular node has not updated a key in last 60 minutes, corresponding key-value pair is considered to be not available and dropped from the local store. The delay of 60 minutes is kept considering a lost update in the network.

Vii-D File Searching

Searching operation is performed one at a time. A user can cancel the search at anytime. Since, there can be delayed replies, a unique search ID is associated with the Get requests to distinguish the results. Any node returning result/s must provide search ID along with the result. The packets received for the current search ID are kept, and all other are discarded.

Vii-D1 Search by keywords

A query is broken down into keywords, and search is conducted with MultiKeyGet procedure. For any reply that comes while search is active, the ranking of the results may change. The file with more number of query keywords in it, gets higher rank. The files with same number of query keywords is distinguished by the replication/popularity in the system. The more popular result gets a higher ranking than the others.

Vii-D2 Search by file ID

A user might receive the ID or the key of a file from another user via some communication channel. If user wishes to download file with given ID, he/she can enter it in Get File dialog which user Get method. On receiving the results, a download request is initiated.

Vii-D3 File Download

When results are fetched using the above search methods, a user can choose to download a file. If chosen file is available with multiple peers, multi-threaded approach for downloading is used. Different chunks are requested from multiple peers, and they are written to the file as they are downloaded.

Viii Annotations

Most PDF viewers have tools to highlight a portion of the text. It is useful for the revision of the text or skimming through the text. But, any editing of the content changes hash value of a file. Modification of hash value of a file is undesirable, as the file in P2P learning environment is archival in nature. Therefore, the features should be provided without editing the file itself. Furthermore, the users may also want to see portions that are highlighted by others. Some other users may wish to tag some text or provide a hyperlink to some other external resource. All these operations are grouped as peer learning activities without any change in the contents. We defined a format for annotations for the purpose. The annotations are stored separately in the local database, and synchronized among peers.

Viii-a Annotation Format

We propose a general annotation format. This general format is a template for specific formats. The following information is stored in an annotation:

  • Annotation ID: A unique ID to identify an annotation.

  • File ID: A unique ID for file. SHA-1 value is used.

  • Time stamp: Time of creation of the annotation.

  • Author: Identity of the creator.

  • Text: Associated text, if any.

  • Properties: Specific properties of the annotation.

The ”Properties” field stores the properties for specific types of annotations. The text field may be in the form of a hypertext, as the support for links and images is provided.

Viii-B PDF Annotations

Highlighting in the PDF documents can be done by either selecting text or selecting a rectangle. We define properties of each type separately.

Viii-B1 Text Selection

Text selection provides the flexibility of selection of a specific word or a set of words, or a set of lines in a particular page. If selection spans multiple pages, the user should select words or lines on each page separately. The user may also provide a comment or add some text related to a highlighted area. The additional text/comments are stored with the annotation. Since each annotation of this type is designed for only one continuous region of a text, we define following properties for an annotation.

  • PageNumber: Denotes the page number on which annotation is present.

  • FirstWord: Denotes the index of the first selected word

  • LastWord: Denotes the index of the last selected word

Viii-B2 Rectangular Selection

Rectangular region selections are generally helpful in inserting annotation related to diagrams, or non-textual information. A user can choose one rectangular region of a page and highlight the portion. Following properties define each rectangular region:

  • PageNumber: Denotes a page number on which the annotation is present.

  • TopLeft: Denotes co-ordinates of the top-left corner of the selected rectangle with respect to the the top-left corner of a page

  • BottomRight: Denotes the co-ordinates of the bottom-right corner of a selected rectangle with respect to the top-left corner of a page.

Viii-C Audio/Video Annotations

Video annotations are similar to Audio annotations. Therefore, we do not distinguish between the two. A user can select a duration of the video and tag it. The two end-points of a duration are rounded to the nearest integers and at least 5 seconds of the duration is selected. The properties for such a selection are as follow:

  • StartTime: The time at the start of duration.

  • EndTime: The time at the end of duration.

Ix Discussion Forum and Announcements

As discussed in the previous section, our system provides a way to create an annotation and save it. A user can use them for personal use. He/she can also share them with others to spread useful information or ask a doubt regarding the material. An annotation is disseminated to other peers and can be discussed among the participating peers. So, we integrated a discussion forum along with annotation announcement.

When a user creates an annotation, he/she has an option to share it with others. If the user chooses to share it, the annotation is converted into a post and gets disseminated among the peers. Other peers can put comments on the shared post.

Ix-a Post

A post inherits the format of the corresponding annotation, adds an additional field for Title. A post also has a constraint of minimum number of 100 characters in the Text field. The maximum number of characters is limited to 1600.

  • ID of Post: A unique ID to identify a post

  • File ID: A unique ID for file. SHA-1 value is used

  • Time stamp: The time of creation of annotation

  • Author: The identity of the creator

  • Text: Associated text

  • Title: A short description of the post

  • Properties: Specific properties of the PDF or Audio/Video annotations are stored.

When an annotation is converted into a post, Text field of Annotation is copied into the Text field of the Post. The properties field of Post supports multiple annotations. Multiple annotations are stored in a list, serialized into a string and then stored in Properties field. If multiple annotations are to be attached to a post, File ID is set to zero.

Ix-B Comment

The comments on a post follow similar format as the post. But, instead of Title field, it has ReplyTo field.

  • Comment ID : A unique ID to identify a comment

  • File ID: A unique ID for file, SHA-1 value is used

  • Time stamp: The time of creation of the annotation

  • Author: The identity of the creator

  • Text: Associated text

  • ReplyTo: ID of post or comment to which this is a reply

  • Properties: Specific properties of the PDF or Audio/Video annotations are stored.

Ix-C Synchronization of Posts and Comments

Posts and comments are synchronized with the help of two protocols.

  • An epidemic dissemination based protocol is used to spread recently published posts and comments.

  • Since, a user won’t be active at all times there will be missed updates. For such cases, reconciliation (an anti-entropy session) has to be performed at regular intervals with the neighbors.

The consistency requirements of the discussion platform is weaker because a post once posted, cannot be updated. For a particular post or a comment, only one person is responsible. Posts and comments need not be delivered immediately. The happened-before relation of the comments and the posts is maintained due to their hierarchical structure. The causal ordering of other messages can only be provided through time-stamps associated with them. We assume that clocks of the machines are synchronized with NTP servers.

Ix-C1 Epidemic dissemination

Every node receives posts and comments through their neighbors. Our application periodically checks if any new post or comment arrived since the last check. If there are new posts or comments, they are sent to all the neighbors except the ones from which they were received. This ensures the delivery of messages to all. The period of such check is set to one minute. The diameter of our system is , therefore, any new message gets delivered to the entire system in less than ten minutes. The interval time can be set to zero, and in that case, messages will be delivered to others instantly.

Ix-C2 Reconciliation

Reconciliation procedure executes at regular intervals of thirty minutes. It begins by randomly picking a neighbor at the start of an interval. A predefined time of seven days is chosen. When posts from last seven days are sorted by time, the first post’s timestamp is picked, and shared with the neighbor. On receiving the reconciliation request. The neighbor chooses all the posts that started on or after the given timestamp, and sorts them according to timestamps. A list of IDs of these posts or comments are created. The list is divided into chunks of 256 entries. A hash value is calculated for each chunk, based on the post/comment IDs. These hash values are shared with the initiator. The initiator also calculates the hash values in the similar fashion, and the hash values are compared. If the hash value is different, then the initiator asks a neighbor to share the IDs in the particular chunk. It then gets the list of IDs in the chunk from the neighbor. Both of them make changes to the corresponding lists by adding missing IDs and recalculate hash values for that chunk. New list of hash values from the mismatched chunk is sent to the initiator. When the last chunk’s hash matches reconciliation is over. Most of the time, only last few chunks would differ, and the number of message exchanges is low.

Ix-D Announcements

Announcements are similar to posts in the discussion forum. They are useful for posting information about locations of newly added documents. The sharing of announcements is carried out in a similar fashion. They are forwarded to other peers as soon as they are received. On the front-end, announcements appear separately from other posts.

X Results of Simulation

Emulab provides an environment for experiments over test beds consisting of large-scale distributed network. We deployed scripts for the experiments using Emulab portal to acquire both physically distributed and purely simulated nodes [emulab2008]. It fitted the kind of experiments we wanted to run for establishing efficacy of our approach to build a low latency file sharing, searching and inline annotations framework ideally suitable for E-learning system using a P2P organization. With respect to whiteboard sharing our primary motivation was to find out how a P2P learning system may work in a LAN environment and support up to a maximum of 250 nodes. However, the simulation experiments were carried out on up to 1000 nodes.

X-a Live Streaming and Whiteboard

The first experiment is to find out the stability of our system. During the process of joining as explained in section III-A, a peer gets a list of active peers. Running simulations on up to nodes, we found that the choice of leads to a stabilization of network as the size of the overlay increases. Once the bootstrap server returns the list, the peer send ’Adopt me’ request to all the active peers in the list. A peer receiving join request could accept or discard the request based on its fanout value. With simulations on up to nodes, we found that the choice of leads to a stabilization of network as the size of the overlay increases. We found that maximum path increases with an increase in the number of nodes, but later it gets stabilized. As shown in figure 11(a) from 700 to 1000 nodes, the maximum path length has stabilized to 6 units.

(a) Maximum path length.
(b) Churning with 10%, 20%, and 30% churn rates.
Fig. 11: The results on Emulab for stabilization of the maximum path length and churning.

Mesh-overlay might get disconnected in the presence of churn. A node is having in-degree implies that it has no parent and hence, it could not receive data until it finds at least one parent. So, even in the presence of churning it is important that the overlay should eventually get stabilized. It means only one node could have in-degree . This node should be the source node only. We performed on simulations on 1000 nodes for churn rate 10%, 20%, 30%. The results appear in figure 11(b). The number of nodes counts the nodes having in-degree equal to zero. The time is measured in seconds. We observed that even with 30% churn rate, the overlay stabilizes within 5 seconds.

Table V depicts the minimum throughput value analyzed on a different number of nodes. Here, throughput is defined as the number of packets received in one second.

Number of nodes Throughput
20 177
50 152
90 173
120 156
150 139
170 173
200 174
TABLE V: Minimum throughput value.

We have obtained these results by performing experiments on Emulab. In Emulab experiments, each node receives 5000 packets from its parent nodes. As discussed in section VI, the number of packets generated per second = 179. Hence, 5000 packets will be generated in seconds. Hence, we can see from Table V that our results are closely related to the theoretical results.

X-B De Bruijn Overlay

De Bruijn graph has diameter and out-degree equal to eight . At the application layer, the diameter does not exceed the value of eight. But, out-degree can vary according to size of a zone. In theory, the maximum out-degree would be less than with high probability, where is the number of nodes in the system. Considering nodes, the maximum out-degree among all runs was 41. Experimenting with different values of , the average out-degree was found to be 7.99. It is due to absence of self-loops.

Fig. 12: Maximum out-degree of nodes
Fig. 13: Maximum out-degree of nodes
Fig. 14: Distribution graph of out-degree of nodes

The graph in Fig 12 and Fig 13 show that the median values of the maximum out-degree of a node. The simulation results match with the theoretical results. The graph shows that the median value for the maximum out-degree is 31 when . The distribution of out-degree for the same is shown in Fig 14. It shows that very few nodes have degree . Just 380 out of 100000 nodes have degree , which is only of the total. None of the nodes have out-degree . In an equivalent Chord implementation, the average out-degree is which is 20. Our experiments also determined that, minimum in-degree=7, and the maximum in-degree=8.

We also carried out experiments on Emulab to find out the maximum latency and the success rate of look-ups. In the experiment, the nodes were connected to each other in a LAN environment. Every node randomly picked twenty-five words from the list of three thousand words available to everyone. After a delay of five minutes, the nodes sent out queries for the same set of words. The boot-up times of machines in Emulab differ up to minutes. Hence, the join procedure for de Bruijn overlay network would have involved a transfer of the load for most of the nodes. The results were calculated for the varying number of nodes as shown in Table VI. It indicates that none of the lookups failed, even during the dynamic joins in the set up.

No. of nodes Success rate Max Lat.(ms) Avg. Lat.(ms)
80 100% 50 16
120 100% 70 20
160 100% 58 21
200 100% 52 21
TABLE VI: Lookup latency and success rate

In another experiment, the nodes are allowed to leave the system with 10% probability after every three minutes. Our approach achieved success rate of 99.39% and the maximum latency of 52ms for 200 nodes. Assuming that the human tolerance limit is about 200ms, the response is pretty good

Xi Conclusion

We have studied two different architectures to support live streaming, and based on the analysis we modified mesh architecture to support dynamic fanout leveraging spare capacity whenever available. Hence, our system could also support heterogeneous nodes satisfying the minimal bandwidth requirement as explained in section VI. In our experience with actual sessions on live board, we found a consistent display of results. The repainting was quite efficient at all peer nodes. The buffered approach has helped to optimize the shareable whiteboard’s performance.

Our simulations on Emulab, proved that even in the presence of churning, the overlay structure for live streaming gets stabilized moderately quickly. Since the maximum path length also gets stabilized the latency is bearable in Live sessions of P2P interactions with live streaming. The Emulab experiments are important to establish that experimental throughput is close to theoretical throughput values.

As a part of future work, we plan to include more tools in our live shareable whiteboard and enable group-undo operation. Various video-coding techniques are available to neutralize the effect of packet loss. In the future, we would like to incorporate those techniques for live streaming. The back end support for file sharing, searching and inline annotation also needs to be integrated.


  • [1] Overleaf, “Overleaf training resources,” 2018, online; accessed 13th-August-2018. Available at
  • [2] Google, “Gsuite learning center,” 2018, online; accessed 13th-August-2018. Available at!/.
  • [3] Github Engineering, “Github guides,” 2018, online; accessed 13th-August-2018. Available at
  • [4] R. Vesselinov and J. Grego, “Duolingo effectiveness,” 2012, online; accessed on 13th-August-2018. Available at
  • [5] coursera, available at
  • [6] kahoot, available at
  • [7] Brainly, - For students. By students., available at
  • [8] S. A. Baset and H. Schulzrinne, “An analysis of the skype peer-to-peer internet telephony protocol,” arXiv preprint cs/0412017, 2004.
  • [9] S. Icard, “Educational technology best practices,” International Journal of instructional technology and distance learning, vol. 11, no. 3, pp. 37–41, 2014.
  • [10] I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan, “Chord: a scalable peer-to-peer lookup protocol for internet applications,” IEEE/ACM Transactions on Networking (TON), vol. 11, no. 1, pp. 17–32, 2003.
  • [11] H. Deshpande, M. Bawa, and H. Garcia-Molina, “Streaming live media over a peer-to-peer network,” Stanford InfoLab, Tech. Rep., 2001.
  • [12] X. Zhang, J. Liu, B. Li, and Y.-S. Yum, “Coolstreaming/donet: A data-driven overlay network for peer-to-peer live media streaming,” in INFOCOM 2005. 24th Annual Joint Conference of the IEEE Computer and Communications Societies. Proceedings IEEE, vol. 3.   IEEE, 2005, pp. 2102–2111.
  • [13] N. Magharei and R. Rejaie, “Prime: Peer-to-peer receiver-driven mesh-based streaming,” IEEE/ACM Transactions on Networking (TON), vol. 17, no. 4, pp. 1052–1065, 2009.
  • [14] D. Stutzbach, D. Zappala, and R. Rejaie, “The scalability of swarming peer-to-peer content delivery,” in International Conference on Research in Networking.   Springer, 2005, pp. 15–26.
  • [15] N. Magharei, R. Rejaie, and Y. Guo, “Mesh or multiple-tree: A comparative study of live p2p streaming approaches,” in INFOCOM 2007. 26th IEEE International Conference on Computer Communications. IEEE.   IEEE, 2007, pp. 1424–1432.
  • [16] D. Loguinov, A. Kumar, V. Rai, and S. Ganesh, “Graph-theoretic analysis of structured peer-to-peer systems: routing distances and fault resilience,” in Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications.   ACM, 2003, pp. 395–406.
  • [17] D.-z. Du, D. F. Hsu, H. Q. Ngo, and G. Peck, “On connectivity of consecutive-d digraphs,” Discrete Mathematics, vol. 257, no. 2-3, pp. 371–384, 2002.
  • [18] M. J. Freedman and R. Vingralek, “Efficient peer-to-peer lookup based on a distributed trie,” in International Workshop on Peer-to-Peer Systems.   Springer, 2002, pp. 66–75.
  • [19] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker, A scalable content-addressable network.   ACM, 2001, vol. 31, no. 4.
  • [20] A. Rowstron and P. Druschel, “Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems,” in IFIP/ACM International Conference on Distributed Systems Platforms and Open Distributed Processing.   Springer, 2001, pp. 329–350.
  • [21] D. Malkhi, M. Naor, and D. Ratajczak, “Viceroy: A scalable and dynamic emulation of the butterfly,” in Proceedings of the twenty-first annual symposium on Principles of distributed computing.   ACM, 2002, pp. 183–192.

Appendix A Send Join Request Algorithm

Require: Addresses of the rendezvous server & its socket;
send connect request to the server;
receive & store own extAddress;
receive ListOfPeers from the server;
randomID pick a number from (0, -1) based on SHA-1 of extAddress;
if (ListOfPeers == ) then
       MyZone entire range of virtual nodes;
       MyID randomID;
       MyOutEdges, MyInEdges ;
end if
       joined FALSE;
       while (!joined) do
             randomPeer select a random peer from ListOfPeers and delete it;
             send JoinRequest(randomID, own extAddress) to randomPeer;
             wait for up to 10 seconds for message from owner(randomID);
             if (timeout) then
                   if (ListOfPeers == ) then
                         receive new peer addresses from rendezvous server;
                         update ListOfPeers with newly received addresses;
                         randomID randomNumber(0, -1);
                   end if
                         if (Zone contains only one virtual ID) then
                               randomID randomNumber(0, -1);
                               continue and try another node;
                         end if
                               recv (ownerID, Zone, OutEdges and InEdges);
                               split the received Zone into two parts Zone.part1, Zone.part2;
                               if (ownerID Zone.part1) discard Zone.part1;
                               MyZone Zone.part2;
                               else Discard Zone.part2;
                               MyZone Zone.part1;
                               MyID randomID(MyZone);
                               MyOutEdges, MyInEdges received OutEdge, InEdges;
                               send notifyJoin(MyZone,MyID) to previous zone owner ;
                               send notifyJoin(MyZone,MyID,extAddress) to all neighbors;
                               check if any new edge exists between previous owner’s new zone and my zone;
                               drop edges (with any of the neighbors) that no longer exists;
                               // For adding to list of peers;
                               notify rendezvous server;
                         end if
                   end if
             end if
       end while
end if
Algorithm 4 SendJoinRequest

Appendix B Received Join Request Algorithm

Require: Join request packet JOIN(ID, JoinExtAddr);
if (ID MyZon) then
       send MyZone, MyID, MyOutEdge, MyInEdges to JoinExtAddr;
       queue new joins until this join completes or timeOut==10s;
       receive ID and Zone information of the new node;
       split MyZone into half and discard the new node’s Zone;
       identify and insert in appropriate lists any new edges with the newNode’s Zone and MyZone;
       notify all neighbors about update of Zone by sending MyZone and MyID;
       drop edges (with any of the neighbors) that no longer exists;
       share existing Key, Value pairs in lying new node’s Zone with the new node;
end if
       // Receiver is a forwarder;
       identify the next node on routing path to which request is to be forwarded;
       send JoinRequest(ID, JoinExtAddr) to next node;
end if
Algorithm 5 ReceivedJoinRequest

Appendix C Send Leave Request

Require: The address of the rendezvous server, socket;
left FALSE;
while (!left) do
       if (MyZone covers entire ID-space) then
             inform the server that all other nodes have left;
             left TRUE;
       end if
             list [MyZone.startID-1, MyZone.endID + 1];
             // succID is the ID of the successor;
             succID randomID(list);
             if succID lies with one of the neighbors then
                   next_node address of the neightbor containing the succID;
             end if
                   create a routing path from MyID to succID;
                   next_node address of the next node from the routing path;
             end if
            send LeaveRequest(ID, Zone, OutEdges, InEdges, extAddress) to next_node;
             wait for reply up to 5 seconds;
             if (Reply == ”Yes”) then
                   send confirmation and keys in MyZone to successor;
                   inform the rendezvous server, that leave is complete;
                   left TRUE;
             end if
       end if
end while
Algorithm 6 SendLeaveRequest

Appendix D Received Leave Request

Require: Leave request packet LEAVE(ID, Zone, OutEdges, InEdges, LeaveExtAddr);
// LeaveExtAddr is the external address of leaving node;
if (ID MyZone) then
       if (not leaving the system) then
             send ”Yes” to LeaveExtAddress;
             wait for confirmation up to 5 seconds;
             if (confirmed) then
                   merge Zone, OutEdges and InEdges;
                   receive the load;
             end if
       end if
             // Leaving self;
             send ”No” to LeaveExtAddress;
       end if
end if
       identify the next node on path to which the request should be forwarded;
       send LEAVE(ID, Zone, OutEdges, InEdges, LeaveExtAddr) to the next node;
end if
Algorithm 7 ReceivedLeaveRequest