Service Rate Region of Content Access from Erasure Coded Storage

We consider storage systems in which K files are stored over N nodes. A node may be systematic for a particular file in the sense that access to it gives access to the file. Alternatively, a node may be coded, meaning that it gives access to a particular file only when combined with other nodes (which may be coded or systematic). Requests for file f_k arrive at rate λ_k, and we are interested in the rate that can be served by a particular system. In this paper, we determine the set of request arrival rates for the a 3-file coded storage system. We also provide an algorithm to maximize the rate of requests served for file K given λ_1,..., λ_K-1 in a general K-file case.



page 1

page 2

page 3

page 4


Distributed Storage Allocations for Optimal Service Rates

Redundant storage maintains the performance of distributed systems under...

BPF for storage: an exokernel-inspired approach

The overhead of the kernel storage path accounts for half of the access ...

On Distributed Storage Allocations of Large Files for Maximum Service Rate

Allocation of (redundant) file chunks throughout a distributed storage s...

Implementation of an Android Framework for USB storage access without root rights

This bachelor thesis describes the implementation of an Android framewor...

Efficient Storage Schemes for Desired Service Rate Regions

A major concern in cloud/edge storage systems is serving a large number ...

Service Rate Region: A New Aspect of Coded Distributed System Design

Erasure coding has been recently employed as a powerful method to mitiga...

Latency optimal storage and scheduling of replicated fragments for memory-constrained servers

We consider the setting of distributed storage system where a single fil...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

The explosive growth in the amount of data stored in the cloud calls for new techniques to make cloud infrastructure fast, reliable, and efficient. Moreover, applications that access this data from the cloud are becoming increasingly interactive. Thus, in addition to providing reliability against node failures, service providers must be able to serve a large number of users simultaneously.

Content files are typically replicated at multiple nodes to cope with node failures. These replicas can also be used to serve a larger volume of users. To adapt to changes in popularity of content files, service providers can increase or decrease the number of replicates for each file, a strategy that has been widely used in content delivery networks [1]. The use of erasure coding, instead of replication, to improve the availability of content is not yet fully understood. Using erasure codes has been shown to be effective in reducing the delay in accessing a file stored on multiple servers [2, 3, 4]. However, only a few works have studied their use to store multiple files. Some recent works [5] have proposed new classes of erasure codes to store multiple files that allow a file to be read from from disjoint sets of nodes. Other works [6, 7] study the delay reduction achieved using these codes.

Besides download latency, it has recently been recognized that another important metric for the availability of stored data is the service rate [8, 9, 10]. Maximizing the service rate (or the throughput) of a distributed system helps support a large number of simultaneous system users. Rate-optimal strategies are also latency-optimal in high traffic. Thus, maximizing the service rate also reduces the latency experienced by users, particularly in highly contending scenarios.

This paper is one of the first to analyze the service rate region of a coded storage system. We consider distributed storage systems in which data for files is to be stored across nodes. A request for one of the files can be either sent to a systematic node or to one of the repair groups. We seek to maximize such systems’ service rate region, that is, the set of request arrival rates for the files that can be supported by a coded storage system.

The problem addressed in this paper should not be confused with the related problem of caching and pre-fetching of popular content at edge devices [11]. Caching benefits for the network are measured in reduction in the backhaul traffic it enables. Quality of service to the user measures include cache hit ratio and cache hit distance. Rather than with the backhaul, this paper is concerned with the access part of the network, namely, with potential service rate increase through work provided, jointly and possibly redundantly, by multiple network edge devices. Consequently, instead of measuring e.g., content download performance by the likelihood of an individual cache hit or cache memory and bandwidth usage, we strive to ensure that multiple caches are jointly in possession of content and can deliver it fast to multiple simultaneous users.

In [9], the achievable service rate region was found for some common classes of codes, such as maximum-distance-separable (MDS) codes and simplex codes. That paper also determined the service rate region when , with arbitrary numbers of systematic and coded nodes. We generalize this service rate region result from files to files and provide an algorithm to maximize the requests served for a given file with general . The paper begins with preliminary notions given in Sec. II. Sec. III addresses the general case where all nodes are coded, and Sec. IV addresses the case. We return to the general case in Sec. V.

Ii Preliminaries

Suppose files are stored across a system that consists of nodes labeled . For , there is a collection of minimal sets that each correspond to a set of nodes that gives access to file . Each such minimal set of nodes is called an –repair group.

Example 1.

Fig. 1 shows one possible way to store two files, and , across four nodes. In this system, the -repair groups are , and . The -repair groups are and .

Fig. 1: A possible way to store two files on nodes.

For , define the function


Suppose that when a request for file is received, that request is sent at random to an –repair group according to a splitting strategy with denoting the fraction of requests sent to repair group , so that for each ,


Let the demand for file be , so the arrival of requests for file to the storage system queue is Poisson with rate , and let record the demand for files .

The average rate that file requests arrive at a storage system node depends both on the splitting strategy for file requests and on the demand . More precisely, the average rate that file requests are received at node is


Let denote the average rate of resolving received file requests at node . Whenever demand is such that at least one node of the storage system receives requests at an average rate in excess of its , the storage system queue will have a tendency to grow. With this in mind, it is appropriate to call the service rate of node . We will consider uniform systems for which for . If, at demand , there exists a splitting strategy under which no storage system node receives requests at a rate in excess of its service rate, then is said to be in the achievable service rate region of the storage system. More formally, the storage system’s achievable service rate region is the set of all such that there exists a splitting strategy with


For any , denote by the -tuple , and for let ). If , then the same splitting strategy whose existence is guaranteed by (4) is also sufficient to give for every satisfying for all , . Thus, given any pair and , the entire interval is in . Moreover, for any storage system (regardless of its coding), if is such that the demand for any file is in excess of , then under all possible assignment strategies (4) is violated for at least one node , and so is not in for any and . In this way, is a non-empty, closed, and bounded subset of . Therefore, given any , there exists a maximal value of such that and for any . When , we call this maximal value . In this notation, the service rate region of any storage system can be described as:

Example 2.

Three examples of how two files, and may be stored across three nodes are shown on the below on the left. The resulting service service rate regions for each system are shown below on the right.

Coding schemes that use a mixture of replication and MDS coding are not conventional. However, if the service rate region is used as a performance metric, then a combination of coded and systematic nodes has been shown to be beneficial [6, 9]. In this paper, we consider storage systems for files whose coded nodes satisfy the following three conditions:

  1. Each –subset of coded nodes forms an –repair group for every .

  2. No subset of coded nodes forms an –repair group, for any .

  3. With addition of systematic nodes for any distinct files (naturally, ) every –subset of coded nodes from the core completes these systematic nodes to form an –repair group for every .

We say that such a system has an MDS core. We consider situations with uniform node capacities .

For convenience, we use to denote the number of coded nodes in such a core. When systematic nodes are also present, we use to denote the number of systematic nodes for file . In this way, the total number of nodes in a storage system for files that has an MDS core is .

Iii All coded nodes

We begin by considering an MDS -file core where there are no systematic nodes in the system. In this situation, all nodes form a repair group for each file, and nodes are required to recover any file.

Theorem 1.

Assume . If there are coded nodes, then the achievable service rate region is the set of all with , and so . If there are coded nodes, then is the point


If , then no file can be recovered and the service rate region is the point .

Assume . Note that since every repair group requires nodes, the total demand that can be served is bounded above by . For each file, there are a total of repair groups, and each node is in repair groups. By sending demand to each repair group, requests to each node occur at the service rate and the system can serve demand . Since this demand can be for any file, the service rate region is . Therefore, the maximum achievable is

The two file case is considered in [9]. The situation becomes increasingly complex depending on the number of files in the system. In the next section, we consider .

Iv Three files

In this section, we consider the service rate region of storage systems for files with MDS cores. As a corollary to Theorem 1, we obtain the service rate region for the case when there are no systematic nodes, which is represented in Fig. 2. Note that when the demand for one file is zero, then this may be considered a system with only two files. For example, if , then the maximum achievable is , which is the region shaded in Fig. 2.

Fig. 2: Achievable service rate regions of all-coded-node systems with files (left) or files (right).

We now consider storage systems that have both coded nodes and systematic nodes. Suppose that a coded storage system has coded nodes and systematic file nodes, . Note that a systematic repair node may be in a repair group with a single node (serving requests for the file it stores) or three nodes (serving requests for any other file). Any repair group using a coded node contains three nodes. For , if requests for file are served using systematic nodes (and any other demand for file is served using a repair group of three nodes), then the total demand that can be served is bounded above by

Given demand for file and for file , the rate of requests that may be served for file is bounded above by This is maximized when for . The splitting strategy in the proof of the following theorem meets this bound.

Theorem 2.

Assume there are , , and systematic nodes for files , , and , respectively, and coded nodes. Assume and . Then has


Consider a system with , , and systematic nodes for files , , and and coded nodes.

Step 1: Send requests to systematic nodes at the service rate to serve demand for files and , as possible. If any () systematic nodes remain available, distribute remaining file demand uniformly across those nodes.

Example 3.

Consider a -file system with , , , and .

If and then requests for will be served by one of the systematic nodes, and the remaining requests for will be split between the other systematic nodes. Also, requests for will be served by the systematic node. After Step 1, the remaining demand for is and the remaining demand for file is . In the system, there are now two systematic nodes that can handle an additional requests as well as one systematic node and three coded nodes each with available service rate .

At the end of Step 1, if for or , then there will be systematic nodes remaining available for , each with service rate reduced to . Since , the remaining demand for file is .

If for or , we exhaust every systematic node. The remaining demand for file is then , and systematic nodes remain.

Step 2: Serve any remaining demand for files and . Finally, serve demand for file .

Example 4.

Consider the system in Example 3. In Step 2 we want to serve the remaining requests for file in a way that maximizes the requests that can be handled for . In particular, we will reserve the use of systematic nodes for accessing file . Note that there are repair groups for file that involve one systematic node and two coded nodes. If we send requests for file to each of these repair groups, then all the requests for file are served, each systematic node can serve more requests (as each node is in repair groups) and each coded node can serve more requests (as each coded node is in repair groups).

Finally, requests for may be served. Sending requests to each of the repair groups with one node and two coded nodes exhausts each node and each coded node. The full service rate of the systematic node may also be used to serve requests for . Thus a total of requests for may be served.

How requests are served in Step 2 depends on the demand and number of systematic nodes for files and . Let be the total demand for files and that remains after Step 1; that is, .

Case 1 (, ): In this case, , so all available system resources may be used to serve demand. The full service rate of file systematic nodes may be used, serving demand for file . Let be a permutation on such that .

There are –repair groups with a systematic node for each of and , and one coded node. Recall, . Since ,

demand for can be served by sending demand to each of these repair groups. The service rate of each is reduced to , while systematic nodes have , and coded nodes have .

There are –repair groups with one of the remaining systematic file nodes and coded nodes. Since , similarly to before, we can serve

demand for file by sending demand equally to each of these –repair groups. Each coded node has remaining service rate and no systematic nodes remain available.

Since , as in the case in Theorem 1 with coded nodes and no systematic nodes, the service rate of these coded nodes can be used to serve demand for file .

Thus, the maximum achievable is

Similar arguments can be used for Case 2: , and Case 3: , (see Example 4).

Case 4 (, ): In this case, all available repair groups consist entirely of coded nodes. Since demand for file () was satisfied in Step 1, the remaining total demand for files and is . Since , this can be served by sending demand equally to every coded repair group. The coded nodes’ remaining ability to service can be used for file . Thus, the maximum achievable is

Note that can be found for systems with coded nodes in a similar way. When , all repair groups must contain systematic nodes for at least distinct files.

V Mds -file cores

Theorem 2 may be generalized to provide an algorithm for maximizing for the general -file case. Assume we have an MDS -file core with systematic nodes for files , respectively, and coded nodes, with demand for files . As in Theorem 2, we again assume . Our goal is to identify the maximal file request rate that can be served.

We can first serve file demand using their respective systematic nodes. This process is analogous to Step 1 in Theorem 2. Note, in this algorithm, the same demand is sent to every file systematic node, and also to every coded node, so we can let and represent the updated service rate of systematic file nodes and coded nodes, respectively.

We can then serve any remaining total demand using -tuples of coded and systematic nodes. This is analogous to Step 2 in Theorem 2. Let denote the number of files (excluding file ) for which the system contains systematic nodes. There are repair groups with systematic nodes and coded nodes. Letting be the index minimizing for positive , we can serve demand by sending demand to each of these repair groups. This exhausts file systematic nodes, while file systematic nodes (, , ) have reduced service rate


which is The remaining coded nodes have reduced service rate


which is We can continue in this way until the systematic node service rate is met for all but file . Then, we can use repair groups that consist entirely of coded nodes, applying Theorem 1. Once all demand for files has been satisfied, we can follow a similar process to utilize any remaining system resources to serve demand for file . Note, once the coded nodes have been exhausted, or if there are too few coded nodes to form a -tuple, no demand may be satisfied using only coded nodes. We may then serve demand for file using systematic file nodes.

  INPUT: ,
   for from to Step 1:
  for  from to  do
     if  then
     end if
  end for Step 2:
  while  and  do
     if  then
         the index minimizing
        if  then
           if  then
           end if
        end if
        if  then
            apply Equation 7
        end if
         apply Equation 6 if for
        if  then
        end if
     end if
  end while
Algorithm 1 Maximize


The initial stages of this work were performed at ICERM (Institute for Computational and Experimental Research in Mathematics) in Providence, RI. We are indebted to the organizers of the ICERM 2017 Women in Data Science and Mathematics Research Collaboration Workshop.


  • [1] S. Borst, V. Gupta, and A. Walid, “Distributed caching algorithms for content distribution networks,” in Proc. IEEE INFOCOM, Mar. 2010, pp. 1–9.
  • [2] G. Joshi, Y. Liu, and E. Soljanin, “On the delay-storage trade-off in content download from coded distributed storage systems,” IEEE JSAC, vol. 32, no. 5, pp. 989–997, May 2014.
  • [3] N. Shah, K. Lee, and K. Ramachandran, “The MDS queue: Analyzing the Latency Performance of Erasure Codes,” in Proc. IEEE Int. Symp. Inform. Theory, Jul. 2014.
  • [4] G. Joshi, E. Soljanin, and G. Wornell, “Efficient redundancy techniques for latency reduction in cloud systems,” ACM Trans. Modeling and Performance Evaluation of Computing Systems, May 2017.
  • [5] A. Rawat, D. Papailiopoulos, A. Dimakis, and S. Vishwanath, “Locality and availability in distributed storage,” in Proc. IEEE Int. Symp. Inform. Theory, June 2014, pp. 681–685.
  • [6] S. Kadhe, E. Soljanin, and A. Sprintson, “When do the availability codes make the stored data more available?” in 2015 53rd Annu. Allerton Conf. Commun., Control, and Computing, Sept 2015, pp. 956–963.
  • [7] M. F. Aktas, E. Najm, and E. Soljanin, “Simplex queues for hot-data download,” in Proc. 2017 ACM SIGMETRICS/Int. Conf. Measurement and Modeling of Computer Systems.   ACM, 2017, pp. 35–36.
  • [8] M. Noori, E. Soljanin, and M. Ardakani, “On storage allocation for maximum service rate in distributed storage systems,” in Proc. IEEE Int. Symp. Inform. Theory (ISIT), July 2016, pp. 240–244.
  • [9] M. Aktas, S. E. Anderson, A. Johnston, G. Joshi, S. Kadhe, G. L. Matthews, C. Mayer, and E. Soljanin, “On the service capacity of accessing erasure coded content,” in Proc. Allerton Conf. Commun., Control and Computing, Oct. 2017.
  • [10] G. Joshi, “Synergy via redundancy: Boosting service capacity with adaptive replication,” in Proc. IFIP Performance, Nov. 2017.
  • [11] M. A. Maddah-Ali and U. Niesen, “Coding for caching: fundamental limits and practical challenges,” IEEE Commun. Mag., vol. 54, no. 8, pp. 23–29, 2016.