Random Construction of Partial MDS Codes

01/17/2018 ∙ by Alessandro Neri, et al. ∙ 0

This work deals with partial MDS (PMDS) codes, a special class of locally repairable codes, used for distributed storage system. We first show that a known construction of these codes, using Gabidulin codes, can be extended to use any maximum rank distance code. Then we define a standard form for the generator matrices of PMDS codes and use this form to give an algebraic description of PMDS generator matrices. This implies that over a sufficiently large finite field a randomly chosen generator matrix in PMDS standard form generates a PMDS code with high probability. This also provides sufficient conditions on the field size for the existence of PMDS codes.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In a distributed storage system a file (where denotes the finite field of cardinality ), is encoded and stored as some codeword , over several storage nodes. Each of these nodes is assumed, for simplicity, to store exactly one coordinate of . In case some of the nodes fail, we want to be able to recover the lost information using as little effort as possible. The locality of a code plays an important role in this context, and it denotes the number of nodes one has to contact for repairing a lost node. We call the set of nodes one has to contact if a given node fails, the locality group of that node. For this work we assume that the locality groups are distinct. Partial MDS codes are maximally recoverable codes in this setting, i.e., any erasure pattern that is information theoretically correctable is correctable with such a code.

It is known that maximally recoverable codes in general, and PMDS codes in particular, exist for any locality configuration if the field size is large enough [1]. Furthermore, some constructions of PMDS codes are known, e.g. [2, 3, 4, 5, 6].

In this paper we describe a random construction of PMDS codes, by prescribing a generator matrix of the respective code in a specific form, which we will call PMDS standard form. If we fill the non-prescribed coordinates of this generator matrix with random values, by high probability, the resulting code is PMDS, if the underlying field size is large enough. We derive a lower bound on this probability (depending on the field size). This gives rise to a lower bound on the necessary field size for PMDS codes to exist. With some final adjustments on the random construction we get a lower bound that improves the one of [1].

2 Preliminaries

2.1 PMDS codes

Consider a distributed storage system with disjoint locality groups, where the -th group is of size () and can correct any erasures. First we set the locality for the code to be . We can divide the coordinates of the code into blocks of length , where , such that each block represents a locality group.

We denote an MDS code of length and dimension by -MDS code. We use the definition of PMDS codes given in [7], which generalizes the definition of [2].

Definition 1.

Let . Define and let be a linear code of dimension with generator matrix

(1)

such that . Then is a -partial-MDS (PMDS) code (with locality ) if

  • for the row space of is a -MDS code, and

  • for any erasures in the -th block (), the remaining code (after puncturing the coordinates of the erasures) is a -MDS code.

The erasure correction capability of PMDS codes is as follows:

Lemma 2.

[7, Lemma 3] A -PMDS code can correct any erasures in the -th block (simultaneously) plus additional erasures anywhere in the code.

We can see that the definition of PMDS codes given makes sense only for . In case of equality, or in the case that there exist only trivial PMDS codes, i.e. the only PMDS codes are MDS codes.

It was shown in [7] that a code is a -PMDS code if and only if it is maximally recoverable (for the respective locality group configuration). The same results had previously been shown in [6, Lemma 4] for the case .

Now we give a summary on known results about PMDS codes.

Proposition 3.

[1] Maximally recoverable (MR) codes of length and dimension exist for any locality configuration over any finite field of size .

MR codes are PMDS codes for disjoint locality blocks. Therefore, Proposition 3 implies that PMDS codes exist for any set of parameters when the field size is large enough.

A construction of PMDS codes based on rank-metric and MDS codes was given in [8], when . This gives the following existence result:

Proposition 4.

[8] -PMDS codes with locality blocks of the same length exist over a finite field of size .

Furthermore, some specific constructions of PMDS codes, for particular values of or of the , are given in [2, 3, 4, 6].

In particular, a general construction for PMDS codes with was given in [7]. This construction is based on the concatenation of several MDS codes as building blocks.

Proposition 5.

[7, Corollary 14]

  1. For any integers and there exists a -PMDS code over any field with .

  2. If there exists such that and , then there exists a -PMDS code over with .

In [7] the authors also show that this construction is basically the only one possible, i.e., every PMDS with is of this form, giving thus a characterization for this set of parameters. However, for there is no characterization yet for PMDS codes.

2.2 Zarisky topology over finite fields

Let be a field, and be the polynomial ring over . Denote by the algebraic closure of . For a subset we define the algebraic set

The Zariski topology on is defined as the topology whose closed sets are the algebraic sets, while the complements of the Zariski-closed sets are the Zariski-open sets [9, Ch. I, Sec. 1].

Definition 6.

A subset is called a generic set if contains a non-empty Zariski-open set.

In classical geometry one studies the Zariski topology over the complex numbers. In this framework, a generic set inside is dense and its complement is contained in an algebraic set of dimension at most .

If one wants to consider generic sets restricted to a finite field , the situation is slightly different. Here, for a subset one can always find a set of polynomials such that

and therefore the Zariski topology restricted to is the discrete topology. This means that it is not useful to extend the notion of generic sets to finite fields since it would not give any information.

However, given a set of polynomials , we can define the set of -rational points as

In this setting the Schwartz-Zippel Lemma implies an analog result to the one of generic sets, as explained in the following.

Lemma 7 (Schwartz-Zippel).

[10, Lemma 1.1] Let be a non-zero polynomial of total degree . Let be a finite set and let be selected at random independently and uniformly from . Then

As a consequence of this result we have that, in case the size of and the total degrees of the polynomials in do not depend on the finite field, the proportion between the cardinality of and the cardinality of the whole space goes to as grows. Vice versa, for growing the probability that a random point is in the complement of tends to . This result will be crucial in Section 5 for our random construction of PMDS codes.

2.3 Rank-metric codes

We now give some known facts about rank-metric codes. Recall that is isomorphic to as an

- vector space. From this it easily follows that

. Then we can give the following definition.

Definition 8.

The rank distance on is defined by

Analogously, if , then is the rank of the difference of the respective matrix representations in .

Observe that the definition of rank distance in the case of vectors in does not depend on the choice of the basis. Moreover it can be shown that the function is a metric.

Definition 9.

An -linear rank-metric code of length and dimension is a -dimensional subspace of equipped with the rank distance. The minimum distance of is defined as

Theorem 10 (Singleton-like Bound).

[11, Theorem 1] Let be an -linear rank-metric code of dimension . Then

Definition 11.

Codes attaining the Singleton-like bound are called Maximum Rank Distance (MRD) Codes.

A necessary and sufficient condition for the existence of MRD codes is that . In this framework, a characterization for -linear MRD codes in terms of their generator matrices was given in [12, Corollary 2.12], which in turn is based on a result given in [13]. For this we define the set

Proposition 12 (MRD criterion).

Let be a generator matrix of a rank-metric code . Then is an MRD code if and only if

for all .

3 General construction using rank metric codes

In this section we generalize the construction given in [8]. In that work the authors use Gabidulin codes in order to build -PMDS codes. We will show that this construction also works for different , and that Gabidulin codes can be replaced by any linear MRD codes.

Fix , and let be the generator matrix of a MRD code. For the existence of an MRD code we need . Moreover, for every , we consider a -MDS code over with generator matrix , and define

(2)

We can now formulate our PMDS construction.

Theorem 13.

Let be the generator matrix of a MRD code and let be the matrix defined in (2). Then the matrix is a generator matrix for a -PMDS code over .

Proof.

Let and let be the submatrix obtained by selecting columns from the th block for , where and . is equal to , where

and is the submatrix of obtained by the respective selected columns. Since generates an -MDS code, any columns of are linearly independent. Thus, and . By Proposition 12 we have that , and we conclude the proof using Proposition 16. ∎

Corollary 14.

Let and , be positive integers. Then, for every prime and every positive integer there exists a -PMDS code over , where

Proof.

A MRD code in exists if . Suitable MDS codes over for the matrix in (2) exist if . The statement follows from Theorem 13 with and . ∎

4 Algebraic description of PMDS codes

We will now define a standard form for generator matrices of PMDS codes. This standard form is the main tool for the random construction of PMDS codes.

Theorem 15 (PMDS standard form).

Let and and let be a -PMDS code over a field . Then has a generator matrix of the form

(3)

where

  • , and for , and

  • the submatrix is of the form

    with being superregular.

Proof.

Let be a generator matrix for of the form (1), i.e.

Puncturing every block in the last columns, we get that the submatrix is the generator matrix of a -MDS code. Operating on the rows of such a submatrix we can transform it to a matrix , with

superregular. I.e., there exists an invertible matrix

such that , and therefore the matrix is a generator matrix of of the required form. ∎

We now consider the entries of as variables for amd . We know that the column space of is inside the column space of , by the parameters of the block MDS codes. This means that every column in is a linear combination of the columns of . If we denote by the th column of , then

(4)

for some , which we also consider variable. This way we can consider a generator matrix as a matrix in (where denotes the polynomial ring in all ).

Let . We denote and . If we replace the variables described above in a matrix in PMDS standard form by the values , we denote the corresponding generator matrix by

Analogously we will denote the variable form by .

However, a general matrix of this form is not necessarily a generator matrix of a PMDS code for any values . The following proposition shows what needs to be fulfilled to generate a PMDS code:

Proposition 16.

A matrix generates a -PMDS code if and only if, every submatrix in the set

has non-zero determinant.

Proof.

This follows from the definition of PMDS, cf. also [7]. ∎

The above results give an algebraic description of the generator matrix of a -PMDS code over , as follows. If we consider the variable form of a generator matrix as above, and the polynomial

(5)

then, we have that generates a -PMDS code over if and only if is non-zero.

5 Topological and probability results

In this section we first deal with the algebraic description of the generator matrix of a PMDS code in the algebraic closure of the finite field where we want our code to be built. After that, we analyze the probability that a code whose generator matrix is of the form is PMDS. Moreover, we also study the existence of PMDS codes for given parameters and , giving sufficient conditions on the field size. Although for this problem was completely solved in [7], for this is still an open problem.

We denote the set of valid entries for PMDS generator matrices over the algebraic closure of the finite field by

Then the following result holds.

Theorem 17.

is a generic set.

Proof.

By Proposition 16 we have that

and therefore is a Zariski open set.

Concerning the non-emptiness, let . From Corollary 14 there exists a -PMDS code over , for some multiple of . By Theorem 15, has a generator matrix of the form , therefore . ∎

This means that over the algebraic closure, by probability , for randomly chosen the matrix generates a PMDS code. For underlying finite fields, this implies that for growing field size this probability will tend to . We now derive a probability formula depending on the field size.

We can easily observe that the entries of are polynomials of total degree or . In particular, if and we write as in (3), then the entries of the blocks are polynomials of degree at most for , and of degree at most for the last blocks.

To estimate the degree of

we need the following lemma.

Lemma 18 (Derivation of Vandermonde’s identity).
Lemma 19.

The total degree of the polynomial , defined as in (5), satisfies the inequality

Proof.

It holds that

hence the polynomial divides the polynomial

Observe that the entries of the first columns of the submatrix have degree 0. Let . Then the entries of the columns corresponding to the blocks for have degree at most , as well as the last columns of . Finally, the columns of the blocks for , have degree at most . In particular, all the entries of the blocks and the last columns of have degree at most . Therefore,

where the last equality follows from Lemma 18. Since we conclude the proof. ∎

We can now formulate a lower bound for the probability that a randomly chosen generator matrix in PMDS standard form generates a PMDS code over a finite field :

Theorem 20.

Let the entries of and be uniformly and independently chosen at random in . Then

Proof.

We have

where the last two inequalities follow from Lemmas 7 and 19, respectively. ∎

From this we can deduce an existence result for PMDS codes over finite fields of a given minimal size.

Corollary 21.

If then there exists a -PMDS code over the finite field .

One notices that this is not an improvement over the known existence result from Proposition 3.

However, we can improve the above result, considering a step-by-step construction. We will again consider a generator matrix in PMDS standard form as in (3). We start with an -MDS code over a finite field and write its generator matrix as . For this purpose it is sufficient that . Then we construct the first column of the block as in (4). Every entry will be a degree polynomial in the variables for . Imposing that every minor of

is non-zero, we get the condition , where

Using Lemma 7, we obtain

In this situation , therefore for we have that there exists at least one evaluation of that is non-zero and such that generates a -PMDS code.

Repeating this construction step by step, we get a -PMDS code. From that code we build the last column of the block again as in (4):

In the end we get the matrix

(6)

where the matrix without the last column generates a -PMDS code, and the entries of the last column are polynomials of total degree at most in the variables , for .

Definition 22.

Let be positive integers such that . Let , and for . We define the set

and as its cardinality.

Proposition 23.

Let be as in (6). The total degree of the polynomial

is less or equal to

Proof.

The polynomial has degree less or equal to . By assumption all the determinants for not containing the last column are non zero elements in . The only polynomials with degree are the determinants of submatrices involving the last column, and they are exactly many. ∎

Corollary 24.

If then there exists a -PMDS code over the finite field .

To our knowledge there is no closed formula for . However, it is easy to see that and that the inequality is strict if any of the conditions is non-empty. Hence, Corollary 24 improves upon Proposition 3.

6 Conclusion

We gave a generalization of a known PMDS code construction based on rank-metric codes. Furthermore, we investigated a random construction of PMDS codes by prescribing a PMDS standard form. We derived a lower bound on the probability that a randomly filled matrix in PMDS standard form generates a PMDS code. This probability implies a lower bound on the field size needed for such codes to exist. In the end we gave a step-by-step construction of such a generator matrix to improve this lower bound on the necessary field size.

References

  • [1] M. Chen, C. Huang, and J. Li, “On the maximally recoverable property for multi-protection group codes,” in 2007 IEEE International Symposium on Information Theory, June 2007, pp. 486–490.
  • [2] M. Blaum, J. L. Hafner, and S. Hetzler, “Partial-MDS codes and their application to RAID type of architectures,” IEEE Transactions on Information Theory, vol. 59, no. 7, pp. 4510–4519, July 2013.
  • [3] M. Blaum, J. S. Plank, M. Schwartz, and E. Yaakobi, “Partial MDS (PMDS) and sector-disk (SD) codes that tolerate the erasure of two random sectors,” in 2014 IEEE International Symposium on Information Theory, June 2014, pp. 1792–1796.
  • [4] ——, “Construction of partial MDS and sector-disk codes with two global parity symbols,” IEEE Transactions on Information Theory, vol. 62, no. 5, pp. 2673–2681, May 2016.
  • [5] J. Chen, K. W. Shum, Q. Yu, and C. W. Sung, “Sector-disk codes and partial MDS codes with up to three global parities,” in 2015 IEEE International Symposium on Information Theory (ISIT), June 2015, pp. 1876–1880.
  • [6] P. Gopalan, C. Huang, B. Jenkins, and S. Yekhanin, “Explicit maximally recoverable codes with locality,” IEEE Transactions on Information Theory, vol. 60, no. 9, pp. 5245–5256, September 2014.
  • [7] A.-L. Horlemann-Trautmann and A. Neri, “A complete classification of partial-MDS (maximally recoverable) codes with one global parity,” arXiv preprint arXiv:1707.00847, 2017.
  • [8] G. Calis and O. O. Koyluoglu, “A general construction for PMDS codes,” IEEE Communications Letters, vol. 21, no. 3, pp. 452–455, March 2017.
  • [9] R. Hartshorne, Algebraic geometry.   Springer Science & Business Media, 2013, vol. 52.
  • [10] D. Lewin and S. Vadhan, “Checking polynomial identities over any field: Towards a derandomization?” in

    Proceedings of the thirtieth annual ACM symposium on Theory of computing

    .   ACM, 1998, pp. 438–447.
  • [11] R. Roth, “Maximum-rank array codes and their application to crisscross error correction,” IEEE Transactions on Information Theory, vol. 37, no. 2, pp. 328 –336, mar 1991.
  • [12] A.-L. Horlemann-Trautmann and K. Marshall, “New criteria for MRD and Gabidulin codes and some rank-metric code constructions,” Adv. in Math. of Comm., vol. 11, no. 3, pp. 533–548, 2017.
  • [13] E. M. Gabidulin, “Theory of codes with maximum rank distance,” Problemy Peredachi Informatsii, vol. 21, no. 1, pp. 3–16, 1985.