A McEliece cryptosystem using permutation codes

by   Adarsh Srinivasan, et al.

This paper is an attempt to build a new public-key cryptosystem; similar to the McEliece cryptosystem, using permutation error-correcting codes. We study a public-key cryptosystem built using two permutation error-correcting codes. We show that these cryptosystems are insecure. However, the general framework in these cryptosystems can use any permutation error-correcting code and is interesting.



There are no comments yet.


page 1

page 2

page 3

page 4


Error correcting codes from sub-exceeding fonction

In this paper, we present error-correcting codes which are the results o...

HQC-RMRS, an instantiation of the HQC encryption framework with a more efficient auxiliary error-correcting code

The HQC encryption framework is a general code-based encryption scheme f...

On the decoding of 1-Fibonacci error correcting codes

The study of new error correcting codes has raised attention in the last...

On the Difficulty of Evolving Permutation Codes

Combinatorial designs provide an interesting source of optimization prob...

Ensemble Learning using Error Correcting Output Codes: New Classification Error Bounds

New bounds on classification error rates for the error-correcting output...

Nonexistence of perfect permutation codes under the Kendall τ-metric

In the rank modulation scheme for flash memories, permutation codes have...

Verifying the Steane code with Quantomatic

In this paper we give a partially mechanized proof of the correctness of...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

These days McEliece and Niederreiter cryptosystems are very popular. One of the reasons behind this popularity is that there are some instances of these cryptosystems that can resist quantum Fourier sampling attacks. These makes them secure, even when quantum computers are built. These kind of cryptosystems are called quantum-secure cryptosystems [3, 4].

McEliece and Niederreiter cryptosystems use linear error-correcting codes. Similar to linear error-correcting codes, we can define permutation error-correcting codes

. These codes use permutation groups the same way linear codes use vector-spaces. This paper is an attempt to build secure public-key cryptosystems using permutation codes in the same spirit as McEliece and Niederreiter cryptosystem are built using linear error-correcting codes. We say it upfront that this attempt was unsuccessful. The cryptosystems that came out of permutation codes is not secure. However, the journey we took to build these cryptosystems is interesting in its own right. Furthermore, this attempt might inspire more work in this direction.

In this paper we ask two questions:

Can one build a secure public-key cryptosystem using permutation codes, similar to cryptosystems built using linear error-correcting codes?

Are there any advantages of using permutation codes compared to linear codes in public-key cryptosystems?

We tried to keep this paper as self contained as possible. In the next section, we present a brief overview of permutation codes. Most of it is a review of Bailey’s work [10, 9, 8] on permutation codes. We have performed some extra computational analysis on the alternative decoding algorithm he described in Section 2.3.1. In the section after that, we present the subgroup distance problem. This problem is related to cryptography that we study in this paper and is interesting in its own right. The section after that presents a public-key cryptosystem using permutation codes whose security depends on the hardness of decoding generic permutation codes. We study this cryptosystem using two permutation codes – the symmetric group acting on 2-subsets and a class of wreath product groups. We show that both these cryptosystems are insecure. The first one is insecure due to an information set decoding attack and the latter due to the inherent structure in the wreath product.

2 Permutation error-correcting codes

The use of sets of permutations in coding theory (also referred to as permutation arrays) was studied since the 1970’s. Blake et al [12, 13] were the first to discuss using permutations this way. They had certain applications in mind. One example was powerline communications. However, permutation codes have never got the level of attention that linear codes did. The idea of using groups of permutations in coding theory was first studied by Bailey and Cameron [8]. Bailey, in his thesis, describes a variety of permutation codes and presents a decoding algorithm which works for arbitrary permutation groups using a combinatorial structure called uncovering-by-bases. They were the first to exploit the algebraic structure of groups to come up with decoding algorithms for several families of groups.

In this section, we explore the use of permutation groups as error-correcting codes. We begin by defining a Hamming metric on permutations, similar to the one defined for vector spaces over finite fields. Let us define a set . For a permutation , we define its support and set of fixed points to be those points it moves and those points it fixes respectively. A permutation, which is a bijective function from to can be represented in two ways. The first is by listing down all its images. This is called the list form of a permutation. For example, is a permutation acting on such that . The other way is as a product of disjoint cycles. The list form of a permutation is more useful in defining a Hamming metric.

Definition 2.1.

For elements , the Hamming distance between them, is defined to be the size of the set .

In other words, the Hamming distance between two permutations is the the number of positions on which they differ when written down in list form. The Hamming distance between two permutations and can be proved to be equal to . A permutation code is a collection of permutations in endowed with a Hamming metric. For a subset of , the minimum distance of the subset is defined to be . In this paper, we assume that the collection of permutations form a subgroup of . In that case, the minimum distance can be shown to be:


This is also known as the minimal degree of the permutation group. The quantity is called the correction capacity of the code.

We can now define the distance between a permutation and a group to be the distance between and the permutation in closest to it.

The subgroup distance problem can be defined as follows: Given the generators of a permutation group , a permutation and an integer , is ? We can also generalise this problem to consider inputs to be lists of length with symbols from which do not necessarily form a permutation.

A nearest neighbour of in is defined to be a permutation in such that . This permutation exists because . If the distance between and is greater than the correction capacity of the group, this nearest neighbour need not be unique. If however, ’s nearest neighbour in is unique.

Hence, can be used as an error-correcting code in the following manner. The information we want to transmit is encoded as a permutation in and transmitted as a list through a noisy channel. We assume that when the message is received, or fewer errors have been induced. Such a channel is called a Hamming channel. The received message is a list of length with symbols from (which need not necessarily form a permutation). Because the correction capacity of is , the nearest neighbour of in is uniquely defined to be . The decoder can now solve the subgroup distance problem with the word and the group as input to recover the original message . However, while we have proved that decoding is possible in theory, because the subgroup distance problem is NP-hard (Section 3), generic algorithms cannot be used to decode permutation codes efficiently. Rather, we need specific algorithms tailor-made for different families of permutation codes.

2.1 A decoding algorithm using uncovering-by-bases

In this section, we review Bailey’s work on decoding algorithms for permutation codes. Consider a permutation group with correction capacity . Bailey’s decoding algorithm uses the following combinatorial structure. An uncovering-by-bases (UBB) is a collection of bases for . To recall, a base is a set of points with the property that the only element of which stabilises all the points is the identity. Given a base, we can define the following subgroup chain:


where is the pointwise stabilizer of in . We say that a base is irredundant if for all . A generating set for is called a strong generating set if for . where is the notation for the group generated by the set . Given a group , a base and a strong generating set for it can be constructed in nearly linear time in the size of the input using the Schreier-Sims algorithm. The orbits are called the fundamental orbits, and the (right) transversals are the set of (right) coset representatives for , with the requirement that the representative for the coset is the identity for all . They can be computed by keeping track of the permutations of that take to each point in the orbit. It follows from the definition of a base that every permutation in is uniquely defined by its action on the elements of a base, and we can reconstruct the permutation using the following procedure.

Corresponding to the base , we construct the fundamental orbits and the right transversals using the Schreier-Sims algorithm. Initially, we set to be the identity. We then find the permutation in such that . If does not lie in the fundamental orbit of , that means that no such exists and we can exit the procedure. We then replace with . In the -th iteration, we check if lies in the orbit of . If it does not, we exit the procedure and if it does we replace with , where is the permutation in the transversal that takes to .

Input: A base for and
Output: such that for to . false if no such exists
2 for  to  do
3       Compute the fundamental orbit and the transversals as a dictionary, with
4for  to  do
5       if  then
7      else
8            return false
9      for  to  do
Algorithm 1 Element reconstruction algorithm

For more details on these concepts, we refer the reader to the book on permutation group algorithms by Seress [22, Chapter 4]

Definition 2.2.

An uncovering-by-bases (UBB) to fix errors in a permutation group acting on is a set of bases for the group, such that, for each -subset of (An -subset of is a subset of with size ); there is at least one base in the UBB that is disjoint from the subset.

It is known that every permutation group has an uncovering-by-bases [10, Proposition 7]. However, they have been constructed only for a few permutation groups. There is no known procedure to construct a small UBB for an arbitrary permutation group. Once an uncovering-by-bases has been obtained we can use it in the following decoding algorithm (Algorithm 2).

Consider a permutation in list form and the list obtained after or fewer errors are introduced to . Note that can have repeated entries and hence need not be a permutation. Let be a subset of of size less than or equal to . Errors were introduced in positions from . The set is not known to the decoder a priory. Let be a UBB for . It is a consequence of the definition of a UBB that there exists a base in which is completely disjoint from . Hence, in the positions indexed by , the lists and are identical. As is a base, we can reconstruct the entire permutation from just its action on the points in using the element reconstruction algorithm. While the decoder does not know which base in is the appropriate base whose positions do not have an error induced, they can just repeat this procedure for every base in .

We now write down the pseudocode for the decoding algorithm. Consider a list , a base and a generator set . The function ElementReconstruction() constructs an element of the group generated by such that for all (Algorithm  1).

Input: A set of generators for with correction capacity , A UBB for , a list of symbols in
Output: such that . false if no such exists
1 for  do
2      if  has no repeated symbols in positions indexed by  then
4             if  and  then
5                  return
return false
Algorithm 2 UBB decoding algorithm

This algorithm is similar to the method of permutation decoding for linear codes proposed by MacWilliams and Sloane [17]. This involves using a subset of the automorphism group of the code which would move any errors out of the information positions.

2.2 Some constructions of uncoverings-by-bases

There are only a few examples of UBB that has been constructed by Bailey. One of them is the acting on -subsets and the other is a wreath product of a regular permutation group with a symmetric group. We talk about them next.

2.2.1 Symmetric group acting on -sets

Let be the set of all -subsets of . Consider the symmetric group acting on with the setwise action: . Hence, is a subgroup of , where in this action. The minimal degree of is and its correction capacity is  [10, Proposition 21].

The set of all -subsets of can also be viewed as the edge set of the complete graph . This means that we can use graph theoretical results such as the following to construct bases for , which we will use to construct a UBB.

A spanning subgraph of which has at most one isolated vertex and no isolated edges forms a base for . A Hamiltonian circuit of is a circuit in containing each vertex exactly once. From a Hamiltonian cycle of , we can obtain such a spanning graph (called a V-graph) by deleting every third edge [10, Fig. 2]. A Hamiltonian decomposition of a graph is a partition of the edge set of the graph into disjoint Hamiltonian circuits and at most one perfect matching. In the 1890’s, Walecki [5] showed that can be decomposed into Hamiltonian cycles when

is odd and that if

is even, it can be decomposed into Hamiltonian cycles and one perfect matching. Using Walecki’s decomposition, we can construct a UBB for . The UBB is the set of all V-graphs which can be obtained from the Hamiltonian decomposition of and it can be proved that any -subset of the edges of avoids at least one of these V-graphs. For more on this topic the reader is referred to Bailey [10]. This UBB is of size , and each base is of size . Hence, we can use this UBB in the decoding algorithm would have time complexity of .

2.2.2 Wreath product of regular groups

Let be a group acting on a set . It is called a regular group if, for every , there exists exactly one such that . The permutation code is the wreath product of , a regular permutation group acting on points with the symmetric group acting imprimitively on points.

The group consists of all tuples of the form , where and . We use the symbol to denote the identity element of and the symbol to denote the identity element of . In this section we redefine consisting of copies of the set as columns each with rows. The action of on is . The group acts on the columns and the group acts on the rows. As a regular group acting on points has size , .

Theorem 2.1.

The minimal degree of is and its error correction capacity is .


Consider the group element for some . This permutation moves all the points in the first column and fixes all the other points. Hence the minimal degree of is less than or equal to . We now show that any non identity permutation in moves at least points. Consider a permutation . If is not the identity, it must move all the points in one of the columns to another column and hence moves at least points. If is the identity permutation, at least one of the must be a non identity element of which will move all points in that column. ∎

Theorem 2.2.

Any subset of forms an irredundant base for if and only if it has exactly one point from each column of .


Consider any set which does not contain any points from the first column. The permutation fixes every point in . Hence, cannot be a base for . To prove the converse, consider the following set with each belonging to the column. Suppose here exists that fixes each of these points. As must lie in the column for each , this implies that must be the identity permutation. Hence, the action of on is . As the action of on , is regular, this implies that each is the identity.

We have proved that is a base for if and only if contains at least one point from each column of . If has more than one point from some of the columns of , we can obtain a subset of which contains exactly one point from each column of and is a base for . This means that must be redundant. Hence, we have proved that is a irredundant base for if and only if it has exactly one point from each column of . ∎

Using this theorem, we can now construct a UBB for . Each row for of forms a base and a collection of rows would form a UBB for , where . This is because any subset of size will intersect with at most of those bases described in the UBB and hence, disjoint from at least one of those bases.

Having obtained a UBB, we can use it in the decoding algorithm to obtain one which runs in .

2.3 An alternative decoding algorithm for wreath product

In this section, we present a simple decoding algorithm introduced by Bailey, which uses majority logic principles for the wreath product groups and compare its performance to the original UBB algorithm. For simplicity, we assume the group to be the cyclic group , although this algorithm can be extended to any regular group.

Consider a permutation in , with less than errors induced to form a word . The permutation acts on , which as we noted before, can be viewed of as copies of the set as columns each with rows. Firstly, we rewrite each member of and the letters in the received word in the form , with and . The columns of form a complete block structure for . As less than errors were introduced, the majority of positions in each block will contain the correct symbol. Let be of the form where each is a cyclic shift. In the column, we consider the position which contains the tuple , and calculate the number , which corresponds to the cyclic shift in that block. We set the value of to be the most frequently occurring cyclic shift and we set to be the most frequently occurring value of in that block. When we repeat this procedure for each block, we can obtain the cyclic shifts corresponding to each and obtain by computing its action on each . We now convert from the form to that of a permutation acting on .

We now estimate the complexity of this algorithm. The first part of the algorithm involves splitting the word into blocks and rewriting each symbol. This could be done in constant time for each position. Computing the value of the block label and the cyclic shift for each position involves some integer arithmetic which would also take a constant amount of time for each position. Hence this would take


The second part of the algorithm involves finding the most frequently occurring value of the block value and cyclic shift in each block. Let’s consider the part where we find the most frequently occurring value of the cyclic shift. We maintain an auxiliary list of size , with each position corresponding to the frequency of that cyclic shift. We iterate through the block, compute the cyclic shift for each location of the block and update its frequency in the auxiliary list. This procedure takes time. We then go through the auxiliary list to find the most frequently occurring cyclic shift. We do a similar procedure for finding the action of on the block using an auxiliary list of size and would hence take time. This procedure is repeated for each block.

The final part of the algorithm, which involves converting the reconstructed word back to a permutation form can be done with arithmetic operations. Hence, the algorithm takes time if and time if . This is faster than the UBB algorithm for the same group, which would take time.

2.3.1 Decoding more than errors

In fact, the algorithm described above can correct more than errors, as long as the majority of elements in each block are correct. Hence there are some error patterns with even up to errors which can be decoded using this algorithm. Bailey obtained the following expression for the number of error patterns of a certain length which can be corrected by this algorithm (an error pattern is a certain set of positions on which the errors are induced).

Let be a positive integer. Let be the set of all partitions of into at most parts with each part having size at most . We write down such a partition in the form , where is the number of times the number appears in the partition. For a partition , we define the quantity to be , which is the number of parts in of size strictly less than .

Theorem 2.3.

For an integer , the following number of patterns of errors can be corrected:


For a proof, we refer to Bailey [8, Section 6.7]. It is a farily simple counting argument using these definitions described above. ∎

We ran some simulations to find the proportion of error patterns which can be corrected for the case , in which case the minimal degree is and has correction capacity

. In practice a much larger number of errors can be corrected, with a high probability of success.

Using a computer program, we calculate the value of for different values of , and calculate the value of to obtain the probability that the alternate decoding algorithm succeeds. We plot this for :

We can deduce from the data the algorithm can decode percent of error patterns of length up to , percent of error patterns of length up to , percent of error patterns of length up to , and percent of error patterns of length up to .

Figure 1: Plot of fraction of errors that can be corrected for ,

Interestingly, with the probability of success fixed, the number of errors which can be corrected increases with . This is despite the minimal degree (and hence the correction capacity) remaining the same. We plot the number of errors which can be plotted with percent probability of success, with respect to , both in terms of absolute number of errors which can be corrected and in terms of the error rate, which is the ratio of the error induced to the degree of the permutation group. Unlike the correction capacity which increases with , the error rate seems to decrease slightly as increases.

Figure 2: Plot of error correction capacity at probability of success

3 The subgroup distance problem

In this section, we investigate the computational complexity of decoding permutation codes.

Given a permutation and a set of generators for a permutation group , the group membership testing problem is to determine if is a member of . With a base and a strong generating set, this problem can be solved efficiently using a procedure called sifting. The natural generalisation of the group membership testing problem would be to find a permutation in whose action on agrees with the target permutation in as many positions as possible. Unlike membership testing, there is no efficient algorithm to solve this problem, with the best known algorithms being exponential in the size of the input.

Problem 3.1 (Subgroup distance problem).

Given a group , a permutation and a natural number , does there exist a permutation whose action is identical to that of on at least points out of ?

For a given permutation and a group , let be the permutation in whose action is identical to that of on the maximum number of points, say . If is equal to , is a member of . Otherwise, the closer is to , the closer can be said to being a member of . The Hamming distance between and the group is defined to be . The subgroup distance problem can be viewed as the problem of computing this distance. The problem of decoding a general permutation code can be viewed as the subgroup distance problem along with a promise that the distance between the permutation and the group is less than the error correction capacity. This problem and generalisations of it using different metrics on permutation groups were studied by Buchheim et al [14] who showed that it is NP-complete, which means that it cannot be solved in polynomial time unless . Arvind and Joglekar [7] also studied this problem and similar weight problems in permutation groups for several metrics.

3.1 Computational complexity

In this section, we investigate the computational complexity of the subgroup distance problem. We now show that it is NP-complete by proving the existence of a polynomial time reduction from the Max--SAT problem, which is known to be NP-complete.

In computational complexity theory, we say that problem A is at least as hard as problem B if there exists an algorithm, called a reduction that runs in polynomial time and converts every instance of problem B to an instance of problem A. This means that an algorithm that solves problem A efficiently would also solve problem B efficiently. The complexity class P denotes the set of problems that can be efficiently (in polynomial time) solved. The complexity class NP denotes those whose solutions can be efficiently verified. An NP-hard problem is a problem in which is at least as hard as every other problem in NP. Assuming that , as is widely believed, this means that polynomial time solutions to NP-hard problems are not possible.

The problem of boolean satisfiability (SAT) was independently proved by Cook and Levin to be at least as hard as every other problem in NP. Most other problems are generally proved to be NP-hard by coming up with a reduction from another problem which is known to be NP-hard, eventually tracing back to SAT. For more details on NP completeness and reductions, we refer to [6].

Boolean variables are variables which can be either True or False. Boolean expressions are built from boolean variables using three logical operations– AND (), also called conjunction, OR (), also called a disjunction and negation, or NOT (). The AND and OR operations take two variables as input and give out one as output while the negation operation takes in one as input and gives out one as output. Given two boolean variables and , is True if and only if and are both True, and is True if and only if either or True. The negation of a variable, is True if and only if is False. An example of a boolean expression would be . A boolean expression is set to be satisfiable if there exists an assignment to the variables such that the truth value of the whole expressions is True. For more details, we refer to [15]. It is possible to simplify every boolean expression into the following form, called the conjunctive normal form. The new expression may, however be exponentially longer.

Definition 3.1 (Conjunctive normal form).

A formula is said to be in -CNF form, if it is a conjunction of one or more clauses, where each clause is a disjunction of exactly variables.

The expression is an example of an expression in -CNF form.

Problem 3.2 (-Sat).

Given a boolean expression in -CNF form, with variables and clauses, does there exist an assignment to these variables such that is satisfied?

The -SAT problem is also NP-hard for  [15, LO2, Appendix A9]. The -SAT problem is a fundamental problem in theoretical computer science, and many important problems in graph theory and combinatorial optimisation have been proven to be NP-hard by obtaining reductions from it. The problem of -SAT is solvable in polynomial time. The problem of determining an assignment to an instance of -SAT that maximises the number of satisfying clauses, is NP-hard [15, LO5, Appendix A9].

Problem 3.3 (max--Sat).

Given a -CNF formula with variables and clauses and a natural number . Does there exist an assignment to the variables that satisfies at least of the clauses?

We now show the existence of a reduction from max--SAT to the subgroup distance problem. This reduction is an algorithm that takes in an instance of max--SAT and outputs and instance of the subgroup distance problem. We note that the permutation group that this reduction outputs are always abelian and have exponent . This means that the problem remains NP-hard even in this restricted class of inputs.

Now, we prove that the subgroup distance problem is NP-hard by constructing a reduction from max--SAT.

Theorem 3.1.

There exists a reduction that runs in polynomial time that maps every instance of Max--SAT on variables to an instance of the subgroup distance problem with the property that .


Let be an instance of Max--SAT with clauses with variables . We create an instance of the subgroup distance problem with acting on points, and .

Corresponding to each clause , we define the clause gadget consisting of points, and define the set to be the union of all these clause gadgets. For every variable , we define a corresponding generator which acts on as follows: A variable can appear in a clause either in the first position or the second position. If appears in the clause in the first position, it swaps with and with . Then appears in the clause in the second position, it swaps with and with . If the variable does not appear in the clause at all, it fixes all the points in the clause gadget.

Some of the clauses in may contain some negations. The target permutation will depend on which clauses and the positions in these clauses these negations occur in. Consider a clause . If is of the form , acts on as . If is of the form , acts on as . If is of the form , acts on as . If is of the form , acts on as .

It remains to show that there exists an assignment for satisfying clauses if and only if there exists a permutation whose action on is same as that of on at points. For an assignment , Consider the permutation

Consider a clause gadget corresponding to a clause containing two variables and . There are four possible assignments to the variables in the clause. Each clause specifies an assignment to the variables that will not satisfy it, and the other three will. Observe that by construction, agrees with on points of if is satisfied and it agrees on no points if is not satisfied. All the other generators fix all points in . Hence, if clauses are satisfied, agrees with on exactly points.

Now, assume that there exists whose action agrees with that of on points. As is a member of and is abelian with every element being an involution, it can be uniquely written as a product of the defined generators. We define the assignment by setting to be True if and only if appears in the composition of . As before, for any clause, agrees with on points of if is satisfied and it agrees on no points if is not satisfied. Hence, if agrees with on points, satisfies clauses of . ∎

As Max-2-SAT is a NP-hard problem, the existence of such a reduction shows that the subgroup distance problem is also NP-hard, which means that an efficient algorithm to solve it very unlikely. Our reduction is slightly different from the reduction demonstrated by Buchheim et al [14]. In that paper, the authors kept the target permutation to be fixed, while here, this permutation varies depending on the negations in the input Max--SAT instance. The advantage of our reduction is that it yields a permutation group of size , whereas their reduction yields a permutation group of size . This fact may come in useful if we are to examine the hardness of this problem from a more fine-grained perspective.

It is important to note that this reduction only proves a worst case complexity result. It says nothing about the average case complexity of this problem. In cryptography, we are looking for problems that are hard on average, not just in the worst case. Currently, only lattice-based schemes have such a theoretical backing [1, 2]. We have also just proved a result on the non-promise version of the problem whereas the cryptosystem we propose uses a promise version of the problem. This is similar to the situation in linear coding theory, where Bounded Distance Decoding has not been shown to be NP hard whereas several hardness results are known for the Nearest Codeword Problem. Based on similar assumptions for analogous problems in linear coding theory, we assume that this problem is hard to solve on average too.

4 A cryptosystem on permutation error-correcting codes

The McEliece cryptosystem is a very popular cryptosystem using linear codes, first proposed by Robert McEliece [18] in 1978. Due to its large key sizes, it never gained a lot of popularity at the time. Recently with the rise of quantum computing, it has gained a lot of attention along with its counterpart proposed by Neiderreiter [19] as a post-quantum alternative to currently used public key cryptosystems. Fundamentally, the McEliece cryptosystem uses two linear codes, one kept private and another made public with the two codes being equivalent in the following way:

Definition 4.1.

Consider two linear codes and generated by the generator matrices and . These codes are considered equivalent if, there exists a non singular matrix and an permutation matrix such that

Consider a linear code generated by a matrix . Pre-multiplying by a non-singular matrix will not change the code it generates, as this is basically a change of basis for the code . Post-multiplying by a permutation matrix creates a new code, one whose codewords are obtained by permuting the positions of the codewords in by the permutation corresponding to the permutation matrix. In other words, there is a distance preserving map, or isometry between two equivalent linear codes. The principle behind the McEliece cryptosystem is that we have a map between an easy instance and a hard instance of the closest vector problem in linear codes using this distance preserving map. The private key matrix, is chosen from a family of linear codes with a known fast decoding algorithm, but the public key matrix generates a linear code is assumed to not have a fast decoding algorithm. The map between these easy and hard instance is known only to the owner of the private key. This provides the one-way property which is essential to all public-key cryptosystems.

In the previous section, we studied using permutation groups as error-correcting codes. In this section, we investigate the following questions:

  • Can we develop a public key cryptosystem similar to the McEliece cryptosystem using permutation codes instead of linear codes?

  • Is there any potential advantage of using permutation codes instead of linear codes from the standpoint of cryptography?

The first step in this direction would be to investigate isometries of the symmetric group with the Hamming metric. Let us say we know an isometry of the Hamming space. We can have the private key as a permutation code (which is a subgroup of ) with an efficient decoding algorithm and the public key as the permutation code

for which the decoding algorithm will not work. The isometries in the symmetric group were studied and classified by Farahat 

[16] who proved that they are of one of the following types:

  • for and being fixed members of

  • for and being fixed members of

However the isometries that interest us should be homomorphisms. This is mainly due to practical reasons. We want both the codes and to be subgroups of . This is because a permutation group can be represented efficiently using a generator set. A permutation array without a group structure cannot be represented efficiently this way. Hence, as we want both and to be groups. We are interested in isometries of that map subgroups to subgroups and are homomorphisms.

Now, we combine these ideas to create a public-key cryptosystem. Our ingredients are a permutation code with a fast decoding algorithm and a conjugation map.

4.1 A McEliece cryptosystem using permutation codes

In this section, we present a public-key cryptosystem using permutation codes that is similar to the McEliece cryptosystem.

Private key:

We choose a subgroup of with a fast decoding algorithm and a permutation from randomly. The private key is a set of generators , for and .

Public key:

The public key is a set of elements which generate the conjugate group and a base for . Note that a security assumption is that does not have a fast decoding algorithm.


The plaintext is an of . We then introduce errors to randomly to create the ciphertext .


Once we receive , we compute . We then use the fast decoding algorithm to obtain , which is its nearest neighbour in . We then compute .

The conjugation map is an isometry of the Hamming space. That is why this cryptosystem work. The distance between and is equal to the distance between and . In the McEliece cryptosystem, post-multiplication by a permutation matrix is the Hamming isometry used. The secret scrambler, which just scrambles the basis of a linear code to create a new basis. It doesn’t change the linear code. Here, the analogue to that would be the fact that we choose a random set of generators for the conjugate group as the public key.

There is a possibility of using other isometries other than the conjuation map. However, the errors have to be induced more carefully. For example, suppose we have a group and a map from to which is an isometry on . Our permutation code is a subgroup . Hence, if we induce errors in so that the ciphertext lies in the group , the cryptosystem using this isometry will work too. As of now, we do not have an example of a situation where doing this would be beneficial.

4.2 The security of our cryptosystem

4.2.1 Security assumptions

There are two security assumptions:


Solving the general decoding problem in is hard. This means that has no good decoding algorithm.


Inability to construct a subgroup of with generators and , such that and there is an efficient decoding algorithm in that can correct up to errors.

We now discuss possible attacks on our cryptosystem and it’s security in the face of these attacks. Like the McEliece cryptosystem, attacks on this cryptosystem can broadly be classified into two types– unstructured and structural attacks. Unstructured attacks attempt to solve the nearest neighbour problem in directly without attempting to obtain the private key. A structural attack would attempt to obtain the private key from the available public information.

The group has an efficient decoding algorithm. For the cryptosystem to be secure, the group must not have an efficient decoding algorithm. Specifically, if the algorithm used to decode in can be modified to work for the conjugate group , the cryptosystem will not be secure. This would be another type of attack which would be specific to the cryptosystem using a particular permutation code. Hence, the security of the cryptosystem depends on the structure of the permutation code used and its decoding algorithm, and not just on parameters like size, correction capacity, etc. This is especially true in the context of structural attacks. Just like the case with the McEliece cryptosystem, we cannot provide any theoretical basis to the claim that the cryptosystem using a certain permutation code is secure. Rather, we construct a cryptosystem and then attempt to come up with attacks to demonstrate its security. Due to the similarity of our cryptosystems with linear code based cryptosystems, which are very well studied and believed to be secure, one good place to start would be to try and extend the well known attacks on the McEliece cryptosystem to our cryptosystem. One of the attacks we consider in this section is the well studied and powerful information set decoding (ISD) attacks on the McEliece cryptosystem. For the case of permutation codes, we have designed an algorithm which is very similar in spirit to the ISD attack and we study its effectiveness.

We now present some generic attacks on our cryptosystem. These attacks work regardless of the choice of the permutation code. Hence, their effectiveness only depend on public parameters like the correction capacity and size of the code. Later, we will demonstrate the existence of a code with parameters that make the cryptosystem secure against these attacks (Section  4.3).

4.2.2 Brute Force Attacks

One obvious method to solve the nearest neighbour problem in is to enumerate all elements of and calculate their hamming distance from the ciphertext. This means that must be large. Its exact size requirement would depend on the level of security we are looking for.

Another way is to enumerate all possible permutations within a distance of from the codeword and check if they belong to using the sifting procedure. The complexity of this attack would be , where and are the degree and correction capacity of the group respectively.

4.2.3 Information Set Decoding

One of the more successful attacks on the McEliece cryptosystem uses the technique of information set decoding. The original idea was proposed by Prange [21] in 1962 and over the years many improvements on this basic attack have been proposed [20, 11]. For an linear code, it involves picking coordinates at random and trying to reconstruct the codeword from those positions. It would be successful if the corresponding columns in the generator matrix form an invertible sub-matrix and no errors were induced in those coordinates. We can then recover the codeword from the ciphertext by solving a system of linear equations. We now recreate a version of ISD for permutation codes:

Input: generators of , A ciphertext
1 Procedure ISD():
2       Choose a base and SGS for at random
3       if There are no repeated symbols in positions indexed by  then
4             Use element reconstruction algorithm to try and find an element agreeing with on these positions.
5             if  exists and  then
6                  return
8      return False
Procedure repeated till it succeeds
Algorithm 3 Information Set Decoding

This algorithm is similar to the uncovering-by-bases decoding algorithm proposed by Bailey, except that here, we do not know what the UBB is and hence, the running time isn’t bounded by the size of the UBB. The correctness of this algorithm follows from the proof in Bailey [10, Proposition 7] in which he shows that given any -subset of , there exists a base for disjoint from it. Each iteration is a success if none of the positions of indexed by the chosen base have an error, which occurs with a finite non-zero probability because such a base exists. The performance of the ISD algorithm is identical for the groups and . This is because is a base for if and only if is a base for , and the algorithm picks a base at random ( is the secret conjugator)


The algorithm uses a random procedure to choose a base for the group. At the same time, a set of size is chosen uniformly at random and errors are induced in those positions. Before making concrete complexity estimates, we would like to make some observations. First, it is clear that if there are more error positions, the probability that the information set doesn’t have any errors is lower and hence the complexity of the attack increases with . Also, if the size of the base chosen is small, the probability that the positions indexed by this base doesn’t have any error positions is high. Hence the complexity of the attack increases with , the expectation of the length of the base.

Let denote the event that the algorithm chooses subset as a base for the group. Conditioned on the event , an iteration of the algorithm succeeds if the sets and are disjoint. Because the set is chosen uniformly at random from ,

As this probability depends only on the size of and not on the set itself, we can obtain an expression for the probability of an iteration of the algorithm succeeding:

Let be the size of the irredundant base of largest size of . As the decreases as the size of the base increases, we can obtain the following lower bound on the probability of the algorithm succeeding:

This is a Monte Carlo algorithm, which decodes correctly with probability

, and otherwise returns false. If we repeat the algorithm till it succeeds, we obtain a Las Vegas algorithm that always decodes correctly but has a running time which is a random variable with expectation

. It is also important to note that as each iteration is independent, this attack can be very effectively parallelized. Using the following well known formula for binomials,

we can show that this algorithm has expected complexity:


where is the binary entropy function and . The running time increases with both error rate and information rate.

The security level of a cryptosystem is measured using the amount of computational resources needed to break it. It is often expressed in terms of ‘bits’, where a cryptosystem that is -bit secure requires operations to be broken. For asymmetric cryptosystems, this security level is obtained using the running time of the best known attacks on them. Using the ISD, attack, we can now obtain the following necessary condition for our cryptosystem to have -bit security:


Hence, if we want a cryptosystem with a -bit security requirement, we need to choose a permutation code with the appropriate parameters. Most commercial applications today require a security level of or even bits in some cases.

So far, we have described a framework to develop cryptosystems using permutation codes and outlined generic attacks on those cryptosystems, whose effectiveness depend on the parameters of the chosen permutation code . Next, we attempt to use the permutation codes described in Section 2.2 in our cryptosystems.

4.3 Our cryptosystem using wreath product groups

The security and the performance of our cryptosystem depends on the choice of the group and its decoding algorithm. In this section, we explore using the wreath product groups with the alternate decoding algorithm discussed in Section  2.3 for this purpose. We show that, using the appropriate parameters, it can be made resistant to information set decoding attacks. The decoding algorithm can be modified to work in the conjugate group as well, which would make the cryptosystem using it insecure. Although this cryptosystem is insecure, it does provide a concrete example of a permutation code with an efficient decoding algorithm which cannot be efficiently decoded using the ISD algorithm.

Let be the wreath product group (more generally, we can take to be a subgroup of with suboptimal decoding). We consider the case of for different values of , for different probabilities of correct decoding. Because the actual minimal degree of this group is , there will be cases in which the receiver of the encrypted word cannot decode correctly and will receive a wrong word. In these cases, the receiver will be able to tell that the decoding is wrong (possibly by using an appended hash function) and ask the sender the resend the message. Consider the case for which the probability of correct decoding is and . We find the largest such value of for which errors are introduced and can be corrected for at least or fraction of the cases for different values of , and estimate the amount of computational resources required for an ISD based attack to break the cryptosystem by plugging in this value of into Equation 3.

Figure 3: Security levels for

As we can see, the computational resources needed to break this cryptosystem is non trivial and increases with , although it is still not close to commercial levels of security. We also repeat this exercise for the case of .

Figure 4: Security levels for

For , we can attain higher security levels for smaller values of compared with . These results mean that, by increasing and to an appropriate amount, we can reach commercial levels of security such as or bits. Our analysis of larger values of and were limited by our computational resources, as our computer program for calculating the proportion of error patterns that can be corrected involves iterating over integer partitions, which is computationally intensive. Judging by the trend in the graphs however, we can conclude that the values of and can be appropriately increased to attain larger levels of security.

4.3.1 Attack using block systems

For a permutation group acting on , a complete block system is a partition of into disjoint sets called blocks, such that if two points and are in the same block, then for all permutations , and are also in the same block. It can be shown that every block in have the same cardinality if is transitive.

The first property of block systems that we shall prove is that conjugation preserves the block structure of a permutation group.

Theorem 4.1.

Consider a permutation group with a block system . The conjugate permutation group has the block system .


Every permutation in is of the form where is a member of . Consider any two points and in the set .

As and are in the same block , and are in the same block of , which means that and are in the same clock of

For the wreath product group acting on a set of points it is easy to see that the columns form a maximal block system, with the group acting on the blocks and the group permuting them. Hence, the public key group has a corresponding block structure. First, we observe that the alternative decoding algorithm to be modified for a more generic type of group, with a block system such that the group acts regularly on each block.

Consider any permutation group acting on a set with a block system consisting of blocks of size each. It is well known that this permutation group can be described as a subgroup of the wreath product group . Every permutation is described by how it acts on each block and by how it permutes the blocks among themselves. In this case the permutation group also acts regularly on each block.

Let the conjugate of be . We compute a maximal block system for (or, if we use a subgroup of G, we can look for a block system with blocks with the required size). We then compute the generators for a subgroup of which stabilises this block system. This subgroup will act regularly on each block. We call these regular subgroups . An element of a regular subgroup is uniquely determined just by its action on one point. This element can also be computed using the Schreier-Sims algorithm.

Hence, for each position in , we compute the block labels and relabel the received word accordingly. For each position, we then compute the element of the corresponding regular group and the value of the block permutation. We then compute the most frequently occurring value of these elements in each block to obtain a group element of the form where and and convert it back to permutation form.

Hence, the same decoding algorithm can be deployed on the public key group, breaking the cryptosystem.

4.4 Our cryptosystem using the symmetric group acting on -subsets

We now examine the use of a group whose decoding algorithm will not work on its conjugates. Because of its parameters it can be trivially broken using ISD attacks.

Let be the permutation image symmetric group acting on the which is defined to consist of all the two sets of . Hence, and we choose to be a subgroup of . The conjugate is chosen such that is not a subgroup of . We use the decoding algorithm using the UBB described in Section  2.2.1. As is a subgroup of , the same decoding algorithm can be used, although the decoding might be suboptimal because the minimal degree of might be greater than that of . Unlike the case of the alternative decoding algorithm for the wreath product groups we cannot easily construct a UBB for the conjugate group using the same techniques used to construct one for . To recall the UBB is constructed using the hamiltonian cycles of a complete graph. The group , as a subgroup of , preserves the graph structure of . This is what that means. Consider two -sets and in . These two sets have one point in common. The -sets and must also have a point in common for all . No other conjugates of in have this graph preserving property and without that, it is unclear how to construct a UBB for without knowledge of the conjugator .

The next thing we attempt to do is to attack the second security assumption. That is, we attempt to find a conjugator and a subgroup of that has an efficient decoding algorithm. We assume that these subgroups are all subgroups of , as we can employ the UBB algorithm for them. While we do manage to come up with an attack which is better than a brute force search for , the group can still be chosen so that the cryptosystem is resistant to this attack.

4.4.1 Conjugator Search Attack

To recall, we are given a sequence of generators for the group . We attempt to construct a set of generators for a subgroup of and such that . We provide a sketch of the attack:

Consider any element of . Its cycle type in the permutation image acting on -subsets is entirely determined by its cycle type acting naturally. It is possible for two different cycle types acting naturally to lead to the same cycle type acting on -sets. Two elements which are conjugate in are also conjugate in , but two elements can be conjugate in but not in .

Consider the elements and . The crucial thing to note is that it is enough to find the cycle type of the pullback in . This is because any element of the same cycle type of is conjugate to in and hence also in , and we can conjugate all the other generators also by the same element to obtain a different subgroup of which would also have an efficient decoding algorithm.

Let us say we find an element of with the same cycle type as (cycle type in ). The conjugator is unique up to a coset of the centralizer of in . Among all the elements in this coset, we need to find a conjugator that takes each into . The only way to do that (to our knowledge) would be by an exhaustive search. The bottleneck in the algorithm is this step. Finding that conjugates an element to an element is not considered a difficult problem in practice and neither is computing the centralizer of an element as there are very effective backtracking methods to solve these problems [22, Section 9.1.2].

Complexity: The algorithm has to exhaustively search through the centralizer of one of the generators of , and as we can choose the generator we start with, we can pick the generator with the smallest centralizer. Hence, for the cryptosystem to be resistant to conjugator search attack, we would need to pick the subgroup of such that its generators have large centralizers in .

4.4.2 ISD attacks

The size of a base for is less than . Hence, the number of bit operations required to break the cryptosystem using information set decoding is less than:

We plotted this as a function of (Fig. 5) and we can see that the security level of even bits isn’t attainable for reasonable values of .

The key reason why the cryptosystem using this code is insecure is the parameters of the code. Both the rate of the code and the correction capacity are too low. What we are aiming for is a code where the quantities (on average) and are asymptotically constant, like the binary Goppa codes. This requirement is satisfied by the wreath product groups using the alternative probabilistic decoding algorithm.

Figure 5: Security of cryptosystem using -sets

4.5 Linear codes as permutation codes

Linear codes are closely related to permutation codes. A linear code of block size over the field is an additive subgroup of and a permutation code of block size is a subgroup of . A linear code with minimum distance which is a subgroup of can be embedded naturally into as a permutation code with minimal degree . This embedding is a distance expanding map. That is, a vector with weight maps to a permutation with weight . For the case of binary linear codes, the -th basis vector of is mapped to the permutation , and we can extend this to a homomorphism from to . For the -ary case, the additive group has a regular permutation homomorphism into . The vector , where is mapped to . Similarly, the vector is mapped to , where is the permutation formed by a single cyclic shift. maps to a permutation acting on the first points, maps to a permutation acting on the next points, etc. This would extend to a homomorphism from into .

Hence, classical coding theory provides us with a rich source of permutation codes. However, for the purposes of our cryptosystems, permutation codes do not suffice. Rather, we need permutation codes with efficient decoding algorithms. The natural question to ask is- Do the classical linear code decoders effectively translate to the permutation code setting?

Consider a linear code and . Once we introduce errors to to form a word , the error pattern is . For a permutation code , once we introduce errors to to form a word , the error pattern is . We abuse notation slightly as this need not be a permutation.

Let us assume the code has an efficient decoding algorithm . We create a permutation code from this, and introduce an error pattern to . Now, if is also a member of , then, we can employ the inverse of the described homomorphism to map this back to , and employ to decode. However, the error pattern need not be in , it just needs to have support of size less than . Hence the linear code decoding algorithms cannot always be used for the corresponding permutation code, and designing effective algorithms in that setting seems to be a hard problem. From the standpoint of cryptography, this problem can be worked around. This is because the person who sends the message introduces the errors and the error pattern is in their hands unlike in the usual coding theory applications! Hence, in our protocol, we can use a linear code as a permutation code, and when the message is sent with an error, there is an additional restriction that the error pattern belongs to . Hence, our framework is a generalisation of traditional code-based cryptosystems.

Most linear code decoders rely extensively on syndrome decoding and the parity check matrix, which makes it difficult to extend them to permutation codes. A corresponding concept of a parity check matrix for permutation groups does not seem to exist, which is a major inconvenience to designing potential permutation code decoders, even for permutation codes designed directly from linear codes.

4.6 Why use permutation codes?

So far, we have described a framework for using permutation codes in public key cryptography and explored this framework using two particular permutation codes. The cryptosystem using wreath product groups can be made resistant to ISD attacks but its decoding algorithm can be adapted for use in any of its conjugates too, which makes it unsuitable. For the symmetric group acting on -sets on the other hand, the decoding algorithm cannot be be modified for use in its conjugates, nor can the conjugator be uncovered easily from and . This is because the decoding algorithm depends on some very specific graph theoretical properties of the complete graph, and the conjugates of cannot be modelled as the symmetric group acting on a complete graph. The parameters of this code mean that any cryptosystem using this is susceptible to ISD attacks. The question is, can we come up with a permutation code with parameters that make it resistant to ISD attacks and a decoding algorithm that cannot be modified for its conjugates? Our cryptosystem using such a permutation code would be secure against both the kind of attacks demonstrated earlier.

Permutation groups differ from vector spaces in their non-commutativity, and it would be an interesting question to see if this would lead to any significant improvements in key size, speed, etc over traditional code based cryptosystems. For example, a rank subspace of needs basis vectors to describe, each of length . On the contrary, very large permutation groups can be generated by a very small number of generators. The symmetric group itself for example, can be generated using just two of its elements. This is a characteristic shared by many non-abelian permutation groups. Hence, a cryptosystem using permutation codes can potentially achieve a quadratic reduction in key size compared to its linear code counterparts for the same level of security! A linear code based cryptosystem using a code of length over would require a key of as one of the components of the public key is a matrix. By contrast, consider a permutation group of comparable size which is a subgroup of which can be generated using just two generators. The space needed to store these generators would be just . One of the common complaints against the McEliece cryptosystem is its large key size so this would be a direction of research worth pursuing.


This paper was part of the first author’s Master’s thesis at IISER Pune. The first author was partially supported by an INSPIRE fellowship. The second author was partially supported by a NBHM research grant. Both authors thank Upendra Kapshikar for stimulating discussions on the topic.


  • [1] M. Ajtai. Generating hard instances of lattice problems. In

    Proceedings of the twenty-eighth annual ACM symposium on Theory of computing

    , pages 99–108, 1996.
  • [2] M. Ajtai and C. Dwork. A public-key cryptosystem with worst-case/average-case equivalence. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing, pages 284–293, 1997.
  • [3] G. Alagic, G. Alagic, J. Alperin-Sheriff, D. Apon, D. Cooper, Q. Dang, Y.-K. Liu, C. Miller, D. Moody, R. Peralta, et al. Status report on the first round of the NIST post-quantum cryptography standardization process. US Department of Commerce, National Institute of Standards and Technology, 2019.
  • [4] G. Alagic, J. Alperin-Sheriff, D. Apon, D. Cooper, Q. Dang, J. Kelsey, Y.-K. Liu, C. Miller, D. Moody, R. Peralta, et al. Status report on the second round of the NIST post-quantum cryptography standardization process. US Department of Commerce, National Institute of Standards and Technology, 2020.
  • [5] B. Alspach. The wonderful Walecki construction. Bulletin of the Institute of Combinatorics and its Applications, 52, 01 2008.
  • [6] S. Arora and B. Barak. Computational complexity: a modern approach. Cambridge University Press, 2009.
  • [7] V. Arvind and P. S. Joglekar. Algorithmic problems for metrics on permutation groups. In International Conference on Current Trends in Theory and Practice of Computer Science, pages 136–147. Springer, 2008.
  • [8] R. F. Bailey. Permutation groups, error-correcting codes and uncoverings. PhD thesis, University of London, 2006.
  • [9] R. F. Bailey. Uncoverings-by-bases for base-transitive permutation groups. Designs, Codes and Cryptography, 2006.
  • [10] R. F. Bailey. Error-correcting codes from permutation groups. Discrete Mathematics, 309:4253–4265, 2009.
  • [11] D. J. Bernstein, T. Lange, and C. Peters. Attacking and defending the McEliece cryptosystem. In International Workshop on Post-Quantum Cryptography, pages 31–46. Springer, 2008.
  • [12] I. Blake. Permutation codes for discrete channels. IEEE Transactions on Information Theory, 1974.
  • [13] I. F. Blake, G. Cohen, and M. Deza. Coding with permutations. Information and Control, 43(1):1–19, 1979.
  • [14] C. Buchheim, P. J. Cameron, and T. Wu. On the subgroup distance problem. Discrete mathematics, 309(4):962–968, 2009.
  • [15] M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W. H. Freeman & Co., USA, 1979.
  • [16] I. Janiszczak and R. Staszewski. Isometry invariant permutation codes and mutually orthogonal latin squares. Journal of Combinatorial Design, 2019.
  • [17] J. Macwilliams. Permutation decoding of systematic codes. The Bell System Technical Journal, 43(1):485–505, 1964.
  • [18] R. J. McEliece. A public-key cryptosystem based on algebraic coding theory. Coding Thv, 4244:114–116, 1978.
  • [19] H. Niederreiter. Knapsack-type cryptosystems and algebraic coding theory. Prob. Control and Inf. Theory, 15(2):159–166, 1986.
  • [20] C. Peters. Information-set decoding for linear codes over . In International Workshop on Post-Quantum Cryptography, pages 81–94. Springer, 2010.
  • [21] E. Prange. The use of information sets in decoding cyclic codes. IRE Transactions on Information Theory, 8(5):5–9, 1962.
  • [22] Àkos. Seress. Permutation group algorithms. Cambridge tracts in Mathematics. Cambridge university press, 2002.