The Hidden Subgroup Problem and Post-quantum Group-based Cryptography

by   Kelsey Horan, et al.
CUNY Law School
NYU college

In this paper we discuss the Hidden Subgroup Problem (HSP) in relation to post-quantum group-based cryptography. We review the relationship between HSP and other computational problems discuss an optimal solution method, and review the known results about the quantum complexity of HSP. We also overview some platforms for group-based cryptosystems. Notably, efficient algorithms for solving HSP in such infinite group platforms are not yet known.



There are no comments yet.


page 1

page 2

page 3

page 4


Post-Quantum Secure Cryptographic Algorithms

An overview on current developments in post quantum cryptography...

Learning With Errors and Extrapolated Dihedral Cosets

The hardness of the learning with errors (LWE) problem is one of the mos...

Quantum Learning Algorithms and Post-Quantum Cryptography

Quantum algorithms have demonstrated promising speed-ups over classical ...

General Linear Group Action on Tensors: A Candidate for Post-Quantum Cryptography

Starting from the one-way group action framework of Brassard and Yung (C...

Pre- and post-quantum Diffie-Hellman from groups, actions, and isogenies

Diffie-Hellman key exchange is at the foundations of public-key cryptogr...

Multilinear Cryptography using Nilpotent Groups

In this paper we develop a novel idea of multilinear cryptosystem using ...

The Asymptotic Complexity of Coded-BKW with Sieving Using Increasing Reduction Factors

The Learning with Errors problem (LWE) is one of the main candidates for...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In August 2015 the National Security Agency (NSA) announced plans to upgrade security standards; the goal is to replace all deployed cryptographic protocols with quantum secure protocols. This transition requires a new security standard to be accepted by the National Institute of Standards and Technology (NIST). Proposals for quantum secure cryptosystems and protocols have been submitted for the standardization process. There are six main primitives currently proposed to be quantum-safe: (1) lattice-based (2) code-based (3) isogeny-based (4) multivariate-based (5) hash-based, and (6) group-based cryptographic schemes.

One goal of cryptography, as it relates to complexity theory, is to analyze the complexity assumptions used as the basis for various cryptographic protocols and schemes. A central question is determining how to generate intractible instances of these problems upon which to implement an actual cryptographic scheme. The candidates for these instances must be platforms in which the hardness assumption is still reasonable. Determining if these group-based cryptographic schemes are quantum-safe begins with determining the groups in which these hardness assumptions are invalid in the quantum setting.

In what follows we address the quantum complexity of the Hidden Subgroup Problem (HSP) to determine the groups in which the hardness assumption still stands. The Hidden Subgroup Problem (HSP) asks the following: given a description of a group and a function for some finite set is guaranteed to be strictly -periodic, i.e. constant and distinct on left (resp. right) cosets of a subgroup , find a generating set for .

It is important to note that Simon’s problem of computing a XOR-mask, Shor’s algorithm for factoring and finding the discrete log, Boneh’s algorithm for finding a hidden linear function, and Kitaev’s algorithm for the abelian stabilizer problem are all special cases of HSP. Therefore, the HSP is directly related to problems such as breaking one-time pad, discrete logarithm problem, graph isomorphism problem (which is now known to be in quali-polynomial), lattice-based problems, the problem for factoring for RSA.

The classical complexity of HSP is known [1]: Suppose that has a set of subgroups, such that . Then a classical computer must make queries to solve the HSP. The classical cases in which HSP is easy are the cases in which has only a polynomial number of subgroups, allowing brute-force for the function on all subgroups.

We provide a survey of results regarding the complexity of quantum algorithms for solving HSP in various group platforms. We also provide information on the relationship between HSP and other computational problems. These results provide insight into potential platforms for quantum safe cryptography, when the underlying hard problem is reducible to HSP.

2 Group-based Cryptography

Group-based cryptography could be shown to be post-quantum if the underlying security problem is NP-complete or unsolvable; firstly, we need to analyze the problem’s equivalence to HSP, then analyze the applicability of Grover’s search problem. Cryptanalysis based on a reduction to solving HSP creates some obstacles as the groups under consideration below are mostly infinite and do not have an efficient algorithm for HSP. In the following cryptosystems a connection to HSP can assist in the analysis of security.

For example in [2] a practical cryptanalysis of WalnutDSA was proposed, a post-quantum cryptosystem using the conjugacy search problem (CSP) over braid groups that was submitted to the NIST competition in 2017  [3]. It has been argued since the braid group does not contain any non-trivial finite subgroups, there does not seem to be any viable way to connect CSP with HSP. It has been shown there is no reduction connection between the CSP and HSP, [4, 5]. As for analysis via Grover’s algorithm [6], it has been mentioned that a majority of the time for signature verification in WalnutDSA is repeated E-Multiplications.

There are alternative group-theoretic problems and classes of groups which have been proposed for post-quantum cryptography. For example, the first proto-cryptosystem based on groups was proposed by Wagner-Magyarik in [7] for which the word choice problem was hard. Later on Flores-Kahrobaei-Koberda proposed right-angled Artin groups for various other cryptographic protocols [8], [9]. Eick and Kahrobaei proposed Polycyclic groups, using the Conjugacy Search Problem [10] for cryptography. Later on Gryak-Kahrobaei wrote a survey and proposed other group-theoretic problems for consideration using polycyclic groups [11]. Kahrobaei-Koupparis [12] proposed a post-quantum digital signature using polycyclic groups. Kahrobaei-Khan proposed a public-key cryptosystem using polycyclic groups [13]. Habeeb-Kahrobaei-Koupparis-Shpilrain proposed the use of a semigroup of matrices with a semidirect product structure [14].

Thompson groups have been considered by Shpilrain-Ushakov based on the Decomposition Search Problem [15]. Hyperbolic groups have been proposed by Chatterji-Kahrobaei-Lu using properties of subgroup distortion and the Geodesic Length Problem [16]. Free metabelian groups have been proposed based on the Subgroup Membership Search Problem by Shpilrain-Zapata [17]. Kahrobaei-Shpilirain proposed Free nilpotent p-groups for a semidirect product public key cryptosystem [18]. Linear groups were proposed by Baumslag-Fine-Xu [19]. Grigorchuk groups, have been proposed by [20]. Groups of matrices, for a Homomorphic Encryption scheme were proposed by Grigoriev-Ponomarenko [21].

3 Relation of HSP to Other Computational Problems

Many computational problems are special cases of the HSP; an efficient algorithm for HSP over a certain group implies an efficient algorithm for some other computational problem. It is important to note that one method of determining an efficient quantum solution to a hard problem consists of reducing the problem to an instance of HSP over a group with a known efficient solution. This consists of determining the appropriate group , the subgroup and the strongly -periodic function . For example, Simon’s problem can be viewed as an instantiation of HSP over with a subgroup of order . Duetsch’s algorithm solves a variant of HSP where is either (f is balanced) or (f is constant). Shor’s algorithm solves period finding as a special case of HSP, allowing for an efficient quantum algorithm for factoring and discrete log.

The graph automorphism (resp. isomorphism) problem can also be framed as an instance of HSP. To solve graph automorphism we consider HSP in the symmetric group on letters, , any function which hides the trivial subgroup is an automorphism. Analogously the graph isomorphism problem is an instance of HSP over the wreath product [22]. Also, solutions to HSP can solve the abelian stabilizer problem; when is acting on a finite set and where is the stabilizer of we have that can be defined such that is strongly -periodic.

A solution to a particular instance of HSP is a solution to the hidden linear functions [23]; if is a permutation of and is such that , we have hiding . Additionally, self-shift-equivalent polynomials can be framed as an instance of HSP, in this case Grigoriev shows how to compute the hidden subgroup [24].

An efficient solution to HSP would imply an efficient solution to certain lattice problems. Specifically, the

-Unique Shortest Vector Problem (USVP) is NP-hard for

, and has a polynomial time classical solution when is large. The dihedral HSP, based on standard-method (found below) can be used to solve poly()-USVP [25]. HSP over the symmetric and dihedral groups are highly motivated open questions in post-quantum group-based cryptography.

Another, related, computational problem is the Hidden Shift Problem, which has been proposed as a basis for post-quantum cryptography in symmetric cryptosystems that are quantum-CPA secure [26]. Other than the use of a generalization of Simon’s algorithm, and Kuperberg’s algorithm discussed above, very little is known about the Hidden Shift Problem. Clearly this problem is closely related to HSP as some solutions coincide. It is important to note that constructions baed on this Hidden Shift problem have also remained quantum secure.

4 Solution Methods

The standard method of solving HSP over performs the following steps. First, the algorithm queries the -periodic function in superposition and discards the register which holds the output. This leaves the first register entangled in a hidden subgroup state, a superposition of coset representatives for some left traversal . Following this, the state can be sampled using post-processing techniques to determine . In the following we have as the coset state. This approach reduces the problem to a problem of quantum mechanics: how to distinguish the members of an ensemble of quantum hidden subgroup states.

How do we measure the state? The problem of distinguishing these quantum states has some proposed solutions. Most namely, the often optimal solution entitled Pretty Good Measurement (PGM) can be used. An obstacle to performing PGM is the lack of an efficient QTF/CFT in the underlying group. For these instances we know of no efficient quantum algorithm for solving HSP.

5 Results

Finite Abelian and Finite Near-Abelian.

The infamous quantum algorithms of Simon and Shor provide quantum solutions to HSP in the abelian cases where and respectively. Shor’s algorithm extends to the general abelian case as well, providing a polynomial time quantum algorithms with bounded error [27, 28, 29]

. The probability of success can be improved to

when is abelian of smooth order, i.e. if all prime factors of are at most [30].

In the case that is nearly abelian, i.e. if the value where is the normalizer for is sufficiently large there are established computational bounds on HSP. The size of this intersection relative to the group is a measure of the abelianness of . Gavinsky [31] gave results to show that an efficient algorithm exists when .

The HSP can be solved in polynomial time by a quantum algorithm to find hidden normal subgroups of solvable groups and permutation groups, finding hidden subgroups of groups with small commutator subgroup and of groups admitting an elementary Abelian normal 2-subgroup of small index or with cyclic factor group [32]. Subexponential algorithms for HSP in any solvable group have been given by Friedl et al. [33].

When is a known finite abelian group with a subgroup , given black-box access to the -hiding function , we know that a quantum computer can uniquely and completely determine in time and query complexity. When is nearly abelian, or built from abelian parts, one can leverage this fact to obtain an efficient algorithm for HSP.

Finite, Non-Abelian.

The finite non-abelian case of HSP is more elusive. Shor’s algorithm extends to any group when

is normal if quantum fourier transform (QFT) can be efficiently computed over the group

[34]. The algorithm also extends to when has few conjugates, requiring the quantum character transform (QCT) over the group algebra [35]. This variation is not applicable when has many conjugates, as in some of the following cases. Alternatively, when is normal in , a black-box group, generators for can be found in time polynomial in the input size + [32] without requiring an efficient QFT over .Additionally, the quantum computation of the discrete log in semi-groups [36] is an instance of HSP.

When the group is the (discrete) Heisenberg group it is sufficient to be able to distinguish cyclic subgroups of order , . Thus, finding an arbitrary reduces to determining two parameters , given the coset state produced by the standard method using which hides . This can be efficiently computed with an overall success probability close to .

In a more difficult case we consider instances of HSP in the dihedral group of order , where the function hides of order . In this case , a hidden reflection, has many conjugates in and the QCT based solution is not applicable. Kuperberg stated that finding an arbitrary hidden subgroup of reduces to finding the slope of a hidden reflection and provides a quantum algorithm with both time and query complexity of , applicable to for all values of but achieving an even tighter complexity bound for specific smooth values of [37] . Kuperberg’s algorithm also provides a solution to the hidden shift problem in an arbitrary finitely generated abelian group . Regev improved upon the bounds of Kuperberg’s original algorithm providing a polynomial space variation to the original superpolynomial space algorithm, which still achieves subexponential complexity [38]. Regev showed that an efficient solution to the dihedral HSP implies a quantum solution to lattice problems [25].

When is a type of wreath product , Roetteler et al. [39] provide a positive result for finding an efficient solution to the non-abelian HSP within . This result is due to the existence of an efficient non-abelian QFT in . Wreath product groups are in turn a subset of semi-direct product groups. When is (one of some groups that are) a semidirect product of abelian groups, alternative efficient algorithms have been proposed. The polycyclic HSP has been addressed for for fixed prime power [40], with and , certain affine groups [41], [42], with , where where and [43], and where has a special prime factorization [44].

In general, when is a group of finite order the HSP problem has quantum query complexity of as shown by Ettinger, Høyer and Knill [45]; for any group , queries provides sufficient statistical information to solve HSP. This result provides no guarantees on computational complexity. The real problem is determining how to implement the queries efficiently, as well as how to control the amount of postprocessing required by the algorithm. In the case of the dihedral group, an algorithm with the lower bound on query complexity has been constructed, but the postprocessing required is exponential. In many cases the inefficiency of a proposed quantum algorithm is primarily due to the inefficiency of the required quantum measurement or post-processing within the group.


What seems to be an obstacle for infinite groups is that the quantum computer should assume the following state: The meaning of this is clear for finite groups. The abelian infinite HSP was clearly first considered with Shor’s algorithm, over . In [46], infinite-dimensional HSP has been mentioned, particularly for infinite abelian groups . Additionally, HSP has been defined and considered for infinite abelian groups of the form for some finite group [47]. Other than the cases of , , , and combinations of these in which an efficient algorithm exists, the infinite and continuous HSP has not been addressed within the literature for the non-abelian case.


  • [1] A. Childs. Lecture notes on quantum algorithms. 2017.
  • [2] D. Hart, D. Kim, G. Micheli, G. Pascual-Perez, C. Petit, and Y. Quek. A practical cryptanalysis of walnutdsa. In IACR International Workshop on Public Key Cryptography, pages 381–406. Springer, 2018.
  • [3] I. Anshel, D. Atkins, D. Goldfeld, and P. Gunnells. Walnutdsa (tm): A quantum resistant group theoretic digital signature algorithm. IACR Cryptology ePrint Archive, 2017.
  • [4] L. Wang and L. Wang. Conjugate searching problem vs. hidden subgroup problem. The Third International Workshop on Post-Quantum Cryptography, Recent Results Session, 2010.
  • [5] L. Wang, L. Wang, Z. Cao, Y. Yang, and X. Niu. Conjugate adjoining problem in braid groups and new design of braid-based signatures. Science China Information Sciences, 53(3):524–536, 2010.
  • [6] L. Grover. A fast quantum mechanical algorithm for database search. In

    Proceedings of the twenty-eighth annual ACM Symposium on Theory of Computing

    , pages 212–219, 1996.
  • [7] N. Wagner and M. Magyarik. A public-key cryptosystem based on the word problem. In Workshop on the Theory and Application of Cryptographic Techniques, pages 19–36. Springer, 1984.
  • [8] R. Flores and D. Kahrobaei. Cryptography with right-angled artin groups. Theoretical and Applied Informatics, 28:8–16, 2016.
  • [9] R. Flores, D. Kahrobaei, and T. Koberda. Algorithmic problems in right-angled artin groups: complexity and applications. arXiv preprint arXiv:1802.04870, 2018.
  • [10] B. Eick and D. Kahrobaei. Polycyclic groups: A new platform for cryptology? arXiv preprint math/0411077, 2004.
  • [11] J. Gryak and D. Kahrobaei. The status of polycyclic group-based cryptography: A survey and open problems. Groups Complexity Cryptology, 8(2):171–186, 2016.
  • [12] D. Kahrobaei and C. Koupparis. on-commutative digital signatures using non-commutative groups. Groups Complexity Cryptology, pages 377–384, 2012.
  • [13] D. Kahrobaei and B. Khan. A non-commutative generalization of elgamal key exchange using polycyclic groups. In IEEE Global Telecommunications Conference, 2006, pages 1–5, 2006.
  • [14] M. Habeeb, D. Kahrobaei, C. Koupparis, and V. Shpilrain. Public key exchange using semidirect product of (semi) groups. In International Conference on Applied Cryptography and Network Security. Springer, 2013.
  • [15] V. Shpilrain and A. Ushakov. Thompson’s group and public key cryptography. In International Conference on Applied Cryptography and Network Security, pages 151–163. Springer, 2005.
  • [16] I. Chatterji, D. Kahrobaei, and N. Y. Lu. Cryptosystems using subgroup distortion. Theoretical and Applied Informatics, 29:14–24, 2017.
  • [17] V. Shpilrain and G. Zapata. Combinatorial group theory and public key cryptography. Applicable Algebra in Engineering, Communication and Computing, 17(3-4):291–302, 2006.
  • [18] D. Kahrobaei and V. Shpilrain. Using semidirect product of (semi) groups in public key cryptography. In Conference on Computability in Europe, Pursuit of the Universal, LNCS, pages 132–141. Springer, 2016.
  • [19] G. Baumslag, B. Fine, and X. Xu. Cryptosystems using linear groups. Applicable Algebra in Engineering, Communication and Computing, 17(3-4):205–217, 2006.
  • [20] G. Petrides. Cryptanalysis of the public key cryptosystem based on the word problem on the grigorchuk groups. In IMA International Conference on Cryptography and Coding, pages 234–244. Springer, 2003.
  • [21] D. Grigoriev and I. Ponomarenko. Homomorphic public-key cryptosystems over groups and rings. arXiv preprint cs/0309010, 2003.
  • [22] J. Kobler, U. Schöning, and J. Torán. The graph isomorphism problem: its structural complexity. Springer Science & Business Media, 2012.
  • [23] D. Boneh and R. Lipton. Quantum cryptanalysis of hidden linear functions. In Annual International Cryptology Conference, pages 424–437, 1995.
  • [24] D. Grigoriev. Testing shift-equivalence of polynomials by deterministic, probabilistic and quantum machines. Theoretical Computer Science, 180(1-2):217–228, 1997.
  • [25] O. Regev. Quantum computation and lattice problems. SIAM Journal on Computing, 33(3):738–760, 2004.
  • [26] G. Alagic and A. Russell. Quantum-secure symmetric-key cryptography based on hidden shifts. In Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 65–93. Springer, 2017.
  • [27] Daniel R Simon. On the power of quantum computation. SIAM Journal on Computing, 26(5):1474–1483, 1997.
  • [28] P. Shor. Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer. SIAM Review, 41(2):303–332, 1999.
  • [29] A. Kitaev. Quantum computations: algorithms and error correction. Russian Mathematical Surveys, 52(6):1191–1249, 1997.
  • [30] G. Brassard and P. Hoyer. An exact quantum polynomial-time algorithm for simon’s problem. In Proceedings of the Fifth Israeli Symposium on Theory of Computing and Systems, pages 12–23. IEEE, 1997.
  • [31] D. Gavinsky. Quantum solution to the hidden subgroup problem for poly-near-hamiltonian groups. Quantum Information & Computation, 4(3):229–235, 2004.
  • [32] G. Ivanyos, F. Magniez, and M. Santha. Efficient quantum algorithms for some instances of the non-abelian hidden subgroup problem. International Journal of Foundations of Computer Science, 14(05):723–739, 2003.
  • [33] K. Friedl, G. Ivanyos, F. Magniez, M. Santha, and P. Sen. Hidden translation and translating coset in quantum computing. SIAM Journal on Computing, 43(1):1–24, 2014.
  • [34] Sean Hallgren, Alexander Russell, and Amnon Ta-Shma. Normal subgroup reconstruction and quantum computation using group representations. In Proceedings of the thirty-second annual ACM Symposium on Theory of computing, pages 627–635, 2000.
  • [35] M. Grigni, L. Schulman, M. Vazirani, and U. Vazirani. Quantum mechanical algorithms for the nonabelian hidden subgroup problem. In Proceedings of the thirty-third annual ACM Symposium on Theory of Computing, pages 68–74, 2001.
  • [36] A. Childs and G. Ivanyos. Quantum computation of discrete logarithms in semigroups. Journal of Mathematical Cryptology, 8(4):405–416, 2014.
  • [37] G. Kuperberg. A subexponential-time quantum algorithm for the dihedral hidden subgroup problem. SIAM Journal on Computing, 35(1):170–188, 2005.
  • [38] O. Regev. A subexponential time algorithm for the dihedral hidden subgroup problem with polynomial space. arXiv preprint quant-ph/0406151, 2004.
  • [39] M. Roetteler and T. Beth. Polynomial-time solution to the hidden subgroup problem for a class of non-abelian groups. arXiv preprint quant-ph/9812070, 1998.
  • [40] K. Friedl, G. Ivanyos, F. Magniez, M. Santha, and P. Sen. Hidden translation and orbit coset in quantum computing. In Proceedings of the thirty-fifth annual ACM Symposium on Theory of computing, pages 1–9, 2003.
  • [41] C. Moore, D. Rockmore, A. Russell, and L. Schulman. The power of basis selection in fourier sampling: Hidden subgroup problems in affine groups. In Proceedings of the fifteenth annual ACM-SIAM Symposium on Discrete algorithms, pages 1113–1122, 2004.
  • [42] Yoshifumi Inui and François Le Gall. An efficient algorithm for the hidden subgroup problem over a class of semi-direct product groups. Technical report, 2004.
  • [43] D. Gonçalves and R. Portugal. Solution to the hidden subgroup problem for a class of noncommutative groups. arXiv preprint arXiv:1104.1361, 2011.
  • [44] D. Gonçalves, T. Fernandes, and C. Cosme. An efficient quantum algorithm for the hidden subgroup problem over some non-abelian groups. TEMA (São Carlos), 18(2):215–223, 2017.
  • [45] M. Ettinger, P. Høyer, and E. Knill. The quantum query complexity of the hidden subgroup problem is polynomial. Information Processing Letters, 91(1):43–48, 2004.
  • [46] Kissinger A. Gogioso, S. Fully graphical treatment of the quantum algorithm for the hidden subgroup problem. arXiv preprint quant-ph 1701.08669, 2017.
  • [47] K. Eisenträger, S. Hallgren, A. Kitaev, and F. Song. A quantum algorithm for computing the unit group of an arbitrary degree number field. In Proceedings of the forty-sixth annual ACM Symposium on Theory of computing, pages 293–302, 2014.