 # Towards Efficient Normalizers of Primitive Groups

We present the ideas behind an algorithm to compute normalizers of primitive groups with non-regular socle in polynomial time. We highlight a concept we developed called permutation morphisms and present timings for a partial implementation of our algorithm. This article is a collection of results from the author's PhD thesis.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

One of the tools to study the internal structure of groups is the normalizer. For two groups and , which are contained in a common overgroup , we call the normalizer of in , denoted , the subgroup of consisting of those elements that leave invariant under conjugation.

We only consider finite sets, finite groups, and permutation groups acting on finite sets. We assume permutation groups to always be given by generating sets and say that a problem for permutation groups can be solved in polynomial time, if there exists an algorithm which, given permutation groups of degree , solves it in time bounded polynomially in and in the sizes of the given generating sets. While many problems for permutation groups can be solved efficiently both in theory and in practice, no polynomial time algorithm to compute normalizers of permutation groups is known.

A transitive permutation group acting on a set is called primitive if there exists no non-trivial -invariant partition of . Primitive groups have a rich and well-understood structure. Hence many algorithms use the natural recursion from general permutation groups to transitive and in turn to primitive ones. For two permutation groups computing the normalizer of in in general is done by searching for the normalizer of in the symmetric group and simultaneously computing the intersection with . We focus on computing the normalizer of a primitive group in . Being able to compute normalizers for primitive groups efficiently may lead to improved algorithms for more general situations.

Our results build substantially on the O’Nan-Scott classification of primitive groups, see [liebeck-praeger-saxl:onan-scott], and on the classification of finite simple groups (CFSG).

Recall that the socle of a group , denoted , is the subgroup generated by all minimal normal subgroups of . Our theoretical main result is the following theorem.

###### Theorem 1.1 ([siccha:phd-thesis, Theorem 9.1]).

Let a primitive group with non-regular socle111This excludes groups of affine and of twisted wreath type. be given. Then we can compute in polynomial time.

As is often the case in computational group theory, ideas from theoretical algorithms can be employed in practical algorithms and vice versa. While the algorithms in [siccha:phd-thesis] are primarily theoretical ones, we also provide probabilistic nearly-linear time versions where possible. The author is developing the GAP package NormalizersOfPrimitiveGroups, hosted at

with the aim to implement practical versions of the algorithms developed in [siccha:phd-thesis]. Until now, algorithms concerning permutation morphisms and primitive groups of type PA are implemented. First experiments indicate that already for moderate degrees these outperform the GAP built-in algorithm Normalizer by several orders of magnitude, see Table 1.

Since no polynomial time solutions are known for the normalizer problem, the generic practical algorithms resolve to backtracking over the involved groups in one way or another. The fundamental framework of modern backtrack algorithms for permutation groups is Leon’s partition backtrack algorithm [leon], which generalizes previous backtrack approaches [butler:backtrack, butler:normalizers, holt:normalizers, sims:backtrack] and generalizes ideas of nauty [mckay-piperno:nauty-traces] to the permutation group setting. Partition backtrack is implemented in GAP [gap] and Magma [magma]. Recently, the partition backtrack approach was generalized to a “graph backtrack” framework [JPWW].

Theißen developed a normalizer algorithm which uses orbital graphs to prune the backtrack search [theissen]. Chang is currently developing specialized algorithms for highly intransitive permutation groups, her PhD thesis should appear shortly. It is to expect that the work in [JPWW] can also be extended to normalizer problems. Hulpke also implemented normalizer algorithms in [hulpke:normalizers] using group automorphisms and the GAP function NormalizerViaRadical based on [glasby-slattery:normalizers]

In Section 2 we outline the strategy behind our algorithms. In Section 3 we recall the O’Nan-Scott Theorem. We present our new concept of permutation morphisms in Section 4. In Section 5 we sketch how we use our results to obtain Theorem 1.1. In Section 6 we discuss our implementation.

## 2 Strategy

We describe the strategy of the theoretical algorithm behind Theorem 1.1. Comments regarding the implementation of its building blocks are given at the end of each following section.

In this section let be a primitive group with non-regular socle . The normalizer of in plays a central role in our algorithm, in this section we denote it by . Observe that to compute it suffices to compute since the former is contained in .

The socle is isomorphic to for some finite non-abelian simple group and some positive integer . The group is isomorphic to a subgroup of the wreath product , see Section 3 for a definition of wreath products. By the O’Nan-Scott Theorem the respective isomorphism extends to an embedding333For twisted wreath type the situation is slightly more complicated. of the normalizer into . Furthermore is of the order . Hence the index of in , and thus also the search-space of the normalizer computation , is tiny in comparison to the index of in .

Our approach can be divided into two phases. First we compute , this is by far the most labor intensive part. To this end we compute a sufficiently well-behaved conjugate of , such that we can exhibit the wreath structure mentioned above. In [siccha:phd-thesis] we make this more precise and define a weak canonical form for primitive groups. Using that conjugate and the O’Nan-Scott Theorem we can write down generators for . In the second phase, we compute a reduction homomorphism with . After this logarithmic reduction, we use Daniel Wiebking’s simply exponential time algorithm [wiebking:normalizers-simply-exponential, wiebking:normalizers-simply-exponential-siam], which is based on the canonization framework [schweitzer-wiebking:canonisation-framework], to compute . Note that the running time of a simply exponential time algorithm called on a problem of size is and thus is bounded by for some constant . Then we use Babai’s famous quasipolynomial time algorithm for graph-isomorphism [babai:graphiso-quasipolynomial, babai:graphiso-quasipolynomial-acm] to compute the group intersection . Notice that since we perform these algorithms on at most points they run in time polynomial in . The homomorphism is constructed in such a way, that computing the preimage of the above normalizer yields . Recall that is equal to .

In our implementation we do not use the algorithms by Wiebking and Babai since these are purely theoretical. Instead we use the partition backtrack implemented in GAP.

## 3 The O’Nan-Scott Theorem

The goal of this and the next section is to illustrate how we use the O’Nan-Scott Theorem to prove the following theorem. In this article we limit ourselves to groups of type PA, which we define shortly.

###### Theorem 3.1 ([siccha:phd-thesis, Theorem 8.1]).

Let a primitive group with non-abelian socle be given. Then we can compute in polynomial time.

###### Proof.

For groups of type PA this will follow from Corollary 3.4 and Lemma 4.1. ∎

The O’Nan-Scott Theorem classifies how the socles of primitive groups can act, classifies the normalizers of the socles, and determines criteria to decide which subgroups of these normalizers act primitively. We follow the division of primitive groups into eight O’Nan-Scott types as it was suggested by László G. Kovács and first defined by Cheryl Praeger in

[praeger:onan-scott-eight-types]. In this section we define the types AS and PA and recall some of their basic properties. In particular we describe the normalizer of the socle for groups of type PA and how to construct the normalizer of the socle, if the group is given in a sufficiently well-behaved form.

The version of the O’Nan-Scott Theorem we use, for a proof see [liebeck-praeger-saxl:onan-scott], is:

###### Theorem 3.2.

Let be a primitive group on a set . Then is a group of type HA, AS, PA, HS, HC, SD, CD, or TW.

The abbreviation AS stands for Almost Simple. A group is called almost simple if it contains a non-abelian simple group and can be embedded into the automorphism group of said simple group. A primitive group is of AS type if its socle is a non-regular non-abelian simple group.

The abbreviation PA stands for Product Action. The groups of AS type form the building blocks for the groups of PA type. To define this type, we shortly recall the notion of wreath products and their product action.

The wreath product of two permutation groups and is denoted by and defined as the semidirect product where acts per conjugation on by permuting its components. We identify and with the corresponding subgroups of and call them the base group and the top group, respectively.

For two permutation groups and the product action of the wreath product on the set of tuples is given by letting the base group act component-wise on and letting the top group act by permuting the components of .

###### Definition 3.3.

Let be a primitive group. We say that is of type PA if there exist an and a primitive group of type AS such that is permutation isomorphic to a group with

 (SocH)ℓ≤ˆG≤H≀Sℓ

in product action on .

The product action wreath products and are examples for primitive groups of type PA.

Let and be as in Definition 3.3. We sketch how to construct the normalizer of the socle of . Let . Since is given acting in product action we can read off and thus compute . By [dixon-mortimer:permutation-groups, Lemma 4.5A] we know that the normalizer of in is . By recent work of Luks and Miyazaki we can compute the normalizer of , in polynomial time [luks-miyazaki:normalisers-poly, Corollary 3.24]. More precisely this approach yields the following corollary:

###### Corollary 3.4.

Let be a primitive group of type PA with socle in component-wise action on . Then can be computed in polynomial time.

In the practical implementation we use the GAP built-in algorithm to compute the normalizer of in . Our long-term goal is to use the constructive recognition provided by the recog package [recog1.3.2]. Computing the normalizer of in is then only a matter of iterating through representatives for the outer automorphisms of .

## 4 Permutation morphisms

In general a group of PA type might be given on an arbitrary set and needs only be permutation isomorphic to a group in product action. In this section we discuss how to construct such a permutation isomorphism:

###### Lemma 4.1.

Let be a primitive group of type PA. Then we can compute a non-abelian simple group , a positive integer , and a permutation isomorphism from to a permutation group such that the socle of is in component-wise action on .

To this end we present the notion of permutation morphisms developed in [siccha:phd-thesis]. They arise from permutation isomorphisms by simply dropping the condition that the domain map and the group homomorphism be bijections. We illustrate how to use them to prove Lemma 4.1.

### 4.2 Basic Definitions

For two maps and we denote by the product map . For a right-action of a group and , we also denote by .

###### Definition 4.3.

Let and be permutation groups on sets and , respectively, let be a map, and let be a group homomorphism. Furthermore let and be the natural actions of and on and , respectively. We call the pair a permutation morphism from to if the following diagram commutes:

 Ω×G Ω Δ×H Δ,

that is if holds for all , . We call the group homomorphism of and the domain map of .

It is immediate from the definition, that the component-wise composition of two permutation morphisms again yields a permutation morphism. In particular we define the category of permutation groups as the category with all permutation groups as objects, all permutation morphisms as morphisms, and the component-wise composition as the composition of permutation morphisms. We rely on this categorical perspective in many of our proofs.

We denote a permutation morphism from a permutation group to a permutation group by . When encountering this notation keep in mind that itself is not a map but a pair of a domain map and a group homomorphism. We use capital letters for permutation morphisms.

It turns out that a permutation morphism is a mono-, epi-, or isomorphism in the categorical sense if and only if both its domain map and group homomorphism are injective, surjective, or bijective, respectively.

For a permutation group we call a map compatible with if there exists a group homomorphism such that is a permutation morphism. We say that a partition of is -invariant if for all and we have .

###### Lemma 4.4 ([siccha:phd-thesis, Lemma 4.2.10]).

Let be a permutation group and a map. Then is compatible with if and only if the partition of into the non-empty fibers is -invariant.

If is transitive, then the -invariant partitions of are precisely the block systems of . Hence for a given blocksystem we can define a compatible map by sending each point to the block it is contained in.

Let be a permutation group and a surjective map compatible with . Then there exist a unique group and a unique group homomorphism such that is a permutation epimorphism, see [siccha:phd-thesis, Corollary 4.2.7]. We call the permutation epimorphism and the group epimorphism of induced by .

###### Example 4.5.

Let , , , and . Further consider the set , the map and the following geometric arrangement of the points :

Observe that acts on by permuting the points horizontally, while acts on by permuting the points vertically. The map projects vertically or “to the top”. Notice how the fibers of correspond to a block-system of . We determine the group epimorphism of induced by . By definition is the permutation which makes the following square commute:

 Ω Ω Ω1 Ω1

Take . We have , , and . Hence . Correspondingly we get .

### 4.6 Products of permutation morphisms

For two permutation groups and we define the product of the permutation groups and as the permutation group given by in component-wise action on . Correspondingly, for an additional permutation group and two permutation morphisms and from to and , respectively, we define the product permutation morphism as . Iteratively, we define the product of several permutation groups or permutation morphisms.

To prove Lemma 4.1 it suffices to be able to compute the following: given the socle of a PA type group compute a non-abelian simple group and permutation epimorphisms, think projections, such that the product morphism is an isomorphism. Since every surjective map compatible with induces a unique permutation epimorphism, it in turn suffices to compute suitable maps .

###### Example 4.7.

Consider the situation in Example 4.5. Let and . Then the map is compatible with and induces the permutation epimorphism . The product maps and are isomorphisms of sets and permutation groups, respectively.

We illustrate how to construct one of the needed projections for PA type groups.

###### Example 4.8.

Let and . We denote by the trivial permutation group on . The subgroup is normal in . Let us denote sets of the form by . Then partitioning into orbits under yields the block system

 Σ={{(∗,δ2)}|δ2∈Δ}.

Note how mapping each to the block of it is contained in is equivalent to mapping each to . Thus we have essentially constructed the map . Observe that we only used the group theoretic property that is a maximal normal subgroup of and thus in particular did not use the actual product structure of .

Analogously we can construct the map . For let be the permutation epimorphisms of induced by and , respectively. Since is an isomorphism, must be a monomorphism. By order arguments is thus an isomorphism.

In general the above construction does not yield permutation epimorphisms with identical images. We can alleviate this by computing elements of the given group which conjugate the minimal normal subgroups of its socle to each other. For the general construction see the (homogenized) product decomposition by minimal normal subgroups in [siccha:phd-thesis, Definitions 5.1.3 and 5.1.5]. Lemma 4.1 then follows from [siccha:phd-thesis, Corollary 5.19].

## 5 Reduction Homomorphism

Recall from Section 2 that a key ingredient of our second phase is a group homomorphism which reduces the original problem on points to a problem on less or equal than points. We illustrate shortly how to construct this homomorphism, for the details refer to [siccha:phd-thesis, Theorem 9.1.6].

Let be a primitive group with non-regular socle and be a socle-component of , confer [siccha:phd-thesis, Chapters 5 and 7] for a definition. Then is a non-abelian simple group, there exists a positive integer such that is isomorphic to , and by [siccha:phd-thesis, Lemma 2.6.1] we have for some . Denote by the permutation group induced by the right-regular action of on itself. We show that we can evaluate the following two group homomorphisms: first an embedding and second an epimorphism , where is the imprimitive wreath product and thus acts on points.

We sketch the proof that . Let and . Note that for we have . Since is regular, we have . Since is a socle-component of , we have by [guralnick-maroti-pyber:normalizers-primitive-groups, Lemma 7.7]. In total we have .

In our implementation we use a modified version of this reduction. For groups of type PA we can directly compute an isomorphism from the product action wreath product into the corresponding imprimitive wreath product.

## 6 Implementation

A version of our normalizer algorithm for groups of type PA is implemented in the GAP package NormalizersOfPrimitiveGroups.

Table 1 shows a comparison of runtimes of our algorithm and the GAP function Normalizer. At the time of writing, there are two big bottlenecks in the implementation. First, the GAP built-in algorithm to compute the socle of a group is unnecessarily slow. State-of-the-art algorithms as in [cannon-holt:chief-series-composition-series-socle] are not yet implemented. Secondly, computing a permutation which transforms a given product decomposition into a so-called natural product decomposition currently also is slow. The latter may be alleviated by implementing the corresponding routines in for example C [c-lang] or Julia [julia-lang]. Note that the actual normalizer computation inside the normalizer of the socle appears to be no bottleneck: in the example with socle type it took only 40ms!

## 7 Acknowledgments

Substantial parts of the work presented in this article were written while the author was supported by the German Research Foundation (DFG) research training group “Experimental and constructive algebra” (GRK 1632) and employed by the Lehr- und Forschungsgebiet Algebra, RWTH Aachen University and the Department of Mathematics, University of Siegen.