# Enumerating five families of pattern-avoiding inversion sequences; and introducing the powered Catalan numbers

The first problem addressed by this article is the enumeration of some families of pattern-avoiding inversion sequences. We solve some enumerative conjectures left open by the foundational work on the topics by Corteel et al., some of these being also solved independently by Kim and Lin. The strength of our approach is its robustness: we enumerate four families F_1 ⊂ F_2 ⊂ F_3 ⊂ F_4 of pattern-avoiding inversion sequences ordered by inclusion using the same approach. More precisely, we provide a generating tree (with associated succession rule) for each family F_i which generalizes the one for the family F_i-1. The second topic of the paper is the enumeration of a fifth family F_5 of pattern-avoiding inversion sequences (containing F_4). This enumeration is also solved via a succession rule, which however does not generalize the one for F_4. The associated enumeration sequence, which we call of powered Catalan numbers, is quite intringuing, and further investigated. We provide two different succession rules for it, denoted Ω_pCat and Ω_steady, and show that they define two types of families enumerated by powered Catalan numbers. Among such families, we introduce the steady paths, which are naturally associated with Ω_steady. They allow to brigde the gap between the two types of families enumerated by powered Catalan numbers: indeed, we provide a size-preserving bijection between steady paths and valley-marked Dyck paths (which are naturally associated to Ω_pCat). Along the way, we provide several nice connections to families of permutations defined by the avoidance of vincular patterns, and some enumerative conjectures.

## Authors

• 1 publication
• 1 publication
• 1 publication
• 2 publications
• ### On 120-avoiding inversion and ascent sequences

Recently, Yan and the first named author investigated systematically the...
03/26/2020 ∙ by Zhicong Lin, et al. ∙ 0

• ### Bijections from Dyck and Motzkin meanders with catastrophes to pattern avoiding Dyck paths

In this note, we present constructive bijections from Dyck and Motzkin m...
04/02/2021 ∙ by Jean-Luc Baril, et al. ∙ 0

• ### Descent distribution on Catalan words avoiding a pattern of length at most three

Catalan words are particular growth-restricted words over the set of non...
03/18/2018 ∙ by Jean-Luc Baril, et al. ∙ 0

• ### A Bijection Between Weighted Dyck Paths and 1234-avoiding Up-Down Permutations

Three-dimensional Catalan numbers are a variant of the classical (bidime...
11/30/2020 ∙ by Justine Falque, et al. ∙ 0

• ### On sequences associated to the invariant theory of rank two simple Lie algebras

We study two families of sequences, listed in the On-Line Encyclopedia o...
11/23/2019 ∙ by Alin Bostan, et al. ∙ 0

• ### Catalan and Schröder permutations sortable by two restricted stacks

We investigate the permutation sorting problem with two restricted stack...
04/03/2020 ∙ by J. -L. Baril, et al. ∙ 0

• ### Constrained Type Families

We present an approach to support partiality in type-level computation w...
06/29/2017 ∙ by J. Garrett Morris, et al. ∙ 0

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction and preliminaries

### 1.1 Context of our work

An inversion sequence of length is any integer sequence satisfying , for all . There is a well-known bijection between the set of all permutations of length (or size) and the set of all inversion sequences of length , which maps a permutation into its left inversion table , where . This bijection is actually at the origin of the name inversion sequences.

The study of pattern-containment or pattern-avoidance in inversion sequences was first introduced in [25], and then further investigated in [15]. Namely, in [25], Mansour and Shattuck studied inversion sequences that avoid permutations of length , while in [15], Corteel et al. proposed the study of inversion sequences avoiding subwords of length . The definition of inversion sequences avoiding words (which may in addition be permutations) is straightforward: for instance, the inversion sequences that avoid the word (resp. the permutation ) are those with no such that (resp. ). Pattern-avoidance on special families of inversion sequences has also been studied in the literature, namely by Duncan and Steingrímsson on ascent sequences – see [17].

The pattern-avoiding inversion sequences of [15] were further generalized in [26], extending the notion of pattern-avoidance to triples of binary relations . More precisely, they denote by the set of all inversion sequences in having no three indices such that , , and , and by . For example, the sets and coincide for every . In [26] all triples of relations in are considered, where “” stands for any possible relation on a set , i.e. for any . Therefore, all the possible triples of relations are examined and the resulting families of pattern-avoiding inversion sequences are subdivided into equivalence classes. Many enumeration results complementing those in [15, 25] have been found in [26]. In addition, several conjectures have been formulated in [26]

. Some (but by far not all!) of them have been proved between the moment a first version of

[26] was posted on the arXiv and its publication, and references to these recent proofs can also be found in the published version of [26].

In this paper we study five families of inversion sequences which form a hierarchy for the inclusion order. The enumeration of these classes – by well-known sequences, such as those of the Catalan, the Baxter, and the newly introduced semi-Baxter numbers [11] – was originally conjectured in the first version of [26]. These conjectures have attracted the attention of a fair number of combinatorialists, resulting in proofs for all of them, independently of our paper. Still, our work reproves these enumeration results. Along the way, we further try to establish bijective correspondences between these families of inversion sequences and other known combinatorial structures. The most remarkable feature of our work is that all the families of inversion sequences are presented and studied in a unified way by means of generating trees. Before proceeding, let us briefly recall some basics about generating trees. Details can be found for instance in [2, 3, 8, 31].

### 1.2 Basics of generating trees

Consider a combinatorial class , that is to say a set of discrete objects equipped with a notion of size such that the number of objects of size is finite, for any . We assume also that contains exactly one object of size . A generating tree for is an infinite rooted tree whose vertices are the objects of each appearing exactly once in the tree, and such that objects of size are at level (with the convention that the root is at level ). The children of some object are obtained by adding an atom (i.e. a piece of the object that makes its size increase by ) to . Since every object appears only once in the generating tree, not all possible additions are acceptable. We enforce the unique appearance property by considering only additions that follow some prescribed rules and call the growth of the process of adding atoms according to these rules.

To illustrate these definitions, we describe the classical growth for the family of Dyck paths, as given by [3]. Recall that a Dyck path of semi-length is a lattice path using up and down unit steps, running from to and remaining weakly above the -axis. The atoms we consider are factors, a.k.a. peaks, which are added to a given Dyck path. To ensure that all Dyck paths appear exactly once in the generating tree, a peak is inserted only in a point of the last descent, defined as the longest suffix containing only letters. More precisely, the children of the Dyck path are , ,…, , .

The first few levels of the generating tree for Dyck paths are shown in Figure 1 (left).

When the growth of is particularly regular, we encapsulate it in a succession rule. This applies more precisely when there exist statistics whose evaluations control the number of objects produced in the generating tree. A succession rule consists of one starting label (axiom) corresponding to the value of the statistics on the root object and of a set of productions encoding the way in which these evaluations spread in the generating tree – see Figure 1(right). The growth of Dyck paths presented earlier is governed by the statistic “length of the last descent”, so that it corresponds to the following succession rule, where each label indicates the number of steps of the last descent in a Dyck path,

 ΩCat=⎧⎪⎨⎪⎩(1)(k)⇝(1),(2),…,(k),(k+1).

Obviously, as we discuss in [12], the sequence enumerating the class can be recovered from the succession rule itself, without reference to the specifics of the objects in : indeed, the th term of the sequence is the total number of labels (counted with repetition) that are produced from the root by applications of the set of productions, or equivalently, the number of nodes at level in the generating tree. For instance, the well-know fact that Dyck paths are counted by Catalan numbers (sequence A000108 in [27]) can be recovered by counting nodes at each level in the above generating tree.

### 1.3 Content of the paper

In our study we focus on five different families of pattern-avoiding inversion sequences, which are depicted in Figure 2. As the figure shows, these families are naturally ordered by inclusion, and are enumerated by well-known number sequences.

The objective of our study is twofold. On the one hand we provide (and/or collect) enumerative results about the families of inversion sequences of Figure 2. On the other hand we aim at treating all these families in a unified way. More precisely, in each of the following sections we first provide a simple combinatorial characterization for the corresponding family of inversion sequences, and then we show a recursive growth that yields a succession rule.

The main noticeable property of the succession rules provided in Sections 2, 3, 4, and 5 is that they reveal the hierarchy of Figure 2 at the abstract level of succession rules. Specifically, the recursive construction (or growth) provided for each family is obtained by extending the construction of the immediately smaller family. Moreover, the ways in which these growths are encoded by labels in succession rules are also each a natural extension of the case of the immediately smaller family. Hence, these examples provide another illustration of the idea of generalizing/specializing succession rules that we discussed in details in [6, Section 2.2]. The outcome of the discussion in [6, Section 2.2] is the following proposed definition for generalization/specialization of succession rules. To say that a succession rule specializes (equivalently, that generalizes or extends ), we require

• the existence of a comparison relation “smaller than or equal to” between the labels of and those of , and,

• for any labels of and of with smaller than or equal to , a way of mapping the productions of the label in to a subset of the productions of the label in , such that a label is always mapped to a larger or equal one.

Comparing Propositions 4, 8, 14 and 17, and mapping the labels in the obvious way, it is easy to see that the succession rules in these propositions satisfy this proposed definition (the comparison relation being here just the componentwise natural order on integers).

We conclude our introduction with a few words commenting on the classes of our hierarchy and our results on them.

• We start in Section 2 with , which we call the family of Catalan inversion sequences. We define two recursive growths for this family, one according to (hence proving that is enumerated by the Catalan numbers) and a second one that turns out to be a new succession rule for the Catalan numbers. The fact that this family of inversion sequences is enumerated by the Catalan numbers was conjectured in [26] and it has recently been proved independently of us by Kim and Lin in [23]. Moreover, we are able to relate the family of Catalan inversion sequences to a family of permutations defined by the avoidance of vincular patterns, proving that they are in bijection with a family of pattern-avoiding permutations.

• In Section 3 we consider the family . This class has been considered independently of us by Lin in the article [24], which proves the conjecture (originally formulated in [26]) that these inversion sequences are counted by sequence A108307 on [27] – defining the enumerative sequence of set partitions of that avoid enhanced 3-crossings [10]. We review Lin’s proof, which fits perfectly in the hierarchy of succession rules that we present.

• In Section 4 we study inversion sequences in , which we call Baxter inversion sequences. This family of inversion sequences was originally conjectured in [26] to be counted by Baxter numbers. The proof of this conjecture was provided in [23] by means of a growth for Baxter inversion sequences that neatly generalizes the previous growth for the family .

• In Section 5, we deal with the family , which we call semi-Baxter inversion sequences. Indeed, this family of inversion sequences was originally conjectured in [26] to be counted by the sequence A117106 [27]; these numbers have been thoroughly studied and named semi-Baxter in the article [11], which among other results proves this conjecture of [26].

• Finally, in Section 6 we deal with , which is the rightmost element of the chain of Figure 2. We call the elements of powered Catalan inversion sequences, since the succession rule we provide for them is a “powered version” of the classical Catalan succession rule.

When turning to powered Catalan inversion sequences, the hierarchy of Figure 2 is broken at the level of succession rules. Indeed, although the combinatorial characterization of these objects generalizes naturally that of semi-Baxter inversion sequences, we do not have a growth for powered Catalan inversion sequences that generalizes the one of semi-Baxter inversion sequences. This motivates the second part of the paper, devoted to the study of this “powered Catalan” enumerative sequence from Section 6 on.

The enumeration of powered Catalan inversion sequences (by A113227, [27]) was already solved in [15]. Our first contribution (in Section 6) is to prove that they grow according to the succession rule , which generalizes the classical rule by introducing powers in it. This motivates the name powered Catalan numbers which we have coined for the numbers of sequence A113227.

Many combinatorial families are enumerated by powered Catalan numbers. Some are presented in Section 7. These families somehow fall into two categories. Inside each category, the objects seem to be in rather natural bijective correspondence. However, between the two categories, the bijections are much less clear. Our result of Section 7 is to provide a second succession rule for powered Catalan numbers (more precisely, for permutations avoiding the vincular pattern ), which should govern the growth of objects in one of these two categories, the other category being naturally associated with the rule .

In Section 8, we describe a new occurrence of the powered Catalan numbers in terms of lattice paths. More precisely, we introduce the family of steady paths and prove that they are enumerated by the powered Catalan numbers. This is proved by showing a growth for steady paths that is encoded by (a variant called of) the succession rule for permutations avoiding the pattern . We also provide a simple bijection between steady paths and permutations avoiding the vincular pattern , therefore recovering the enumeration of this family, already known [4] to be enumerated by A113227.

Finally, in Section 9 we bridge the gap between the two types of powered Catalan structures, by showing a bijection between steady paths (representing the succession rule ) and valley-marked Dyck paths (emblematic of the succession rule ).

## 2 Catalan inversion sequences: I(≥,−,≥)

The first family of inversion sequences considered is . It was originally conjectured in [26] to be counted by the sequence of Catalan numbers [27, A000108] (hence the name Catalan inversion sequences) whose first terms we recall:

 1,1,2,5,14,42,132,429,1430,4862,16796,58786,208012,742900,…

We note that this conjectured enumeration has recently been proved independently from us by Kim and Lin in [23]. Their proof does not involve generating trees, but displays a nice Catalan recurrence for the filtration of where the additional parameter is the value of the last element of an inversion sequence.

We provide another proof of this conjecture in Proposition 3 by showing that there exists a growth for according to the well-known Catalan succession rule . Moreover, we show a second growth for , thereby providing a new Catalan succession rule, which is appropriate to be generalized in the next sections. In addition, we show a direct bijection between and a family of pattern-avoiding permutations, which thus results to be enumerated by Catalan numbers.

### 2.1 Combinatorial characterization

Let us start by observing that the family of Catalan inversion sequences has a simple characterization in terms of inversion sequences avoiding patterns of length three.

###### Proposition 1.

An inversion sequence is in if and only if it avoids , , , , and .

###### Proof.

The proof is rather straightforward, since containing , , such that , with , is equivalent to containing the listed patterns. ∎

In addition to the above characterization, we introduce the following combinatorial description of Catalan inversion sequences, as it will be useful to define a growth according to the Catalan succession rule .

###### Proposition 2.

Any inversion sequence is a Catalan inversion sequence if and only if for any , with ,

if forms a weak descent, i.e. , then , for all .

###### Proof.

The forward direction is clear. The backwards direction can be proved by contrapositive. More precisely, suppose there are three indices , such that . Then, if , forms a weak descent and the fact that concludes the proof. Otherwise, since , there must be an index , with , such that forms a weak descent and . This concludes the proof as well. ∎

The previous statement means that any of our inversion sequences has a neat decomposition: they are concatenations of shifts of inversion sequences having a single weak descent, at the end. A graphical view of this decomposition is shown in Figure 3.

### 2.2 Enumerative results

###### Proposition 3.

Catalan inversion sequences grow according to the succession rule ,

 ΩCat=⎧⎪⎨⎪⎩(1)(k)⇝(1),(2),…,(k),(k+1).
###### Proof.

Given an inversion sequence , we define the inversion sequence as the sequence , where the entry is inserted in position , for some , and the entries are shifted rightwards by one. By definition of inversion sequences, is the largest possible value that the th entry can assume. And moreover, letting , it holds that , for all ; namely the index is the rightmost index such that . For example, if and , then .

Then, we note that given a Catalan inversion sequence of length , by removing from the rightmost entry whose value is equal to its position minus one, we obtain a Catalan inversion sequence of length . Note that for every Catalan inversion sequence, thus such an entry always exists.

Therefore, we can describe a growth for Catalan inversion sequences by inserting an entry in position . By Proposition 2, since the entry forms a weak descent in , the inversion sequence is a Catalan inversion sequence of length if and only if . Then, we call active positions all the indices , with , such that is a Catalan inversion sequence of length . According to this definition, and are always active positions: indeed, both and are Catalan inversion sequences of length .

We label a Catalan inversion sequence of length with , where the number of active positions is . Note that the smallest inversion sequence has label , which is the axiom of rule .

Now, we show that given a Catalan inversion sequence of length with label , the labels of , where ranges over all the active positions, are precisely the label productions of in .

Let be the active positions of from left to right. Note that and . We argue that, for any , the active positions of the inversion sequence are , and . Indeed, on the one hand any position which is non-active in is still non-active in . On the other hand, by Proposition 2, the index becomes non-active in , since by definition. Similarly, any position , with , which is active in becomes non-active in . Thus, the active positions of are , and . Hence, has label , for any .∎

Furthermore, we can provide a new succession rule for generating Catalan inversion sequences: the growth we provide in the following is remarkable as it allows generalizations in the next sections.

###### Proposition 4.

Catalan inversion sequences grow according to the following succession rule

 ΩCat2=⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩(1,1)(h,k)⇝(0,k+1)h,(h+1,k),(h+2,k−1),…,(h+k,1).
###### Proof.

We consider the growth of Catalan inversion sequences that consists of adding a new rightmost entry, and we prove that this growth defines the succession rule . Obviously, this growth is different from the one provided in the proof of Proposition 3.

Let be the maximum value among the entries of . And let be the maximum value of the set of all entries that form a weak descent of ; if has no weak descents, then . By Proposition 1, since avoids , and , the value is or . In particular, if , then .

By Proposition 2, it follows that is a Catalan inversion sequence of length if and only if . Moreover, if , then forms a new weak descent of , and becomes the value ; whereas, if , then since the weak descents of and coincide.

Now, we assign to any Catalan inversion sequence of length the label , where and . In other words, (resp. ) marks the number of possible additions smaller than or equal to (resp. greater than) the maximum entry of .

The sequence has no weak descents, thus it has label , which is the axiom of . Let be a Catalan inversion sequence of length with label . As Figure 4 illustrates, the labels of the inversion sequences of length produced by adding a rightmost entry to are

• , for any ,

• , when ,

which concludes the proof that Catalan inversion sequences grow according to . ∎

It is well worth noticing that although the above succession rule generates the well-known Catalan numbers, we do not have knowledge of this succession rule in the literature.

### 2.3 One-to-one correspondence with AV(1-23,2-14-3)

In this section we show that Catalan inversion sequences are just left inversion tables of permutations avoiding the patterns and , thereby proving that the family of pattern-avoiding permutations forms a new occurrence of the Catalan numbers. We start by recalling some terminology and notation.

A (Babson-Steingrímsson-)pattern of length is any permutation of where two adjacent entries may or may not be separated by a dash – see [1]. Such patterns are also called generalized or vincular. The absence of a dash between two adjacent entries in the pattern indicates that in any pattern-occurrence the two entries are required to be adjacent: a permutation of length contains the vincular pattern , if it contains as pattern, and moreover, there is an occurrence of the pattern where the entries of not separated by a dash are consecutive entries of the permutation ; otherwise, avoids the vincular pattern . Let be a set of patterns. We denote by the family of permutations of length that avoid any pattern in , and define .

###### Proposition 5.

For any , Catalan inversion sequences of length are in bijection with . Consequently, the family is enumerated by Catalan numbers.

###### Proof.

The second part of the statement is a immediate consequence of the first part, which we now prove.

Let be the mapping associating to each its left inversion table . We will use many times the following simple fact: for every , if (i.e. the pair is an inversion), then .

Let be the reverse operation on arrays. We can prove our statement by using the mapping , which is a bijection between the family of permutations and integer sequences such that . We will simply show that the restriction of the bijection to the family yields a bijection with Catalan inversion sequences. Precisely, we want to prove that for every , an inversion sequence is in the set if and only if it is a Catalan inversion sequence of length (i.e. belongs to ).

• We prove the contrapositive: if , then contains or . Let . Then, is the left inversion table of a permutation , i.e. . Since , there exist three indices, , such that and .

Without loss of generality, we can suppose that there is no index , such that and and . Namely is the leftmost entry of that is at least as large as both and . Then, we have two possibilities:

1. either ,

2. or , and in this case it holds that or .

First, from and it follows that and .

Now, we prove that both in case 1. and in case 2. above we have .

1. Let us consider the subsequence . We have and . If also , then it forms a .

Otherwise, it must hold that , and thus . Since the pair is an inversion of and , there must be a point on the right of such that is an inversion and is not. Thus, forms a .

2. First, if , consider the subsequence . It follows that , since , and thus . In addition, we know that . Then, forms an occurrence of if . Otherwise, it must hold that . As in case 1., the pair is an inversion, and . Therefore, there must be an element on the right of such that is an inversion and is not. Hence forms a .

Now, suppose , and consider the subsequence . According to case 2., it must be that , and since , it holds that . Since both and hold, forms an occurrence of .

• By contrapositive, if a permutation contains or , then is not in .

• If contains , there must be two indices and , with , such that forms an occurrence of . We can assume that no points between and are such that . Otherwise we consider as our occurrence of .

Then, two relations hold: and , and thus .

• If contains , and avoids , there must be three indices and , with , such that forms an occurrence of . We can assume that no points between and are such that . Indeed, in case held, would be an occurrence of ; whereas, if , we could consider as our occurrence of .

Then, as above , and because is an inversion of . Nevertheless, is an inversion of as well, and . Thus, and . ∎

We mention that although inversion sequences are actually a coding for permutations, it is often not easy (if at all possible) to characterize the families in terms of families of pattern-avoiding permutations. A few examples of bijective correspondences between pattern-avoiding inversion sequences and pattern-avoiding permutations have been collected in [26]. We report below the examples of [26] where the permutations are defined by the avoidance of classical patterns:

• and , [26, Theorem 1];

• and , [26, Theorem 9];

• and , [26, Section 2.6.1];

• and , [26, Theorem 16];

• and , [26, Theorem 27];

• and and , [26, Theorem 37-38];

• and , [26, Theorem 40];

• and , [26, Theorem 45].

In addition, [26, Theorem 56] shows a bijective correspondence between and a family of permutations avoiding a specific marked mesh pattern. Our case of and shows another example of such bijective correspondences, where the excluded patterns on permutations are however vincular.

## 3 Inversion sequences I(≥,≥,≥)

Following the hierarchy of Figure 2, the next family we turn to is . This family was originally conjectured in [26] to be counted by sequence A108307 on [27], which is defined as the enumerative sequence of set partitions of that avoid enhanced 3-crossings [10]. In [10, Proposition 2] it is proved that the number of these set partitions is given by and the recursive relation

 8(n+3)(n+1)E3(n)+(7n2+53n+88)E3(n+1)−(n+8)(n+7)E3(n+2)=0, (1)

which holds for all . Thus, the first terms of sequence A108307 according to recurrence (1) are

 1,1,2,5,15,51,191,772,3320,15032,71084,348889,1768483,9220655,49286863,…

At the conference Permutation Patterns 2017 in Reykjavik, we presented [30] a proof that the enumerative sequence of the family is indeed the sequence A108307. Our proof works as follows. First, we build a generating tree for , which is encoded by a succession rule that generalizes the one in Proposition 4. Then, we solve the resulting functional equation using a variant of the so-called kernel method – see [8, 22] and references therein – which is sometimes referred to as obstinate kernel method. The Lagrange inversion formula can then be applied to yield a closed formula for the number of inversion sequences in . And finally, using the method of creative telescoping, we deduce from this closed formula a recurrence satisfied by the considered enumerating sequence.

The details of this proof are not provided in the following. The interested reader may however find them in a previous version of our paper [7], or in the PhD thesis of the third author [21, Section 5.2]. The reason for this omission is that essentially the same proof has been independently found by Lin [24]. In the following, we simply give some statements that constitute the main steps of the proof, together with a reference to the corresponding statements in the paper of Lin.

We also point out to the interested reader that Yan [32] has now also provided a bijective proof that inversion sequences in and set partitions avoiding enhanced 3-crossings are enumerated by the same sequence.

### 3.1 Combinatorial characterization

To start, we provide a combinatorial description of the family , which is useful to prove Proposition 8.

As Figure 2 shows, the family properly includes as a subfamily. For instance, the inversion sequence is both in and in , while is not in despite being in . The following characterization makes this fact explicit.

###### Proposition 6.

An inversion sequence belongs to if and only if it avoids , , and .

###### Proof.

The proof is a quick check that containing such that , with , is equivalent to containing the above patterns. ∎

The above result makes clear that every Catalan inversion sequence is in . In addition, Proposition 6 proves the following property stated in [26, Observation 7].

###### Remark 7.

Let any inversion sequence be decomposed into two subsequences , which is the increasing sequence of left-to-right maxima of (i.e. entries such that , for all ), and , which is the (possibly empty) sequence comprised of all the remaining entries of .

Then, an inversion sequence is in the set if and only if and are both strictly increasing sequences – see decomposition in Figure 5 where the sequence is highlighted.

### 3.2 Enumerative results

###### Proposition 8.

The family grows according to the following succession rule

 ΩI(≥,≥,≥)=⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩(1,1)(h,k)⇝(h−1,k+1),(h−2,k+1),…,(0,k+1),(h+1,k),(h+2,k−1),…,(h+k,1).

This proposition corresponds to Lemma 2.2 in [24]. It is proved by letting inversion sequences of grow by adding a new rightmost entry, and by giving to each such inversion sequence a label as follows. Let is the maximum value of and be the rightmost entry of , if there is any, otherwise . The label of is then defined by and . The growth of inversion sequences of is illustrated in Figure 6.

The next steps toward the enumeration of the family are to translate the succession rule of Proposition 8 into a functional equation, and then to solve it.

For , let denote the size generating function of inversion sequences of the family having label . The rule translates using a standard technique into a functional equation for the generating function .

###### Proposition 9.

The generating function satisfies the following functional equation

 A(y,z)=xyz+xz1−y(A(1,z)−A(y,z))+xyzz−y(A(y,z)−A(y,y)). (2)

The above statement coincides with Proposition 2.3 in [24].

Equation (2) is a linear functional equation with two catalytic variables, and , in the sense of Zeilberger [34]. Similar functional equations have been solved by using the obstinate kernel method (see [8, 11], and references therein), which allows us to provide the following expression for the generating function of . Note that the same method was also applied in [24] to derive the following theorem (Theorem 3.1 in [24]).

###### Theorem 10.

Let be the unique formal power series in such that

 W=x¯a(W+1+a)(W+a+a2).

The series solution of Equation (2) satisfies

 A(1+a,1+a)=[Q(a,W)(1+a)3]≥,

where is a polynomial in whose coefficients are Laurent polynomials in defined by

 Q(a,W) =(−1a6−3a5−3a4−1a3+1+3a+3a2+a3)W +(1a5+1a4−1a−1)W2+(1a6−1a4+1a3−1a)W3+(−1a5+1a4)W4,

and the notation stands for the formal power series in obtained by considering only those terms in the series expansion of that have non-negative powers of .

Note that and are algebraic series in whose coefficients are Laurent polynomials in . It follows, as in [8, page 6], that is D-finite. Hence, the specialization , which is the generating function of , is D-finite as well.

Applying the Lagrange inversion formula to the expression of in Theorem 10, we can derive an explicit, yet very complicated, expression for the coefficients of the generating function . Although this expression is complicated, Zeilberger’s method of creative telescoping [28, 33] can be applied to it, and provides a much simpler recursive formula satisfied by these numbers. This is also how the proof that is enumerated by [27, A108307] is concluded in [24], giving the following statement.

###### Proposition 11.

Let . The numbers are recursively defined by and for ,

 8(n+3)(n+1)an+(7n2+53n+88)an+1−(n+8)(n+7)an+2=0.

Thus, is sequence A108307 on [27].

## 4 Baxter inversion sequences: I(≥,≥,>)

The next family of inversion sequences according to the hierarchy of Figure 2 is . This family of inversion sequences was originally conjectured in [26] to be counted by the sequence A001181 [27] of Baxter numbers, whose first terms are

 1,2,6,22,92,422,2074,10754,58202,326240,1882960,11140560,67329992,…

This conjecture has recently been proved in [23, Theorem 4.1]. Accordingly, we call the family of Baxter inversion sequences.

The proof of [23, Theorem 4.1] is analytic. Precisely, [23, Lemma 4.3] provides a succession rule for . It is then shown to generate Baxter numbers using the obstinate kernel method and the results in [8, Section 2]. This succession rule is however not a classical one associated with Baxter numbers, and no other Baxter family is known to grow according to this new Baxter succession rule. It would be desirable to establish a closer link (either via generating trees, or via bijections) between and any other known Baxter family.

### 4.1 Combinatorial characterization

The family of Baxter inversion sequences clearly contains , as shown by the following characterization.

###### Proposition 12.

An inversion sequence is a Baxter inversion sequence if and only if it avoids , and .

###### Proof.

The statement is readily checked, as in Propositions 1 and 6. ∎

Another characterization for this family is the following. Recall that for an inversion sequence , we call an entry a LTR maximum (resp. RTL minimum), if , for all (resp. , for all ).

###### Proposition 13.

An inversion sequence is a Baxter inversion sequence if and only if for every and , with and , both is a LTR maximum and is a RTL minimum.

###### Proof.

The proof in both directions is straightforward by considering the characterization of Proposition 12. ∎

### 4.2 Enumerative results

We choose to report here a proof of [23, Lemma 4.3] (which is omitted in [23]). This proof is essential in our work, since it displays a growth for Baxter inversion sequences that generalizes the one for the family provided in Proposition 8.

###### Proposition 14.

Baxter inversion sequences grow according to the following succession rule

 ΩBax=⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩(1,1)(h,k)⇝(h−1,k+1),…,(1,k+1),(1,k+1),(h+1,k),…,(h+k,1).
###### Proof.

We show that the growth of Baxter inversion sequences by addition of a new rightmost entry (as in the proofs of Propositions 4 and 8) can be encoded by the above succession rule .

As in the proof of Proposition 8, let be the value of the rightmost entry of which is not a LTR maximum, if there is any. Note that is also the largest value that is not a LTR maximum, since avoids by Proposition 12. Otherwise, if such an entry does not exist, we set equal to the smallest value of , i.e. .

Moreover, if this rightmost entry of which is not a LTR maximum exists, it can either form an inversion (i.e. there exists an entry on its left such that ) or not. We need to distinguish two cases in order to define the addition of a new rightmost entry to :

• in case either all the entries of are LTR maxima, or the rightmost entry of which is not a LTR maximum does not form an inversion;

• in case the rightmost entry of which is not a LTR maximum exists and does form an inversion.

Then, according to Proposition 13, we have that

• The sequence is a Baxter inversion sequence of length if and only if . Moreover, if , where as usual is the maximum value of , then and falls in case (b). Else if , then again , yet falls in case (a). While, if , is a LTR maximum of , which thus falls in the same case (a) as , and .

• The sequence is a Baxter inversion sequence of length if and only if . In particular, if , then and falls in case (b). Else if , then again and falls in case (a). While, if , as above is a LTR maximum of , which thus falls in the same case (b) as , and .

Now, we assign to any Baxter inversion sequence of length a label according to the above distinction: in case (a) (resp. (b)) we assign the label , where (resp. ) and .

The sequence of size one falls in case (a), thus it has label , which is the axiom of . Now, let be a Baxter inversion sequence of length with label . Following the above distinction, the inversion sequences of length produced by adding a rightmost entry to have labels:

• , when ,

• , for