 # Lattice paths with a first return decomposition constrained by the maximal height of a pattern

We consider the system of equations A_k(x)=p(x)A_k-1(x)(q(x)+∑_i=0^k A_i(x)) for k≥ r+1 where A_i(x), 0≤ i ≤ r, are some given functions and show how to obtain a close form for A(x)=∑_k≥ 0A_k(x). We apply this general result to the enumeration of certain subsets of Dyck, Motzkin, skew Dyck, and skew Motzkin paths, defined recursively according to the first return decomposition with a monotonically non-increasing condition relative to the maximal ordinate reached by an occurrence of a given pattern π.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction and notations

Let be a combinatorial class, that is a collection of similar objects (lattices, trees, permutations, words) endowed with a size function whose values are non-negative integers, so that the number of objects of a given size is finite. With we associate the generating function

 A(x)=∑a∈Axℓ(a)=∑n⩾0|ℓ−1(n)|xn,

which appears to be the main source of interest for researchers working in enumerative combinatorics.

Given a function , called statistic, and an integer , we define the combinatorial class consisting of objects such that , and associate the following generating function with this class:

 Ak(x)=∑a∈Akxℓ(a), which implies A(x)=∑k⩾0Ak(x).

In three recent articles [1, 2, 6], the authors enumerate particular subclasses of lattice paths (Dyck, Motzkin, -Motzkin and Schröder paths) defined recursively according to the first return decomposition with a condition on the path height. More precisely, in the first paper, they focus on the Dyck path set consisting of the union of the empty path with all Dyck paths having a first return decomposition satisfying the conditions:

 {α,β∈Dh,⩾,h(UαD)⩾h(β), (1)

where is the height statistic, i.e. the maximal ordinate reached by a path. In these studies, the authors need to consider a system of equations involving the generating functions , , for the subset of paths satisfying :

 Dk(x)=x⋅Dk−1(x)⋅k∑i=0Di(x) for k⩾1 (2)

anchored with initial condition .

They deduce algebraically a close form of the generating function for by solving System (2), and prove that is enumerated by the Motzkin sequence, that is A001006 in the On-Line Encyclopedia of Integer Sequences (OEIS) . The second paper is a similar study in the context of Motzkin and Schröder paths.

Inspired by these two works, the motivation of the paper is to extend such studies by investigating the solutions of a more general system defined as follows. For an integer and two functions and , we consider the system of equations:

 Ak(x)=p(x)Ak−1(x)(q(x)+k∑i=0Ai(x)), % for k⩾r+1, (3)

anchored with for where , , are some given functions. For the sake of brevity, we set , , and .

Before considering how Equation (3) is related to particular sets of lattice paths defined in the same way as the set , let us recall some definitions and notations. A skew Motzkin path (see ) of length is a lattice path in the quarter plane starting at point , ending on the -axis (but never going below), consisting of steps of the four following types: up steps , down steps , flat steps and left steps , such that left and up steps never overlap. Let be the set of all skew Motzkin paths of any length. A Motzkin path is a skew Motzkin path with no left steps. Let be the set of all Motzkin paths. A Dyck path is a Motzkin path with no flat steps. Let be the set of all Dyck paths. A skew Dyck path is a skew Motzkin path with no flat steps. Let be the set of all skew Dyck paths. Obviously, we have and .

A pattern of length in a lattice path consists of consecutive steps. The height of an occurrence of a pattern in a given path is the maximal ordinate reached by its points. For a pattern , we define the statistic on lattice paths so that is the maximal height reached by the occurrences of in . The amplitude of a pattern is the height of the path considered as a path touching the -axis in the quarter plane. For instance, the path contains the pattern and , while the amplitude of is .

The paper is organized as follows. In Section 2, we show how we can obtain a close form for defined by the system (3). In Section 3, we provide several concrete applications in the context of lattices paths (Dyck, Motzkin, skew Dyck, and skew Motzkin paths) for the statistic giving the maximal ordinate reached by the occurrence of in a path. Finally in Section 4, we provide some possible future research directions.

## 2 General result

For , we set , and thus we have . The next theorem provides a close form of whenever Equation (3) admits a solution, that is when for all .

###### Theorem 2.1.

If Equation (3) has a solution then for we have

 Bk(x)=a+bBk−1(x)c+dBk−1(x), (4)

where

 ⎧⎪ ⎪ ⎪⎨⎪ ⎪ ⎪⎩a=p2qv(q+u+v)−pqu−u−vb=−p(pq+1)(q+v+u)c=−p2v(q+u+v)−qp−pv−1d=p2(q+v+u).

Moreover is a root of the polynomial

 d⋅A(x)2+(c−b)⋅A(x)−a. (5)
###### Proof.

Using , we solve the following linear system for

 ⎧⎨⎩Br+1(x)(c+dBr(x))=a+bBr(x)Br+2(x)(c+dBr+1(x))=a+bBr+1(x)Br+3(x)(c+dBr+2(x))=a+bBr+2(x),

and obtain (modulo a multiplicative factor) the claimed values. However, it remains to prove that

 Bk(x)=a+bBk−1(x)c+dBk−1(x)

for any . We proceed by induction on . Obviously, for it is true. So, we assume that the formula is true for and we prove that this is also true for . More precisely, we will prove that equals zero.

Considering Equation (3), we have . The recurrence hypothesis provides , and we have . Replacing respectively , and in , and multiplying it by , we obtain

 (qp(a+bSc+dS−S)+a+bSc+dS)(c+d(a+bS)c+dS)−(a+b(a+bS)c+dS)(1−p(a+bSc+dS−S)),

where . A simplification of this expression (whatever the value of ) implies that , which completes the induction. Finally, taking the limit when of both sides of , and using the fact that , one gets

 d⋅A(x)2+(c−b)⋅A(x)−a=0.

Notice that an alternative proof can be obtained by using Chebyshev polynomials of the second kind (see Proposition D.5 p. 508 in ). ∎

## 3 Some applications

In this section, we apply Theorem 2.1 to study special kinds of classes of lattices paths: Dyck, Motzkin, skew Dyck, and skew Motzkin paths. We extend the definition (1) to these paths using the maximal height reached by the occurrences of , the statistic instead of statistic . Notice that the statistic equals the statistic on all considered paths, which allows us to focus only on . We will deal with patterns of length at most three.

Given a pattern , we define the set as the union of the empty path with paths satisfying a recurrence condition (), which depends on the first return decomposition of a path. The four cases are described in the following table with .

Paths First return decomposition with recurrence condition ()
Dyck       with
Motzkin with and
Skew Dyck
Skew Motzkin

We keep the notations used in the previous sections. The generating functions , , correspond to the sets of Dyck paths so that . We set and .

In any case, the set of paths such that is the set of paths having no occurrence of at height at least one. In the case where the amplitude of is at least one, consists of the paths avoiding . So, we can easily obtain using the first return decomposition of such paths. For Dyck and Motzkin paths, these generating functions (for short g.f.) are already known (see [3, 4, 8, 9, 11, 12]). For skew Dyck and Motzkin paths, we can obtain them by similar methods, which are classic and present no difficulty. So, we do not give here all the details to get .

Note that the maximal height of an occurrence of pattern in is necessarily greater or equal than its amplitude . This means that the generating functions for the sets , satisfy . Finally, if (which is equivalent to for all ) then , is the set of paths having at least one occurrence of and such that , i.e. all occurrences of touch the -axis in . Similar methods as for can be used in order to obtain . Theorems 3.1 and 3.2 focus on two examples of these methods in the context of Dyck paths.

### 3.1 Dyck and Motzkin paths

First we consider Dyck paths. For a given pattern , Condition () implies that for we have

 Ak(x)=x⋅Ak−1(x)⋅k∑i=0Ai(x)

which is a particular case of Equation (3) for , , , , and . From Theorem 2.1, we can easily deduce the following.

###### Corollary 3.1.
 A(x)=x2uv+x2v2−ux+1−√Δ2x2(v+u) (6)

with .

In order to convince the reader that all generating functions can be obtained easily by calculating algebraically and , we give here the methods for the two patterns and .

###### Theorem 3.1.

If then we have , , , and .

###### Proof.

As already mentioned above, the set is the set of Dyck paths avoiding the pattern . So, a nonempty path is of the form where , which implies that . On the other hand, the set is the set of Dyck paths such that , which cannot be possible because the amplitude ; so, . Finally, the set is the set of Dyck paths such that , which means that can be written with and . So, we have the functional equation , which gives the expected result. ∎

###### Theorem 3.2.

If then we have , , , and .

###### Proof.

The set is the set of Dyck paths avoiding the pattern . So, a nonempty path is of the form where and for some , which implies that , and thus . On the other hand, the set is the set of Dyck paths such that , which cannot be possible because the amplitude ; so, . Finally, the set is the set of Dyck paths such that , which means that can be written with and .

So, we have the functional equation , which gives the expected result. ∎

So, we present in the following table the first values of the sequences corresponding to the cardinality of for patterns of length at most three.

Statistic OEIS
, , , , , , , 1, 2, 4, 9, 21, 51, 127, 323 A001006
1, 2, 4, 8, 17, 39, 94, 233, 588
, 1, 2, 5, 13, 35, 97, 274, 786, 2282
, 1, 2, 4, 9, 22, 56, 146, 389, 1053
, 1, 2, 5, 13, 34, 89, 234, 621, 1669

Now, we examine the set of Motzkin paths. For a given pattern , Condition () implies that for we have

 Ak(x)=x2⋅Ak−1(x)⋅k∑i=0Ai(x)

which is a particular case of Equation (3) for , , , , and . Then, the generating function is the generating function found above in the context of Dyck paths (see ()) applied on the variable ( is not replaced with in and ). The following table gives the first values of generating functions obtained for all patterns of length at most two.

Statistic OEIS
, , 1, 2, 3, 6, 11, 22, 43, 87, 176 A026418
1, 2, 4, 8, 17, 36, 78, 170, 374
, 1, 2, 4, 9, 20, 46, 107, 253, 604
1, 2, 3, 7, 13, 29, 61, 138, 308
1, 2, 4, 9, 20, 46, 107, 252, 599
, 1, 2, 4, 8, 17, 37, 82, 185, 422
, 1, 2, 4, 8, 17, 36, 79, 175, 395
1, 2, 4, 9, 20, 47, 111, 268, 653

Looking at the two tables above, one observes that some patterns have same sequences, for example and , and . We actually can generalize this as follows. Let be a pattern of length and be the reversed complement of , that is where

 ¯¯¯x=⎧⎨⎩F if x=F,D if x=U,U if x=D.

For instance we have .

###### Theorem 3.3.

If is the generating function for the set , then we have

 πA(x)=C(π)A(x).
###### Proof.

For short we set . Since the reversed complement operation preserves the amplitude, we have . Now, we will exhibit a bijection from to . Suppose , if a path does not contain , we set . It avoids and belongs to . So, we have . If for some , we have and we also set . We easily deduce again that .

Now, let us consider the case where and prove that g.f. for pattern is actually equals for pattern . To do this we consider a path having at least one occurrence of and such that . We can write either as () or () where .

Case (). We assume and . Since the pattern is not (because ), we necessarily have which means that avoids . Therefore, the pattern occurs once in and it straddles and . We deduce that belongs to and it has at least one occurrence of such that . So in this case, we define the function with .

Case (). We assume and . If then we have and using the same argument as previous, belongs to and it has exactly one occurrence of straddling and , and such that . In this subcase, we set . If then the occurrence of in is at the beginning or at the end (or touches both extremities) of since it must touch the -axis, but the occurrence does not straddle and . This means that avoids and either avoids (i.e. , as ) or contains with . In this subcase, we recursively define as . It belongs to and has at least one occurrence of such that . So, is a one-to-one correspondence between paths in having at least one occurrence of such that and paths having at least one occurrence of such that . So, we have .

Since for , finally we have for , and Equation (3) implies for . ∎

### 3.2 Skew Dyck and Motzkin paths

Let us discuss the skew Dyck paths first. For a given pattern , Condition () implies that for we have

 Ak(x)=x⋅Ak−1(x)⋅(k∑i=0Ai(x)+1)

which is a particular case of Equation (3) for , , , , and . Then, we deduce the following.

###### Corollary 3.2.
 A(x)=uvx2+v2x2−x2u−xu−x2+1−√Δ2x2(1+v+u) (7)

with

The following table gives the first values of generating functions obtained for all patterns of length at most two. Notice that none sequences appear in .

Statistic OEIS
, , , , 1, 3, 8, 23, 68, 211, 668, 2169, 7145
, 1, 3, 9, 28, 91, 307, 1062, 3748, 13429
1, 3, 9, 29, 96, 327, 1136, 4014, 14365
1, 3, 9, 27, 82, 255, 813, 2655, 8847
1, 3, 10, 35, 126, 463, 1728, 6529, 24916
1, 3, 10, 35, 128, 485, 1890, 7531, 30545

In the context of skew Motzin paths we have

 Ak(x)=x2⋅Ak−1(x)⋅(k∑i=0Ai(x)+x⋅A0(x)+1)

which is a particular case of Equation (3) for , , , , which means that when and otherwise. Then, the generating function is the generating function for skew Dyck paths (see Equation (7)) applied on the variable ( is not replaced with in and ), and we obtain the following results for patterns of length at most one. The sequences do not appear in .

Statistic OEIS
, 1, 2, 4, 9, 20, 45, 101, 229, 524, 1211, 2820
1, 2, 4, 10, 23, 55, 131, 318, 774, 1899, 4678
1, 2, 5, 11, 27, 64, 157, 383, 946, 2347, 5854
1, 2, 5, 12, 30, 76, 196, 513, 1359, 3639, 9831

## 4 Conclusion and research directions

Except Dyck and Motzkin cases, where we obtain two sequences appearing in  for the pattern , all the others have never been studied in the literature. So, this work provides a new catalog of sequences which is a possibly fertile ground in number theory, and more precisely in the analyse of their modular congruences (see  for instance). Another research direction could be the investigation of the distribution of a given pattern in the set . For instance, if in the context of Dyck paths, then the paths in avoiding the pattern are enumerated by the generalized Catalan numbers (see A004148 in ), which also count peak-less Motzkin paths. It would be interesting to find a one-to-one correspondence between these objects.

In this paper, we focus on the generating of , but we do not deal with the generating functions for paths such that since they can be easily obtained. Observing these g.f. for small , we have identified some of them that are already known in OEIS . For instance, the number of -length Dyck paths such that is the classical number of Fibonacci minus one (see A000071 in ). Such a result suggests us to look for new bijections with known classes counted by this sequence.

Finally, it would be interesting to extend this work to other classes of objects counted by generating functions satisfying System (3).

## Acknowledgements

This work was partly supported by the project ANER ARTICO financed by Bourgogne-Franche-Comté region.

## References

•  J.-L. Baril, S. Kirgizov, and A. Petrossian, Dyck paths with a first return decomposition constrained by height, Discrete Math., 341 (2018), 1620–1628.
•  J.-L. Baril, S. Kirgizov, and A. Petrossian, Motzkin paths with a restricted first return decomposition, Integers: Electronic Journal of Combinatorial Number Theory, 19 (2019), A46.
•  C. Brennan and S. Mavhungu, Peaks and valleys in Motzkin paths, Quaestiones Mathematicae, 33 (2010), 171–188.
•  E. Deutsch, Dyck path enumeration, Discrete Math., 204 (1999), 167–202.
•  E. Deutsch and B. Sagan, Congruences for Catalan and Motzkin numbers and related sequences, J. Number Theory, 117 (2006), 191–215.
•  R. Flórez, L. Junes, and J. Ramírez, Enumerating several aspects of non- decreasing Dyck paths, Discrete Math., 342 (2019), 3079–3097.
•  Q. L. Lu, Skew Motzkin paths, Acta Math. Sinica, 33 (2017), 657–667.
•  T. Mansour, Counting peaks at height in a Dyck path, J. Integer of Sequences, 5 (2002), Article 02.1.1.
•  T. Mansour, Statistics on Dyck paths, J. Integer Sequences, 9 (2006), Article 06.1.5.
•  T. Mansour, Combinatorics of set partitions, CRC Press, 2012.
•  T. Mansour, E. Deng, and R. Du, Dyck paths and restricted permutations, Discrete Applied Math., 154 (2006), 1593–1605.
•  D. Merlini, R. Sprugnoli, and M. Verri, Some statistics on Dyck paths, J. Statis. Plann. Inference, 101 (2002), 211–227.
•  N. Sloane, The on-line encyclopedia of integer sequences, Available electronically at http://oeis.org.