Plane partitions were a hot topic back in the 1970’s and 1980’s (as beautifully described in ), and they still keep combinatorialists busy. For example, the -enumeration formula of totally symmetric plane partitions, conjectured independently by David Robbins and George Andrews in 1983, remained open for almost 30 years and was finally proved in 2011  using massive computer algebra calculations. The problem that we treat in this paper originates around the same time, when combinatorialists started to employ determinants to reformulate the counting problem of plane partitions.
The following determinant counts descending plane partitions, and it was famously evaluated by George Andrews  in 1979:
where denotes the Kronecker delta, i.e., if and otherwise. The same determinant is also mentioned in Krattenthaler’s classic treatise on determinants [11, Thm. 32] (where is replaced by ). One year later, in 1980, Andrews [3, page 105] came up with a curious determinant which is a slight variation of the above:
He conjectured a closed-form formula for the quotient . It was mentioned again (and popularized) as Problem 34 in Krattenthaler’s complement , and it was proven, for the first time, by the authors of the present paper in 2013 .
However this proves only “half” of the formula for . The quotient remained mysterious, due to an increasingly large “ugly” (i.e., irreducible) polynomial factor that is always shared between two consecutive determinants. Thus the determinant does not completely factor into linear polynomials, while many similar determinants do. Not fully satisfied with this situation, the first-named author made a monstrous conjecture [9, Conj. 6] of the full formula for . In this paper, we derive and prove a nicer formula for (Section 4) and also show that it is equivalent to our previous conjecture (Section 5). In order to obtain the nice formula for , we have to evaluate some related determinants (Section 3), which we then combine via the Desnanot-Jacobi-Dodgson identity. In Section 6, we identify these determinants as special cases of some more general (infinite) families of determinants and present several theorems and conjectures for their closed forms. All of them have a combinatorial meaning, as will be explained in Section 2. We first introduce the main object of study of this article, the generalized determinant with shifted corner:
For , , and an indeterminate, we define to be the following -determinant:
Note that Andrews’s determinant is a special case of it, namely , and that (1.1) equals after replacing by .
We employ the usual notation for the Pochhammer symbol (also known as rising factorial), that is defined as follows:
The short-hand notation is to be interpreted as . The double factorial is defined, as usual, as
2 Combinatorial Background
Before we go into details about the evaluations of the mentioned determinant , and more generally , we want to give a combinatorial interpretation of these determinants, namely we exhibit certain combinatorial objects (rhombus tilings) that are counted by them.
The determinant , which is given in (1.1), was evaluated by George Andrews , because it counts descending plane partitions. Christian Krattenthaler  observed that it equivalently counts cyclically symmetric rhombus tilings of a hexagon with a triangular hole, where the size of the hole is related to the parameter [5, Thm. 6]. From this, we deduce that our generalized version can count similar objects. Throughout this section, we use the transformed parameter , which turns out to be more natural in this context (compare also with Andrews’ paper ).
The first observation is that can be written as a sum of minors. For this purpose, we rewrite it by performing index shifts on and :
For the sake of readability, we abbreviate the latter binomial coefficient by , and do not denote the dependency on and . Let be such that , i.e. the -th row contains one entry where the Kronecker delta evaluates to , then by Laplace expansion with respect to the -th row one obtains
where denotes the -minor of the corresponding matrix. More generally, for any matrix , we can write , where denotes the matrix after subtracting from its -entry. Applying this formula recursively to the determinant , until all ’s coming from the Kronecker deltas are eliminated, yields the following identity
where and where denotes the matrix that is obtained by deleting all rows with indices in and all columns with indices in from the matrix . In other words, we are summing over all subsets of positions where the Kronecker delta evaluates to , and for each such subset we add or subtract the corresponding minor .
The second observation is that, by the Lindström–Gessel–Viennot lemma [14, 6], counts -tuples of non-intersecting paths in the integer lattice : the start points are , , …, , the end points are , and the allowed steps are and ; see Figure 1 (left) for an example. The number of paths starting at and ending at is given by , which is precisely the -entry of . Note that this counting is only correct if ; in the following we will assume that this condition is satisfied. We do not know of a combinatorial interpretation when .
If then counts the -tuples of non-intersecting paths where the start points with indices and the end points with indices are omitted. In the case , the expression counts all tuples of non-intersecting paths for all subsets of start points (and the same subset of end points). If then we use with . This means that we never omit the last start points on the horizontal axis and we never omit the first end points on the vertical axis (counted from bottom to top). Moreover, the omitted start and end points follow the same pattern, shifted by . If then we never omit the first start points and the last end points.
The third and final observation is that the previously described non-intersecting lattice paths are in bijection with rhombus tilings of a lozenge-shaped region, where certain triangles on the border are cut out. They correspond to the start and end points; see the right part of Figure 1 where these triangles are colored black. The two types of steps (left and up) correspond to two orientations of the rhombi (colored white and light-gray), while rhombi of the third possible orientation (colored dark-gray) fill the areas which are not covered by paths. From Figure 1 it is apparent that the lozenge has width and height , and that black triangles are placed at the right end of its lower side and another black triangles at the top of its left vertical side. From the bijection with lattice paths we see that the number of rhombus tilings of such a lozenge is given by the determinant .
In order to give a combinatorial interpretation to the determinant , we have to sum up the counts of many similar tiling problems, according to the sum of minors (2.1). More precisely, label the black triangles on the lower side of the lozenge with numbers from to (from left to right), and similarly those on the vertical side (from bottom to top). Then counts rhombus tilings of the lozenge where all black triangles on the lower side with labels in are removed, and similarly, all black triangles on the vertical side with labels in . Instead of adding up the results of many counting problems, we can elegantly obtain the same result from a single counting problem, by introducing cyclically symmetric rhombus tilings of hexagonal regions.
For this purpose, we rotate the lozenge by and by and put the three copies together such that corresponding triangles share an edge. We illustrate this procedure in Figure 2: on the left we show the three copies of the lozenge from Figure 1 with parameters , , , and . Since we never omit the last two start points and the first two end points. Therefore, the corresponding triangles are colored black. The fact that the remaining start and end points may be omitted, is indicated by lighter colors. The relation between and is visualized by matching colors: for two triangles of the same color we have that either both are present or both are omitted. The three copies of the lozenge are glued together such that triangles of the same color share an edge. Note that this implies that none of the black triangles will have a partner.
Now we obtain a region that is either a hexagon (if ) or that otherwise has the shape of a pinwheel; see Figure 3. In both cases, there remains a “hole” in the center, except when . If then this hole has the shape of an equilateral triangle of side length , pointing to the right if and pointing to the left if . We have to ensure that no rhombus crosses the border of the original lozenge except for those positions that correspond to the start and end points of the paths. For this reason, we place a “border line” of length at each corner of the triangular hole and prohibit any rhombus to lie across this border. Note that in the case the vertical border actually starts at the lower vertex of the (left pointing) triangular hole, so that unit segments of the border coincide with the right side of the triangular hole (and similarly for the other two border lines). Each of these border lines is continued by unit triangular holes that point either in clockwise direction (if ) or in counter-clockwise direction (if ). The same number of triangles appears at the “wings” of the pinwheel, at a distance of from the end of the border line; these triangles point in the opposite direction.
Since we have now three copies of the original domain, we have to avoid overcounting: this is done by restricting the count to rhombus tilings that are cyclically symmetric. At the same time this restriction automatically ensures that the relation between start and end points is satisfied, namely that they are distributed in the same manner, only shifted by , as described before.
By construction, we have obtained a region whose cyclically symmetric rhombus tilings are counted by the determinant , provided that is even. If is odd, the count is weighted by and : the sign is determined by the parity of the number of rhombi crossing the original vertical side of the lozenge. Recall that the sign comes from in (2.1). The cardinality corresponds to the number of vertical line segments between the two vertical strips of black triangles that are “visible”, i.e., that are not covered by a horizontal rhombus. In other words: if there is an even number of line segments that are not crossed by a horizontal rhombus then the count is weighted with , otherwise with . By a “horizontal rhombus” we mean one that is built of two triangles sharing a vertical edge.
The construction can be simplified by noting that a row of small triangular holes induces a unique rhombus tiling when completing it to a big equilateral triangle. Hence the pinwheel-shaped region can be replaced by a hexagon, by cutting off three equilateral triangles of size , without changing the number of rhombus tilings. Similarly, the holes inside the region can be re-interpreted as four triangular holes, of size resp. , that are connected by boundary lines. We give an illustration of these regions in Figure 4.
As an example, we have worked out all cyclically symmetric rhombus tilings of the hexagon that corresponds to with ; see Figure 5. In this case, one can easily calculate
Another example that illustrates our combinatorial construction is the identity
that follows directly by the mirror symmetry of the underlying tiling regions. Assuming , the determinant counts cyclically symmetric rhombus tilings of a hexagon that has a triangular hole of size pointing to the right, with border lines of length , to each of which another triangular hole of size is attached, pointing in clockwise direction if . When we consider the transformed parameters , , and , we obtain a hexagonal region with a hole of size pointing to the left, with border lines of length , each of which shares a segment of length with the hole (so only units are visible), and with three other triangular holes of size each, pointing in counterclockwise direction if ( ). Thus these two regions are symmetric w.r.t. to a vertical axis and therefore possess the same number of rhombus tilings.
3 Related Determinants
In this section, we prove a few easier results about particular instances of the determinant , with specific shifted corners, by using computer proofs. Later, we put all these results together and obtain from it a “closed-form” formula for , via the celebrated Desnanot-Jacobi-Dodgson identity: let be a doubly infinite sequence and denote by the determinant of the -matrix whose upper left entry is at , more precisely the matrix . Then:
For an excellent overview of this topic see .
Let the determinant be as in Definition 1. Then the following equation holds:
In the following we state five lemmas with computer proofs, concerning special cases of the general determinant . They are employed afterwards to obtain closed-form formulas for the determinants , and ; see Propositions 8, 9, and 10, respectively. These in turn will be used in the main formula for in Section 4.
for all integers .
In order to prove that the determinant vanishes, we exhibit a concrete nontrivial linear combination of the columns of the matrix:
where the coefficients are rational functions in . For all the nullspace of has dimension , and it seems likely that this is the case for all . However, we need not care whether this is true or not, the important fact is that the coefficients for and are determined uniquely if we impose . Hence they are easily computed by linear algebra, and we can use these explicitly computed values to construct recurrence equations satisfied by them (colloquially called “guessing”). Now we consider the infinite sequence that is defined by these recurrence equations, subject to initial conditions that agree with the explicitly computed . We want to show that for all
the vectorlies in the kernel of (so far we only know this for up to ). This reduces to proving the holonomic function identity
for all integers .
Note that is basically the same determinant as (1.1) (upon replacing by ). Its evaluation was first achieved by George Andrews . The above statement is a corollary of his result, so there is nothing to prove. Just for completeness, and to show that all statements presented here can be treated with the same uniform approach, we give also a computer algebra proof in . ∎
We employ computer algebra methods to prove the statement, following Zeilberger’s holonomic ansatz . The overall proof strategy is similar to the one in Lemma 3: using an ansatz with undetermined coefficients (“guessing”) we find the holonomic description of an auxiliary bivariate sequence that certifies the correctness of the statement. In contrast to Lemma 3, the statement we want to prove implies that the determinant is nonzero, and hence we shall not succeed in finding a nonzero vector in the nullspace of the corresponding matrix. Instead, we delete its last row and consider the nullspace of the obtained -matrix, and proceed as in the proof of Lemma 3: for concrete small compute a vector of length that spans this (one-dimensional) nullspace, normalize it such that its last component equals , and construct bivariate recurrence equations satisfied by this data. This holonomic description (recurrences plus finitely many initial values) uniquely defines an infinite sequence . We use the HolonomicFunctions package  to prove some general properties and identities of this sequence.
First, we show that holds for all , by constructing a linear combination of our recurrences (and possibly their shifted versions) in which only terms of the form occur. Substituting yields a recurrence for the univariate sequence and we can verify that the constant sequence is among its solutions.
Second, we prove the following summation identity, where we denote by the -entry of :
It follows by linear algebra that is closely related to the -minor of the matrix of :
Third, one observes that the with are the cofactors of the Laplace expansion of with respect to the last row, divided by , which implies that
Hence, the proof is concluded by proving that this sum equals the asserted quotient of Pochhammer symbols. The proofs of the summation identities are carried out with HolonomicFunctions, and the details of these computations are contained in the electronic material . ∎
The proof is analogous to that of Lemma 6; details can be found in . However, we want to point out one issue that we encountered in the computations: In the guessing step we had to omit some of the data, as it was inconsistent with the rest of the data. More concretely, the recurrences we found were not valid for at . For the rest of the proof, this is irrelevant, but being unaware of this issue, one could get the impression that no recurrences exist at all. This phenomenon is explained by the fact that for the Kronecker delta does not appear in the matrix, and hence this case is somehow special. (For the same reason, we have the condition in Corollaries 22 and 23, for example.) ∎
We have , in other words , where
from which the claimed formula follows by using Theorem 2. The claims, and , were stated in Lemma 3 and Lemma 4. If is odd, the result is a direct consequence of Lemma 5. For the product formula, note that . ∎
We have where
We have where
As an aside, we want to mention that our original plan was to use the quotient of the two consecutive determinants and , which also factors nicely. However, we did not succeed in applying the holonomic ansatz to solve this problem. More precisely, we were not able to guess a holonomic description for the corresponding . Nevertheless, using our other results, we can now state:
4 Nice Closed Form for
From Propositions 8, 9, 10 we have now the values of , and at our disposal, and we will use them to derive, for the first time, a kind of a closed-form for the mysterious determinant . In Figure 5 it is shown what kind of rhombus tilings are counted by . Once again, we will use the Desnanot-Jacobi-Dodgson identity (DJD) (see p. DJD) to glue the previous results together. By doing so, we obtain a recurrence equation for :
We replace with , divide by , and apply Proposition 8:
Splitting into even and odd is reasonable, since it is anyway defined differently for these cases. Now, by unrolling the recurrence, we get a “closed form”, namely an explicit single sum expression, for :
First, we investigate the factor inside the product:
By taking the product of this last expression, we get the asserted formula. ∎
Let be an indeterminate and let be defined as in Definition 1. Let be defined as and for . If is an odd positive integer then
If is an even positive integer then
Starting from formula (4.1) we want to derive the asserted evaluation of the determinant . By noting that we can write