    # Pictures of Combinatorial Cubes

We prove that the 8-point algorithm always fails to reconstruct a unique fundamental matrix F independent on the camera positions, when its input are image point configurations that are perspective projections of the vertices of a combinatorial cube in R^3. We give an algorithm that improves the 7- and 8-point algorithm in such a pathological situation. Additionally we analyze the regions of focal point positions where a reconstruction of F is possible at all, when the world points are the vertices of a combinatorial cube in R^3.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1. Introduction

The 8-point algorithm [6, Algorithm 11.1] is one of the key algorithms in epipolar geometry. It is successfully used in a vast number of applications to compute the fundamental matrix between two views. The 8-point algorithm purely relies on methods from linear algebra and is extremely fast. At the same time it only gives slightly inferior results compared to more involved algorithms that rely on nonlinear optimization.

An algorithm to reconstruct the fundamental matrix is defeated by a world point configuration in , if this algorithm fails to produce a unique fundamental matrix from the projections in of that world point configuration independent on the choice of cameras. The 8-point algorithm is defeated by the eight vertices of a unit cube. We extend this result to arbitrary convex 3-polytopes bounded by six quadrilateral faces, whose vertex-facet incidence is the same as that of a cube. Such a polytope in is called combinatorial cube. The vertex-facet incidence of a polytope is the undirected bipartite graph formed by the containment of vertices within the facets [8, §3.5].

In the last section of this paper we suggest how to handle the situation when the 8-point algorithm is defeated by the vertices of a combinatorial cube by using a modified version of the 7-point algorithm in this case.

There are multiple papers in multiview geometry which are concerned with critical configurations [10, 5]. Critical configurations in two-view geometry are point configurations consisting of world points together with the two focal points of the cameras. For these configurations there exist two ambiguous fundamental matrices. They are a feature of the geometry itself and not of the choice of algorithm used to reconstruct the fundamental matrix . Hartley and Kahl  give the complete description of critical configurations in multiview geometry. In the two-view case a necessary condition for a focal-world point configuration to be critical, is that the world points and the two camera centers lie on a ruled quadric . Thus it is not possible to reconstruct a unique fundamental matrix, independent of the chosen algorithm. Here we consider world point configurations where the 8-point algorithm always fails to reconstruct the fundamental matrix independent of the camera centers. These configurations are related to critical configurations but they do not align. There are only a few results about these configurations and usually rely on dimensional degeneracies, e.g. too many points on a plane or on a line. In this case the quadric running through the points and the two cameras centers is degenerate. For example in  it is shown, that all points but one on a plane defeat the 8-point algorithm. If we add the two focal points to this configuration then the point off the plane and the focal points span a plane. Hence we can fit a ruled quadric of two intersecting planes through the points.

By we denote the -th point in a point configuration of points in .

Pinhole cameras are represented as matrices with real entries. If two image points in two different pictures are perspective projections of the same world point they must satisfy the perspective relation with . As these equations must be satisfied at the same time one can deduce a bilinear relation which both points must satisfy.

 YTiFXi=0,

where is the fundamental matrix. To reconstruct the fundamental matrix from the image point configurations different algorithms are available [6, §11]. The most commonly used algorithm to reconstruct the fundamental matrix is the 8-point algorithm [6, Algorithm 11.1]. However standard implementations of the 8-point algorithm assume that the image point configurations are suitably generic, such that the fundamental matrix

can be determined as the solution of a system of linear equations. For exact data the 8-point algorithm then is as follows. Via vectorizing

denoted by it can be computed as the kernel of the matrix

 Z=⎡⎢ ⎢⎣X1⊗Y1⋮Xn⊗Yn⎤⎥ ⎥⎦∈Rn×9,

where

is the row wise tensor product of the image point configurations

. The kernel of is one-dimensional and . This yields the fundamental matrix . One necessary condition for the 8-point algorithm to successfully compute a fundamental matrix is

 (1) dim(ker(Z))=1.

Algorithm 1 states the 8-point algorithm in the absence of noise.

However, even if condition 1 is violated and the 8-point algorithm fails to construct a fundamental matrix, it is sometimes possible to retrieve a unique fundamental matrix using a different algorithm by additionally enforcing the rank two constraint of the fundamental matrix , namely if there exists a unique rank two matrix in the kernel of .

###### Example 1.1.

Let be a standard cube and

 A1=⎡⎢⎣100201030012⎤⎥⎦,A2=⎡⎢⎣100201030011⎤⎥⎦

then

 X=⎡⎢⎣131313132244224411113333⎤⎥⎦,Y=⎡⎢⎣131313132244224400002222⎤⎥⎦

and

 Z=\footnotesize⎡⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢⎣1212420009636420001414164000912312164000123246246969646646143416122869129121612686⎤⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥⎦

The kernel of is two-dimensional and the 8-point algorithm is defeated by the vertices of . However, there is only one matrix in that is of rank two

 F=⎡⎢⎣010−100000⎤⎥⎦

and a unique reconstruction of is possible.

## 2. The 8-Point Algorithm and a Cube

A quadric in is defined by the algebraic equation , where and . Since this is a quadratic equation in the indeterminate we can choose the matrix to be symmetric. Clearly and any multiple of it define the same quadric. By a slight abuse of notation we will refer to both the quadric and the matrix defining the quadric as .

One can try to fit a quadric through a point configuration , then every point in the configuration gives a linear equation on the ten entries of the symmetric matrix . This results in a linear equation system with indeterminate vector

 \vvQ=[Q00,…,Q04,Q11,…,Q14,Q22,Q23,Q24,Q33,Q34,Q44].

Its coefficient matrix can be constructed with the Veronese map. The Veronese map in degree two and four indeterminates is the map from the four indeterminates to all monomials of degree two in these indeterminates.

 ν2,4:P3→P9[x1,x2,x3,x4]↦[x21,x1x2,x1x3,x1x4,x22,x2x3,x2x4,x23,x3x4,x24].

For convenience of notation we apply the map to each point in a configuration separately. The map applied to gives a matrix . Therefore, if there exists a quadric through the points in a configuration , it can be computed via the linear equation system .

The rank of is very essential to the 8-point algorithm and there is a relation to .

###### Lemma 2.1.

Let , be two cameras and be a world point configuration. Let and , then the rank of is bounded by the rank of .

###### Proof.

The statement can be rewritten. The number of linear independent equations in the system of linear equations is bounded by the number of linear independent equations in the system of linear equations , where is a generic symmetric matrix of indeterminates. Without loss of generality we assume that is of rank and the first equations are independent. Setting we can write every point in the span of as and by the independence of the first equations we get

 yTFx=∑kλiPTiQPi=∑kλiyTiFxi. Figure 2. The chosen labeling of word points and camera centers.

Let represent a configuration of ten points. The question whether the ten points of

are inscribable to a quaternary quadric has been studied in classical algebraic geometry. However, no geometric interpretation is known up till now and it probably would be too complicated to be of any use. In algebraic terms this condition can easily be phrased as

. Turnbull and Young give a description of the invariant in the bracket algebra (Turnbull-Young invariant) . Later the Turnbull-Young invariant has been straightened to a bracket polynomial of degree 5 with 138 monomials . Using the vertex labeling of Figure 2 the Turnbull-Young invariant given in [14, p. 8-9] reduces to the bracket polynomial

 (2)

where denote the points in the configuration .

We use this invariant to show that is not of full rank if the points in are the vertices of a combinatorial cube.

###### Proposition 2.2.

Let be the vertices of a combinatorial cube in , then the rank of is at most seven.

###### Proof.

Consider the equation system . These are the equations concatenated with the two equations of two arbitrary points . If the rank of is at most seven, then the rank of equation system is at most nine. Thus the ten points of the configuration are in special position and are inscribable to a quarternary quadric. This is equivalent to satisfying the Turnbull-Young invariant. We checked with Macaulay2  that the polynomial of Equation 2 vanishes for all combinatorial cubes with the eight vertices independent of the choice of the two points and .∎

Without using some speed-ups and simplifications solving Equation 2 is computationally out of reach. Thus we performed the computation in Macaulay2  as follows. Since we are only interested in points in we fixed the fourth coordinate of every point to one. Further we have the freedom of choice of a coordinate system in . We choose the point as the origin and the points , , as the three unit vectors. This implies that is on the -plane, is on the -plane and is on the -plane. In the Listing LABEL:algoCritical one can find the Macaulay2  code used to check Proposition 2.2.

###### Remark 2.3.

Various statements about eight points and quadric surface are known in classical algebraic geometry, like the three-dimensional version of Miquel’s Theorem [1, p. 18] and the statements about eight associated points .

Due to Proposition 2.2 we understand the behavior of the 8-point algorithm if its input configurations are images of the vertices of a combinatorial cube.

###### Theorem 2.4.

Let be two arbitrary cameras and let be the vertices of a combinatorial cube. Then the -point algorithm with input fails to compute a fundamental matrix . It is defeated by the vertices of .

###### Proof.

By Proposition 2.2 the matrix is of rank seven at most, thus by Lemma 2.1 the matrix is of rank seven at most. Hence the assumption in the -point algorithm that is of rank eight at least is not satisfied. ∎

Theorem 2.4 states that if we take two pictures from the vertices of a combinatorial cube, then the matrix has at most rank seven and thus the 8-point algorithm is not able to compute the fundamental matrix.

###### Remark 2.5.

Unlike to the conditions on critical configurations Theorem 2.4 does not impose any constraints on the camera centers.

## 3. Reconstruction of F From Cubes

Even if it is sometimes possible to reconstruct the fundamental matrix . Since the matrix is of rank at most seven one can try to reconstruct the fundamental matrix by additionally enforcing the singularity condition of the fundamental matrix. Solving for means finding the real roots of a univariate cubic polynomial. However for certain regions in of the two focal points, this polynomial has more than one solution and a unique reconstruction of is not possible. These regions are semi-algebraic sets. We study these by first studying the simplest case, when are the vertices of the unit cube .

For the three-dimensional unit cube the matrix defining the family of quadrics through its vertices diagonalizes to , , such that . This results in a two parameter family of quadrics running through the eight vertices of . If we include the two camera centers then there is exactly one quadric running through all ten points and it is given as the solution of the linear equation system

 (3) tr(Q)=0cT1Qc1=0cT2Qc2=0⇔⎡⎢ ⎢⎣1111x21x22x23x24y21y22y23y24⎤⎥ ⎥⎦:=M⎡⎢ ⎢ ⎢ ⎢⎣αβγδ⎤⎥ ⎥ ⎥ ⎥⎦=0

where and . Figure 3. Region of failure (orange) of the 8-point algorithm (enforcing the singularity constraint) with a unit cube as input. We fixed the focal point of the first camera and randomly sampled the focal point of the second camera in a chosen plane.

By Cramer’s rule we construct the solution of this linear equation system, as the vector of the four signed maximal minors of the matrix M,

 (4) ⎡⎢ ⎢⎣∣∣ ∣ ∣∣111x22x23x24y22y23y24∣∣ ∣ ∣∣,−∣∣ ∣ ∣∣111x21x23x24y21y23y24∣∣ ∣ ∣∣,∣∣ ∣ ∣∣111x21x22x24y21y22y24∣∣ ∣ ∣∣,−∣∣ ∣ ∣∣111x21x22x23y21y22y23∣∣ ∣ ∣∣⎤⎥ ⎥⎦.

We denote by the coordinate wise square of the vector . Then Equation 4 can be written as the cross product of three vectors .

Let be the quadric running through the vertices of a combinatorial cube and the two camera centers . Further let be the quadric running through the vertices of the unit cube and the cameras centers .

###### Proposition 3.1.

If there is a projective transformation from to , then has the same sign pattern as and have the same sign pattern.

###### Proof.

Let be the vertices of together with and be the vertices of together with .

 (PTuQuPu)ii=0∀i⇔(PTTTQuTP)ii=0∀i⇒TTQuT=Q.

Now by Sylvester’s law of inertia and have the same sign pattern. ∎

From Proposition 3.1 we are able to compute the type of the quadric by finding a projective transformation that maps the vertices of to the vertices of . In particular is ruled if is ruled.

###### Remark 3.2.

If we interprete the vector of diagonal entries of as a point in the condition is equivalent to being on the intersection of the three planes with normal vectors .

The quadric is ruled if has a sign pattern of the following types , , or . The boundaries of the components of the semi-algebraic set (where the signature of the quadric changes) are given as the vanishing set of the determinants , , and of Equation 4.

In some cases the signature changes, but still the quadric stays non-ruled. Therefore, to get a more explicit answer for this example it is useful to break up the symmetry of Equation 4. Since the quadric is independent on a scaling factor of we can set the last diagonal entry without loss of generality to . Thus the equation system of Equation 3 degenerates to an equation system of three equations in three variables and we can solve it explicitly. Then . There are two distinct cases when the quadric is ruled:

1. If and .

2. If have different signs and .

The vector of diagonal entries of then is given up to scale by

 \footnotesize⎡⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢⎣∣∣∣x22−x23x23−1y22−y23y23−1∣∣∣∣∣∣x21−x22x23−1y21−y22y23−1∣∣∣−∣∣∣x22−x23x23−1y22−y23y23−1∣∣∣−∣∣∣x21−x22x23−1y21−y22y23−1∣∣∣−∣∣∣x21−x22x22−x23y21−y22y22−y23∣∣∣∣∣∣x21−x22x22−x23y21−y22y22−y23∣∣∣⎤⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥⎦∈P3

## 4. How to Handle Pictures of Cubes

In the sections above we did not consider noisy pictures. As seen in Theorem 2.4 the rank of drops by at least one if we take pictures of combinatorial cubes. However, in the presence of noise the matrix again is of rank eight, but it is close to being singular. This results in a very bad performance of the algorithm in practice and we strongly advise against using it. It is simply the wrong choice of algorithm, since it is incapable of dealing with this set-up.

As discussed in Section 2 the matrix has at most rank seven in the noisefree case, hence a standard implementation of the -point algorithm run on a 7-element subset of vertices can be used to retrieve the fundamental matrix. Thus a natural fix for the flaw of the -point algorithm is to use a modified version of the

-point algorithm, that allows eight points as inputs. We use singular value decomposition on

to obtain a matrix that is of rank seven and minimizes the Frobenius norm of . Let be the singular value decomposition of , then by the Eckart-Young-Mirsky theorem . Now we use the -point algorithm to obtain (one to three) possible solutions for the fundamental matrix. If there are multiple solutions, we chose the one that minimizes the residual error on the input points. The -point algorithm for cubes then is given in Algorithm 2.

## 5. Numerical Experiments

We performed random tests on synthetic data to compare the performance of different algorithms. To do so, we sampled random cubes within the box

. The cameras were chosen with focal points roughly on a sphere with radius six. Gaussian noise was applied onto the images with standard deviation between

of the image sizes and zero mean. For each noise level we chose 2000 random samples and respectively computed 2000 approximations of fundamental matrices. As a measure to analyze the results of the algorithm we used the metric on the Grassmanian between two linear subspaces, namely the angle between the vectorization of the true fundamental matrix and the approximated fundamental matrix. Figure 4. Comparison between algorithms to reconstruct the fundamental matrix. Plotted is the distance as angle (rad) between two one-dimensional subspaces versus the noise level.

The modified version of the -point algorithm for cubes (Algorithm 2) gives good results. Its running time is almost the same as of the usual

-point algorithm, but unlike the unmodified version it is noise correcting. It also gives better results than algorithms that rely on non-linear optimization to estimate the fundamental matrix, like the

Gold Standard algorithm [6, Algorithm 11.3]. These algorithms usually use an initial guess of the fundamental matrix computed via the -point algorithm. But the results of the -point algorithm are so far off from the true fundamental matrix, such that the non-linear solvers get stuck in a local optimum far off from the global one. However by using estimates of the fundamental matrix computed with the modified -point algorithm for cubes (Algorithm 2) the Gold Standard algorithm can be improved. For example using an estimate of the fundamental matrix computed with Algorithm 2 as initial input, instead of a fundamental matrix computed with the 8-point algorithm [6, Algorithm 11.1], improves the Gold Standard algorithm [6, Algorithm 11.3]. In Figure 4 this version of the Gold Standard algorithm is denoted by Cube Gold Standard.

Note that there are also global solvers to find fundamental matrices based on semidefinite programming . Noteworthy the situation in our case is different from the one depicted . Usually there are more than 8 correspondent image point pairs available. In  ten points and more are considered. Thus due to noisy data the matrix is of rank nine. For the pathological case of only eight points and a rank drop in , Algorithm 1 in  has not been able to certify global optimality based on GloptiPoly 3 .

If one has the freedom of choice to place the eight points in

we suggest using the skew octagon for a more robust reconstruction of the fundamental matrix. The skew octagon is computational the optimal solution to various sphere placement problems, e.g. the Thomson problem

.

### 5.1. Acknowledgment

We would like to thank Michael Joswig for his guidance and Fredrik Kahl for our correspondences about critical configurations.

## 6. Computations

Below you can find the code we used to check Proposition 2.2. Lines 2-11 define the vertices of a cube. Lines 14-17 define the facets of the cube. In lines 22-34 the reduced Turnbull-Young invariant of Equation 2 is defined. This invariant vanishes in the quotient ring of line 19.

1R=QQ[x_1..x_15]
2Cube=matrix{ {0,0,0,1},           --0
3             {x_1,x_2,0,1},       --1
4             {0,1,0,1},           --2
5             {1,0,0,1},           --3
6             {x_10,x_11,x_12,1},  --4 f_1
7             {x_13,x_14,x_15,1},  --5 f_2
8             {x_5,0,x_6,1},       --6
9             {0,x_3,x_4,1},       --7
10             {x_7,x_8,x_9,1},     --8
11             {0,0,1,1}}           --9
12-- the six facets of the cube are coplanar
13-- their determinants vanish
14J=ideal(
15det submatrix(Cube,{0,1,2,3},),det submatrix(Cube,{6,7,8,9},),
16det submatrix(Cube,{0,3,6,9},),det submatrix(Cube,{1,2,7,8},),
17det submatrix(Cube,{0,2,7,9},),det submatrix(Cube,{1,3,6,8},))
18
19S=R/J
20F=map(S,R)
21-- the remaining 4 bracket monomials of the Turnbull-Young invariant
22p=F(
23det submatrix(Cube,{0,1,3,5},)*det submatrix(Cube,{0,2,4,7},)*
24det submatrix(Cube,{1,2,6,8},)*det submatrix(Cube,{3,4,6,9},)*
25det submatrix(Cube,{5,7,8,9},)-
26det submatrix(Cube,{0,1,3,4},)*det submatrix(Cube,{0,2,5,7},)*
27det submatrix(Cube,{1,2,6,8},)*det submatrix(Cube,{3,5,6,9},)*
28det submatrix(Cube,{4,7,8,9},)+
29det submatrix(Cube,{0,1,2,5},)*det submatrix(Cube,{0,3,4,6},)*
30det submatrix(Cube,{1,3,7,8},)*det submatrix(Cube,{2,4,7,9},)*
31det submatrix(Cube,{5,6,8,9},)-
32det submatrix(Cube,{0,1,2,4},)*det submatrix(Cube,{0,3,5,6},)*
33det submatrix(Cube,{1,3,7,8},)*det submatrix(Cube,{2,5,7,9},)*
34det submatrix(Cube,{4,6,8,9},))

## References

•  Alexander I. Bobenko, John M. Sullivan, Peter Schröder, and Günter Ziegler, Discrete differential geometry, Springer, 2008.
•  Kevin Brown, Skew octagon solving the Thomson problem of eight points, Available at http://mathpages.com/home/kmath005/elec/elec8.htm, 2017.
•  Florian Bugarin, Adrien Bartoli, Didier Henrion, Jean-Bernard Lasserre, Jean-José Orteu, and Thierry Sentenac, Rank-constrained fundamental matrix estimation by polynomial global optimization versus the eight-point algorithm, Journal of Mathematical Imaging and Vision 53 (2015), no. 1, 42–60.
•  Daniel R. Grayson and Michael E. Stillman, Macaulay2, a software system for research in algebraic geometry, Available at http://www.math.uiuc.edu/Macaulay2/.
•  Richard Hartley and Fredrik Kahl, Critical configurations for projective reconstruction from multiple views

, International Journal of Computer Vision

71 (2007), no. 1, 5–47.
•  Richard Hartley and Andrew Zisserman, Multiple view geometry in computer vision, Cambridge University Press, 2003.
•  Didier Henrion, Jean-Bernard Lasserre, and Johan Löfberg,

Gloptipoly 3: moments, optimization and semidefinite programming

, Optimization Methods & Software 24 (2009), no. 4-5, 761–779.
•  Michael Joswig and Thorsten Theobald, Polyhedral and algebraic methods in computational geometry, Springer Science & Business Media, 2013.
•  Josef Krames, Zur Ermittlung eines Objektes aus zwei Perspektiven.(Ein Beitrag zur Theorie der “gefährlichen Örter”.), Monatshefte für Mathematik und Physik 49 (1941), no. 1, 327–354.
•  Stephen Maybank, Theory of reconstruction from image motion, vol. 28, Springer Science & Business Media, 2012.
•  Johan Philip, Critical point configurations of the 5-, 6-, 7-, and 8-point algorithms for relative orientation, Technical report TRITA-MAT-MA (1998).
•  Herbert Turnbull and Alfred Young, The linear invariants of ten quaternary quadrics, Trans. Camb. Phil. Soc 23 (1926), 265–301.
•  HW Turnbull, On the vector algebra of eight associated points of three quadric surfaces, Mathematical Proceedings of the Cambridge Philosophical Society, vol. 22, Cambridge University Press, 1925, pp. 481–487.
•  Neil White, Implementation of the straightening algorithm of classical invariant theory, Proceedings, Invariant Theory and Tableaux, IMA (1988).