1. Introduction
In many computer vision applications, the correspondences among views are unknown. Hence the world points in and their images points in will be unlabeled. The multiview variety [1] is a fundamental object in multiview geometry, it encodes the relations among image points of one world point in images taken by cameras. To study the problem of unlabeled data, we propose to work with the unlabeled multiview variety. This is the variety of products of multiview varieties [1] with unknown correspondences. An unlabeled point configuration in is a point in the Chow variety [6, §8.6]. Algebraically the unlabeled multiview variety is the image of the multiview variety under the quotient map for the symmetric group action. Our focus is mostly on two unlabeled points. We design Algorithm 2 to triangulate two unlabeled points.
While labeled world and image configurations are points in and , unlabeled image configurations are points in the Chow varieties which live as subvarieties in and . This is the variety of ternary forms that are products of linear forms (cf. [6, §8.6]), respectively quaternary forms that are products of linear forms. It is embedded in the space of all ternary forms of degree .
We start by giving a brief introduction to multiview geometry. A camera is a linear map from the three-dimensional projective space to the projective plane , both over . We represent cameras by matrices of rank . A world point is mapped by the perspective relation of the camera
to the image point . The kernel of is the focal point . Each image point of camera has a line through as its fiber in . This is the back-projected line. On that back-projected line lies the world point . A camera is called normalized if it is of the form
form some vector
, withbeing the identity matrix.
We assume throughout the paper that the focal points of the cameras are in general position, i.e. all distinct, no three on a line, and no four on a plane. Let denote the line in spanned by the focal points and . This is the baseline of the camera pair . The image of the focal point in the th-image plane of the camera is the epipole .
For three cameras the plane spanned by their focal points is called trifocal plane.
Computing the point of intersection of the back-projected lines to reconstruct is called triangulation[4, §9.1]. The triangulation can be based on multiple views and amounts to solve the linear equations
(1) |
For generic data the world point is represented by the first four entries of any element in the kernel of
. In practice this equation system is solved by a singular value decomposition and similar approaches
[3]. Such that a reconstruction of restricted to only two views is possible the image points must satisfy a bilinear relationThe matrix is called fundamental matrix of views and .
The multiview variety of the camera configuration was defined in [1] as the closure of the image of the rational map
(2) |
The points are the consistent views in cameras. The prime ideal of was determined in [1, Corollary 2.7]. It is generated by the bilinear polynomials plus trilinear polynomials. See [7] for the natural generalization of this variety to higher dimensions.
Remark 1.1.
Let be the matrix with the -row of the identiy matrix omitted. Then the multiview variety given by the four cameras is isomorphic up to a change of coordinate system to any multiview variety with four cameras in linearly general position.
We define the unlabeled multiview variety to be the closure of the image of the rational map
Then maps an order symmetric tensor to order symmetric tensors. For each world point these tensors have one axis. This gives rise to the unlabeled multiview variety in , where is the multiview variety. Let be two labeled world points. We denote their images in the -th picture as ,
Then and are the coordinates of the image points.
Example 1.2.
Let . The Chow variety is the hypersurface in defined by the determinant of a symmetric -matrix . From the two image points we can construct an unlabeled image point on the Chow variety , this is the symmetric matrix . Up to the symmetry of coordinates of , this reads as
The quotient map is given by the formulas above. Similarly, for the two unlabeled images of the the second camera we use
We compute the image of in , denoted . Its ideal has seven minimal generators, three of degree , and one each in degrees . The generators in degrees and are and . The five others depend on the cameras .
If we can use symmetric matrices in and to describe the unlabeled multiview variety, even for larger than two. This is computationally easier to handle. Then the unlabeled world point configuration of the two labeled world points can be represented by a symmetric rank two matrix . The unlabeled image point configuration of two labeled image points in the -th picture can be represented by the symmetric rank two matrix .
Remark 1.3.
Since are points in projective spaces the symmetric rank two matrices are only defined up to scale.
For a symmetric rank two matrix that is suitably generic, maps the matrix to symmetric of rank two matrices. Thus the unlabeled analog of the perspective relation for a pinhole camera reads as
(3) |
The unlabeled multiview map of two points is
(4) |
If the unlabeled multiview variety is the closure of the image of the rational map .
We will pay special attention to the algebraic variety of the closure of the image of in Section 3.
2. Relabeling the Unlabeled
The Chow variety is a tool to easier handle unlabeled points, in order to understand the geometry it is sometimes more convenient not to think about unlabeled points as symmetric tensors. While in the section above we viewed two unlabeled points as one point on the Chow variety , in this section we will stick to unlabeled points as points in and . Let be the map that takes a labeled point configuration in to its unlabeled configuration represented as a symmetric tensor, e.g. if , then takes and , both in , to the same symmetric matrix . The preimage of the unlabeled multiview variety under the map is the union of labeled multiview varieties with the labeling of their image points interchanged.
Example 2.1.
The image points are on the product of two two-view varieties if they satisfy
The image points are on the the closure of the preimage under of the unlabeled two-view variety of two points if they satisfy (s. Figure 2)
(5) |
This variety is cut out set-theoretically by the ideal
The approach of Example 2.1 extends to more pictures and more unlabeled world points. The idea here is to permute the labeling of the image points in each picture. Each permutation of image points gives a new product of multiview varieties with permuted coordinates. The union over all these product varieties then is the unlabeled multiview variety.
We construct a graph with all the image points as vertices, see Figure 3. Let be the -partite complete graph on vertices, where each partition of has cardinality . Let be a perfect -dimensional matching and by we denote the set of all such perfect -dimensional matchings on . For a fixed we denote , that is an image point in the -th picture, as the vertex in the -th partition of in the -th path of , with .
In this section we have not been working with unlabeled points in but with labeled points and their orbits under group action on the image points of the symmetric group. The following theorem illustrates the relation between these two approaches.
Proposition 2.2.
The closure of the preimage under of the unlabeled multiview variety is the union of products of multiview varieties, with permuted image points correspondences
Proof.
The usual multiview variety knows the labeling of the images points, however the unlabeled does not. Thus the unlabeled variety is nothing but the union of all multiview varieties with permuted image point correspondences. A permuted image point correspondence is an -dimensional matching through the images, as depicted in Figure 3. Taking the union over all such possible correspondences yields the preimage of the the unlabeled multiview variety. ∎
Remark 2.3.
The multiview variety is cut out set-theoretically by the bilinear constraints, if the cameras are in linearly general position [5]. Thus the unlabeled multiview variety is cut out by the products of bilinear constraints.
The usual two-view triangulation does result in either zero solutions (, are not on the two-view variety), one solutions (, are on the two-view variety) or a one dimensional linear subspace of solutions (, are the epipoles). The situation is a little different if we forget the labeling of the image points. Anyway, the triangulation problem still amounts in intersecting the back-projected lines. The reconstruction of the original world points is called unlabeled triangulation. In the following its ambiguities are studied. The unlabeled triangulation is ambiguous if there are two or more unlabeled world point configurations that project down to the same unlabeled image point configuration.
Proposition 2.4.
Let n=m=2. The unlabeled triangulation is not unique, if and only if the image points are on the unlabeled multiview variety and satisfy the two conditions
If the image points are distinct from the epipoles , then there are exactly two possible unlabeled world point pairs and that are possible reconstructions.
Proof.
By construction the arrangement of the four back-projected lines of have the focal points and as intersection points. Such that the triangulation is ambiguous the four back-projected lines must have at least four additional intersections. Thus the back-projected lines need to be coplanar by the Veblen-Young axiom. Because are on the unlabeled multiview variety this is equivalent to being collinear, as depicted in Figure 4. Now if are pairwise distinct and are pairwise distinct, then the back-projected lines have in total six intersections of which four are not the focal points. ∎
Remark 2.5.
Let n=m=2. In world coordinates the ambiguity of the unlabeled triangulation revolves around the world points , and the baseline through the focal points being coplanar. Hence the pencil of planes with the baseline as its axis encodes the ambiguities of the two view unlabeled triangulation.
The ambiguity of the unlabeled triangulation of two views and two unlabeled points extends to the case .
Corollary 2.6.
If a subset of the world point configuration and the baseline are coplanar, then the unlabeled triangulation is ambiguous.
Proof.
Follows immediately from Proposition 2.4. ∎
When a subset of unlabeled world points and the baseline are on one plane, then all their back-projected lines intersect each other pairwise. There are in total such intersection points. To construct an alternative world point configurations, each back-projected line has to be used exactly one time. Otherwise the resulting image point configuration does not align with the original image point configuration.
Remark 2.7.
Consider world points that are coplanar with the baseline, then there are different solutions to the unlabeled triangulation from two views.
For generic points on the unlabeled multiview variety the back-projected lines intersect in only two points and the reconstructing of the unlabeled world point configuration is unique.
When the back-projected lines represent a line arrangement in , where through each focal point lines pass. Let the intersection degree of a point in denote the number of back-projected lines that intersect in that point. The unlabeled triangulation amounts in finding points in with intersection degree equal to that are distinct from the focal points. A configuration of world points is a valid triangulation of the image points if it covers all back-projected lines.
In general classifying the ambiguities of the unlabeled triangulation is a complex problem.
For three views and three unlabeled points one can employ Pappus’s hexagon theorem to construct an ambiguous unlabeled triangulation. In this case a configuration that possesses an ambiguous unlabeled triangulation must lie on the trifocal plane. Figure 5 depicts such an situation on the trifocal plane.
3. Two Unlabeled Points
In this section we will only consider the case of two (m=2) unlabeled world points. The complicated part of the unlabeled multiview map of Equation 4 is the rank two constraint on the symmetric matrix . However if has rank two we can be sure that its projections have rank two.
Lemma 3.1.
The matrices have rank at most two if has rank two. If , then can be written as for some .
Proof.
If has rank two, then . The second statement is obtained from the fact
∎
Remark 3.2.
The reverse of the second statement of Lemma 3.1 does not hold
We first analyze by dropping the rank two constraints in . This gives us the rational map
We will denote the ideal and the variety of the closure of the image of as and . This variety is a relaxation of the unlabeled multiview variety . The map is linear in the entries of the symmetric matrix , thus by row-wise vectorizing the upper triangular entries of to the map can be rewritten in its standard form of linear equations, with the coefficient matrices . In the coordinates of the coefficient matrix then reads as
and the according rational map is
(6) |
This map strongly resembles the conventional multiview map of Equation 2 and can be understood as a higher dimensional analogue of it. We use the techniques developed in [7] for a more general setup to describe the closure of the image of in Corollary 3.7. But we first study the unlabeled triangulation. The results we obtain to understand the unlabeled triangulation will come in handy for Corollary 3.7.
Let with and , then define as
(7) |
Let the unlabeled focal point of the cameras and be .
Proposition 3.3.
Let . Let and be two generic points on the unlabeled two-view variety and denote their unlabeled triangulation . Then and there exists , such that
Proof.
Since and are generic points on the unlabeled multiview variety the triangulation is unique and is its solution. Hence there are , such that is in . On the other hand is equivalent to , thus inserting gives and inserting gives . Hence for the vector is in . ∎
Remark 3.4.
By the proof of Proposition 3.3 is spanned by .
Proposition 3.5.
Let . For a generic choice of cameras the matrix has full rank.
Proof.
The unlabeled focal point is the kernel of the submatrix . Since does not depend on the other cameras it is not in the kernel of any of the other for a generic choice of cameras. ∎
We can use the matrix to design a triangulation algorithm for the unlabeled multiview variety. We call this algorithm the unlabeled triangulation algorithm. The Proposition 3.3 indicates that the unlabeled triangulation is more complicated than the usual labeled triangulation. By Propositions 2.4 and Corollary 2.6 it can have multiple solutions. Especially the case of two views is more elaborate. Nonetheless it has a unique solution for generic points on the unlabeled multiview variety. We need a lemma before we can state the unlabeled triangulation algorithm.
Lemma 3.6.
Consider a two-dimensional linear subspace spanned by two symmetric matrices. Suppose that contains two distinct rank two symmetric matrices , such that . Then the symmetric matrices are the only rank two symmetric matrices in . Also the subspace contains no rank three symmetric matrix.
Proof.
We can choose and as basis of . The rank two matrices in can be computed as some roots of . Clearly zero and one are roots of this univariate polynomial. However the roots of
are the generalized eigenvalues to the matrix equation
. But since has rank two the generalized eigenvalue has double algebraic and geometric multiplicity. Then switching the roles of and shows that also has double algebraic and geometric multiplicity. ∎The unlabeled triangulation of two views results in finding the rank two matrices in linear two-dimensional subspaces. That subspace is constructed from . We can find these matrices by finding the roots of a univariate quartic polynomial, as their determinant needs to vanish. If we only consider generic points on the unlabeled multiview variety, this polynomial has exactly two real solutions, the unlabeled focal point and the unlabeled triangulation of the unlabeled image points.
Proof.
Since and are symmetric with real entries only has real roots. By construction is a root. Because there is a real value such that . Now by Lemma 3.6 these two roots are the only roots of . ∎
Since the roots of the univariate quartic polynomial have double multiplicity for generic points, we can find the roots by using the quadratic formula.
If , then the procedure is simple and amounts to computing , because the kernel is one dimensional for generic data by Proposition 3.5. The complete triangulation algorithm can be stated as follows:
In some cases one might not want to keep on working with symmetric matrices, but actually reconstruct the original world points. This amounts to find a decomposition of into . Finding such a decomposition is equivalent to splitting a degenerate quadrics into two planes. To do this, the approach studied in [8] for quadrics in generalizes to our case, if one replaces the adjoint matrix by the matrix of signed 2-minors of .
We go back to studying the relaxed ideal . With the help of Proposition 3.3 we are able to determine the generators of the prime ideal of the image of .
Corollary 3.7.
Let . Then the ideal of the image of is generated by the maximal minors of (for all ), and the 11-minors of (for all ).
Proof.
By [7, §2.2] the ideal is generated by the minors of . If then by the proof of Proposition 3.3 the matrix has rank nine. If then has full rank by Proposition 3.5. Further maximal minors of matrices with are monomial multiples of the other maximal minors. We obtain a generating set by just computing the minors of correct size. ∎
Remark 3.8.
Let n=2. Consider the ideal generated by the 11-minors and the ideal generated by the three degree polynomials of Example 1.2. These two ideals are equal.
Clearly the relaxed ideal is not a good relaxation of , but it can easily be tightened by enforcing the singularity condition . We define the ideal
Still it is possible to construct rank two matrices on the variety that are not on . Let be . Then independent of , thus requiring that the minors of vanish (s. Corollary 3.7) does not impose any constraints on . Remember that Equation 5 gives a description of the preimage of the unlabeled two-view variety of two points. However
gives constraints on , namely (). Thus not all points of
(8) |
are on the unlabeled multiview variety , but they are on .
Proposition 3.9.
For n=2, the unlabeled multiview variety is minimally generated by 7 polynomials in 12 variables. It is Cohen-Macaulay and its Betti table is given in Table 1.
Proof.
By Remark 4.4 of [1] the toric set-up of Remark 1.1 is universal in the sense that every multiview variety with cameras in linearly general position is isomorphic to the multiview variety of Remark 1.1. Since the unlabeled multiview variety with cameras in linearly general position is the union of multiview varieties with correspondences interchanges, any unlabeled multiview variety is isomorphic to the unlabeled multiview variety with the camera matrices of Remark 1.1. For two views we only chose the camera matrices and of Remark 1.1 and then ran the code of Listing 1 in Macaulay2[2] with these cameras as input. ∎
Remark 3.10.
For cameras in linearly general position we have the freedom of choice of a world coordinate system in and coordinate systems in the images in without changing the unlabeled multiview variety up to isomorphism. In that sense we can freely choose five cameras with focal points in linearly general position to compute an isomorphic description of the unlabeled multiview variety.
As we see in Example 1.2 the generators of the unlabeled two-view variety of two points (, ) fall into three different classes according to their multidegree, namely the ones with multidegree , the ones with degree type and the ones with degree type . The ones of degree type are the determinants of and .
Proposition 3.11.
The ideal of three degree equations of the unlabeled two-view variety is a subset of the ideal generated by the entries of .
Proof.
We currently have no way to construct the generators with degree type . However we do believe that they relate to the variety of Equation 8.
The following statements have been derived with Macaulay2[2]. They are concerned with the prime ideal of the unlabeled multiview variety of three, four and five views.
Proposition 3.12.
For n=3, the unlabeled multiview variety is minimally generated by 60 polynomials in 18 variables and its Betti table is given in Table 2.
Proof.
Proposition 3.13.
For n=4, the unlabeled multiview variety is minimally generated by 215 polynomials up to degree six in 24 variables. For n=5, the unlabeled multiview variety is minimally generated by 620 polynomials up to degree six in 30 variables.
Proof.
Similarly to the proof of Proposition 3.9 we choose special cameras to compute the unlabeled multiview variety. For , the cameras of Remark 1.1 were chosen for the computation with Macaulay2[2]. For , we chose the four cameras of Remark 1.1 and additionally the normalized camera with focal point for the computation with Macaulay2[2]. ∎
4. More than Two Unlabeled Points
The study of more than two unlabeled points is quite more elaborate compared to the case of two points as done in Section 3. In general, equations describing the Chow variety are unknown. Also when more than two unlabeled points are present we work with symmetric tensors instead of symmetric matrices. In Section 3 we strongly relied on tools from matrix algebra, these are not available or far more complicated for symmetric tensors. We can still extend some results to the case of .
One can construct a point on the Chow variety from their unlabeled world configuration by taking the sum over all permuted tensors products
where denotes the symmetric group. The tensor is symmetric and only defined up to scale. Thus it can be embedded in . Similarly, the unlabeled image point configurations are represented by symmetric tensors defined up to scale, embedded in . The perspective relation of Equation 3 for a pinhole camera generalizes to
We can vectorize this equation above as done in the Section 3. This yields the linear equation
for some coefficient matrix , with rows and columns. Let with and then define (analogously to the two point case of Equation 7) the matrix as
This matrix has rows and columns . The unlabeled triangulation can be reconstructed from the matrix . Similar to the two point case one can construct the unlabeled focal point of views, this is
For exactly views the unlabeled focal point concatenated with zeros is in the kernel of . For views the kernel of is one dimensional and can be computed by linear algebra.
It is of interest to characterize the pictures of unlabeled points using cameras and their ambiguities. Further it would be desirable to know the prime ideal of for any and . In Propositions 3.9, 3.12 and 3.13 generators of degree at most six appear, thus we believe that the generators of can be constructed from the information obtained by two and three views.
4.1. Acknowledgment
We would like to thank Joe Kileel for his comments and our debates about this topic during his stay in Berlin. Additionally, we would like to thank Michael Joswig for his guidance.
5. Computations
We performed several random experiments. Our hardware was a cluster with Intel Xeon X2630v2 Hexa-Cores (2.8 GHz) and 64GB main memory per node. The software was Macaulay2, version 1.9.2 [2]. All computations were single-threaded.
Our result in Proposition 3.11 was proved by computations with Macaulay2[2]. Following standard practice in computational algebraic geometry, we carried out the computation on many samples in a Zariski dense set of parameters, and then conclude that it holds generically. Further instead of using special cameras as we did in the proofs of Propositions 3.9, 3.12, 3.13, we were also able to compute the unlabeled multiview variety with random cameras as input.
The computations were repeated several times with random input. It is not surprising that increasing , the number of cameras, increases the running times considerably. In particular using the toric setup of Remark 1.1 is much faster than choosing dense camera matrices .
For Proposition 3.11 we performed at least 1000 computations to verify its correctness.
Lines 1–4 define the rings in which the computations take place. Lines 6–14 produce random camera matrices. However, our experiments suggest that it suffices to check that the focal points of the cameras are in linearly general position. The unlabeled multiview map from Equation 3 is encoded in lines 17–21. Lines 13–14 are unlabeled perspective relations (Equation 3) and line 26 are the determinantal constraints on the matrices. The actual computation is the elimination in line 28. The unlabeled multiview variety is defined in lines 30–31.
References
- [1] Chris Aholt, Bernd Sturmfels, and Rekha Thomas, A hilbert scheme in computer vision, arXiv preprint arXiv:1107.2875 (2011).
- [2] Daniel R. Grayson and Michael E. Stillman, Macaulay2, a software system for research in algebraic geometry, Available at http://www.math.uiuc.edu/Macaulay2/.
- [3] Richard Hartley and Peter Sturm, Triangulation, Computer vision and image understanding 68 (1997), no. 2, 146–157.
- [4] Richard Hartley and Andrew Zisserman, Multiple view geometry in computer vision, Cambridge University Press, 2003.
- [5] Anders Heyden and Karl Åström, Algebraic properties of multilinear constraints, Mathematical Methods in the Applied Scinces 20 (1997), no. 13, 1135–1162.
- [6] J.M. Landsberg, Tensors: Geometry and applications, Graduate studies in mathematics, American Mathematical Soc.
- [7] Binglin Li, Images of rational maps of projective spaces, arXiv preprint arXiv:1310.8453 (2013).
- [8] Jürgen Richter-Gebert, Perspectives on projective geometry: A guided tour through real and complex geometry, Springer Science & Business Media, 2011.
Comments
There are no comments yet.