    # Geometric Interpretations of the Normalized Epipolar Error

In this work, we provide geometric interpretations of the normalized epipolar error. Most notably, we show that it is directly related to the following quantities: (1) the shortest distance between the two backprojected rays, (2) the dihedral angle between the two bounding epipolar planes, and (3) the L_1-optimal angular reprojection error.

## Authors

##### This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

## 1 Introduction

Consider two cameras and observing the same 3D point . If the internal calibration and the relative pose of the two cameras are known, we can backproject the measured point in each image and obtain the rays from each camera, pointing to . Now, we define the normalized epipolar error as follows:

 (1)

where and are the backprojected unit rays from and , respectively, is the rotation matrix and

is the translation vector that together transform a point from the reference frame

to , i.e., and . The second equality in (1) follows from the fact that the scalar triple product is invariant to a circular shift. In the literature, the error is often expressed as follows:

 ˆe=∣∣ˆf⊤1Eˆf0∣∣, (2)

where is the essential matrix and

is the skew-symmetric operator. Figure 1: A perfectly accurate epipolar geometry. The two backprojected rays intersect at the exact position of the 3D point. The depths of each ray are denoted by λ0 and λ1. All vectors are expressed in the coordinate system of c1.

If the image measurements, calibration and pose data are all perfectly accurate, this error would be zero because , and would be coplanar (see Fig. 1). This is called the epipolar constraint . In practice, the raw data contain inaccuracies, so they do not satisfy this constraint most of the time. For this reason, many existing works in 3D vision try to solve geometric reconstruction problems by minimizing the cost based on this error [1, 2, 4, 5, 10, 12, 16]

and/or using it to identify outliers

[11, 16].

In the literature, the normalized epipolar error has mostly been treated as an algebraic quantity that has no geometric meaning

[1, 2, 4, 5, 12, 14, 16]. We believe that this misconception stems from the fact that the “standard” epipolar error is an algebraic quantity [3, 9, 13, 15]:

 e:=∣∣f1⋅(ˆt×Rf0)∣∣=∣∣f⊤1Ef0∣∣, (3)

where and are the normalized image coordinates of the point in and , respectively. Notice that the only difference between (1) and (3) is the way the rays are normalized: In (1), they are normalized by their lengths, whereas in (3), they are normalized by the last element in the vector.

In , a geometric interpretation was given for the following error:

 (4)

which corresponds to the cosine of the angle between and . This is equal to the perpendicular distance between the point at and the plane containing and .

In this work, we provide geometrically intuitive interpretations of (1) by relating it to the following quantities:

1. The volume of the tetrahedron where , and form the three edges meeting at one vertex.

2. The shortest distance between the two backprojected rays and .

3. The dihedral angle between the two bounding epipolar planes, , one plane containing and and the other containing and .

4. The -optimal angular reprojection error. Figure 2: (a) The volume of this tetrahedron is proportional to the normalized epipolar error (1). (b) d is the shortest distance between the two backprojected rays corresponding to the same point.

## 2 Geometric Interpretations of (1)

1. Relation to the volume of a tetrahedron:

Consider the tetrahedron shown in Fig. 2reda. One of its vertices is placed at (, the position of camera , which is the origin in the reference frame of ), and the other three at , and . Then, using the well-known formula for the volume of a tetrahedron, its volume is obtained by

 V=16∣∣ˆt⋅(Rˆf0×ˆf1)∣∣ˆe6. (5)

Therefore,

 ˆe=6V. (6)

The nice thing about this interpretation is that it allows for a simple visualization of the error, as shown in Fig. 2reda. As the degree of coplanarity increases among the three edges (, and ), the common vertex will be “pulled” towards the opposite side, flattening the tetrahedron. When the three edges are coplanar, the tetrahedron becomes completely flat, , , and thus .

2. Relation to the distance between the two rays: Figure 3: (a) The distance between the two skew lines is the same as the distance between the two parallel planes as shown here. The normal of the two planes is given by n=m0×m1. (b) α is the dihedral angle between the two bounding epipolar planes.

We can also relate the normalized epipolar error (1) to the shortest distance between the two backprojected rays, , in Fig. 2redb. To show this, we will first derive the formula for the shortest distance between two skew lines. Consider two skew lines and . The distance between them is given by the distance between the closest pair of points on each line ( and ), and they lie on the common perpendicular to both lines111We can easily prove this by contradiction. We omit the proof.. Now, consider two parallel planes with the normal : plane containing and plane containing , as illustrated in Fig. 3reda. Notice that is the same as the distance between the planes, which is the same as where is the projected position of in . Since , we get

 d=|(c0−c1)⋅(m0×m1)|∥m0×m1∥. (7)

This means that in Fig. 2redb is given by

 d=∣∣t⋅(Rˆf0×ˆf1)∣∣∥Rˆf0×ˆf1∥∥t∥ˆe∥Rˆf0×ˆf1∥. (8)

Let be the angle between and (also known as the raw parallax angle ), ,

 β:=∠(Rˆf0,ˆf1)∈[0,π/2]. (9)

Then, (8) can be written as

 ˆe=sin(β)∥t∥d. (10)

Therefore, we can interpret as the distance between the two backprojected rays, weighted by

. For relative pose estimation between two views, we can assume

without loss of generality, so minimizing the cost based on (10) is equivalent to minimizing the cost based on . We can interpret as the factor that downweights the residual when the parallax angle is small. Note that and the equality holds if and only if and are both perpendicular to . If the two rays intersect (at infinity), then (or ), and thus .

As a side note, it should be mentioned that given by (8) is the distance between the lines rather than the rays. Technically speaking, it is the shortest distance between line for and line for .

3. Relation to the angle between the two planes:

In Fig. 2redb, consider the following planes: one plane containing and , and another containing and . Let and be their normal vectors. These two planes are drawn in Fig. 3redb. We can think of them as the two bounding planes between which the epipolar plane is usually found. This is the case for most two-view triangulation methods (, midpoint methods  and optimal methods ). The dihedral angle between these two bounding planes is given by

 (11)

This can be rearranged to

 ∥∥Rˆf0×ˆt∥∥∥∥ˆf1×ˆt∥∥sin(α)=∥∥(Rˆf0×ˆt)×(ˆf1×ˆt)∥∥. (12)

For any 3D vector , , , , the vector quadruple product is equal to . Therefore, the right-hand side of (12) can be written as

 ∥∥(Rˆf0×ˆt)×(ˆf1×ˆt)∥∥ =∥∥((Rˆf0×ˆt)⋅ˆt)0ˆf1−((Rˆf0×ˆt)⋅ˆf1)ˆt∥∥ (13) (14)

Note that the third equality follows from the fact that is a unit vector. Substituting (14) into (12) leads to

 ˆe =∥∥Rˆf0×ˆt∥∥∥∥ˆf1×ˆt∥∥sin(α) (15) =sin(ϕ0)sin(ϕ1)sin(α), (16)

where

 ϕ0 :=∠(Rˆf0,ˆt)∈[0,π/2], (17) ϕ1 :=∠(ˆf1,ˆt)∈[0,π/2]. (18)

These two angles are shown in Fig. 2redb. From (16), we can interpret as the sine of the dihedral angle between the two bounding epipolar planes, weighted by . Therefore, would be small if either of , or is very small. This makes sense because the epipolar geometry degenerates as or approaches zero. Also, when is small, the two bounding epipolar planes are close to coplanarity, and so do the vector , and .

4. Relation to the optimal angular reprojection error: Figure 4: The angular reprojection errors (θ0, θ1) measure the angular difference between the backprojected rays (Rˆf0, ˆf1) and the corrected rays (Rˆf′0, ˆf′1) that are made to intersect.

The -optimal angular reprojection error  is defined as follows:

 θ∗L1=minˆf′0,ˆf′1(∠(ˆf′0,ˆf0)+∠(ˆf′1,ˆf1))  s.t.  ˆf′1⋅(ˆt×Rˆf′0)=0. (19)

In other words, it is the minimum of where and are the angles by which we correct the backprojected rays to make them intersect. Fig. 4 illustrates these angles. In , it was shown that

 sin(θ∗L1)=min⎛⎝∣∣Rˆf0⋅(ˆf1×ˆt)∣∣∥Rˆf0×ˆt∥,∣∣Rˆf0⋅(ˆf1×ˆt)∣∣∥ˆf1×ˆt∥⎞⎠. (20)

Rearranging this, we get

 ∣∣ˆf1⋅(ˆt×Rˆf0)∣∣=sin(θ∗L1)max(∥Rˆf0×ˆt∥,∥ˆf1×ˆt∥). (21)

Using (1), (17), (18), this can be written as

 ˆe=sin(max(ϕ0,ϕ1))sin(θ∗L1). (22)

Therefore, we can interpret as the sine of the -optimal angular reprojection error, weighted by . It follows that would be small if either of or is very small. This makes sense because small means that only a little correction is needed for the two backprojected rays to intersect. Also, small means that the vector , and are all close to parallelism, which brings the epipolar geometry close to degeneracy. What may seem peculiar in (22) is the fact that does not reflect the degeneracy when either or is zero. However, this is not an issue, because the term is necessarily zero whenever degeneracy occurs. In the Appendix, we verify (22) using simulation. Figure 5: Histograms of the normalized epipolar error ˆe before and after the L1-optimal correction based on angular errors . The corrected rays yield minuscule error, which implies that they now intersect (within the numerical accuracy). Figure 6: The percentage of the simulation runs where the angular error θ∗L1 obtained from the L1-optimal method  is smaller than that of the perturbed point. In our experiment, θ∗L1 is always smaller unless the triangulated point is perturbed by extremely small noise (<10−12 unit) beyond the numerical accuracy.

## 3 Conclusion

In this work, we presented several geometric interpretations of the normalized epipolar error defined in (1). Specifically, we revealed the direct relations between this error and the following quantities:

1. The volume of the tetrahedron where , and form the three edges meeting at one vertex (see Fig. 2reda). The relation is given by (6).

2. The shortest distance between the two backprojected rays and (see Fig. 2redb). The relation is given by (10).

3. The dihedral angle between the two bounding epipolar planes, , one plane containing and and the other containing and (see Fig. 3reda). The relation is given by (16).

4. The -optimal angular reprojection error defined in (19). The relation is given by (22).

## Acknowledgement

This work was partially supported by the Spanish government (project PGC2018- 096367-B-I00) and the Aragón regional government (Grupo DGA-T45_17R/FSE).

## Appendix Figure 7: Histogram of |ˆeest−ˆe|, where ˆe and ˆeest are the normalized epipolar errors computed using (1) and (22) respectively. Figure 8: Histogram of |ˆeest−ˆe|/|ˆe|, where ˆe and ˆeest are the normalized epipolar errors computed using (1) and (22) respectively.

The contributions of this work are the derivations of (6), (10), (16) and (22). No approximation is made in the derivations, so strictly speaking, experiments are redundant as long as the mathematics are correct. Having said that, we understand that some readers may have doubts about the derivations, and also, it is essential to verify the theoretical results whenever possible (as a sanity check). In the case of (6), (10) and (16), however, performing experiments is pointless because the only sensible method to compute the volume , the distance and the angle is to use the very same formulas used in the derivations. For this reason, we only focus on the verification of (22) in this section.

In order to verify (22), we compare the values of computed using (1) and (22). This is done in the following steps:

1. [leftmargin=*]

2. In simulation, we create two cameras and one point. The two cameras are placed at position and where is a random 3D vector of length 0.5 unit and . This ensures that unit. The image size is set to pixel and the focal length to pixel. We place the point at where

follows the uniform distribution

. Then, we orient the cameras randomly until the point is visible in both views. The image coordinates of the projected point are perturbed by Gaussian noise with pix.

3. We correct the backprojected rays using the -optimal triangulation method described in  and obtain the angular error using (19). To check if this is locally optimal, we perturb the resulting 3D point by small random noise and see if we achieve smaller error. We set the noise magnitude to unit with , and for each magnitude, we perturb the point one hundred times independently.

4. We compute using (22) and compare it to from (1).

We repeat this procedure times and aggregate the results. All computations are done in Matlab. Fig. 5 shows the histograms of the normalized epipolar error computed using (1) before and after the -optimal ray correction . Comparing the two histograms, we see that the corrected rays do intersect. Fig. 6 presents the result of the perturbation test. It shows that the angular error of the corrected rays is (locally) minimum within the numerical accuracy. Plugging into (22), we obtain an estimate of , , . In Fig. 7, we plot the histogram of the absolute difference . Notice that it is as small as the normalized epipolar error of intersecting rays (see Fig. 5). Therefore, we can safely conclude that within the numerical accuracy. In Fig. 8, we provide, for completeness, the histogram of the relative difference .

## References

•  J. Briales, L. Kneip, and J. Gonzalez-Jimenez. A certifiably globally optimal solution to the non-minimal relative pose problem. In

IEEE Conf. Comput. Vis. Pattern Recognit.

, pages 145–154, 2018.
•  M. Garcia-Salguero, J. Briales, and J. Gonzalez-Jimenez. Certifiable relative pose estimation. CoRR, abs/2003.13732, 2020.
•  R. Hartley and A. Zisserman.

Multiple View Geometry in Computer Vision

.
Cambridge University Press, New York, NY, USA, 2 edition, 2003.
•  U. Helmke, K. Hüper, P. Y. Lee, and J. Moore. Essential matrix estimation using gauss-newton iterations on a manifold. Int. J. Comput. Vis., 74(2):117–136, 2007.
•  L. Kneip and S. Lynen. Direct optimization of frame-to-frame rotation. In IEEE Int. Conf. on Comput. Vis., pages 2352–2359, 2013.
•  S. H. Lee and J. Civera. Closed-form optimal two-view triangulation based on angular errors. In IEEE Int. Conf. Comput. Vis., pages 2681–2689, 2019.
•  S. H. Lee and J. Civera. Triangulation: Why optimize? In Brit. Mach. Vis. Conf., 2019.
•  H. C. Longuet-Higgins. A computer algorithm for reconstructing a scene from two projections. Nature, 293(5828):133–135, 1981.
•  Q. Luong and O. D. Faugeras. The fundamental matrix: Theory, algorithms, and stability analysis. Int. J. Comput. Vis., 17(1):43–75, 1996.
•  A. Pagani and D. Stricker. Structure from motion using full spherical panoramic cameras. In IEEE Int. Conf. on Comput. Vis. Workshops, pages 375–382, 2011.
•  S. H. Lee and J. Civera. Robust uncertainty-aware multiview triangulation. CoRR, abs/2008.01258, 2020.
•  A. L. Rodríguez, P. E. López-de-Teruel, and A. Ruiz. Reduced epipolar cost for accelerated incremental SfM. In IEEE Conf. Comput. Vis. Pattern Recognit., pages 3097–3104, 2011.
•  P. Torr and D. Murray. The development and comparison of robust methods for estimating the fundamental matrix. Int. J. Comput. Vis., 24(3):271–300, 1997.
•  J. Yang, H. Li, and Y. Jia. Optimal essential matrix estimation via inlier-set maximization. In Eur. Conf. Comput. Vis., pages 111–126, 2014.
•  Z. Zhang. Determining the epipolar geometry and its uncertainty: A review. Int. J. Comput. Vis., 27(2):161–195, 1998.
•  J. Zhao. An efficient solution to non-minimal case essential matrix estimation. CoRR, abs/1903.09067, 2019.