Stereo vision is one of the most commonly used technologies for 3D information extrapolation [1, 2, 3]. This technology extrapolates the depth of the real-world scenery by comparing the difference between a pair of stereo images [4, 5]. The relative positional difference between each pair of correspondence points is generally known as disparity [2, 6]. The disparity information enables autonomous systems to segment road regions more effectively . For example, Labayrade et al.  introduced an accurate obstacle detection method based on the analysis of a so-called “v-disparity” image, which was created by computing the histogram of each horizontal row of the disparity map. Since then, many algorithms based on v-disparity image analysis have been proposed for road region segmentation [9, 10, 11]. These algorithms generally hypothesize that the road surface is planar and the stereo rig baseline is perfectly parallel to the horizontal road surface [12, 13]. These assumptions make it feasible to represent the road disparity projections using a linear model [14, 15, 16]. The road regions can, therefore, be extracted by comparing the difference between the actual disparity values and the estimated linear road disparity projection model .
However, in practice, the stereo rig baseline is not always perfectly parallel to the horizontal road surface. This fact can introduce a non-zero roll angle , shown in Fig. 1, into the imaging process, where and represent the baseline and the mounting height of the stereo rig, respectively; and are the origins of the left and right camera coordinate systems, respectively. The stereo rig baseline can become perfectly parallel to the horizontal road surface by transforming the original world coordinate system to a new world coordinate system . The original disparity map is shown in Fig. 2, where readers can observe that the disparity values change gradually in the horizontal direction. This makes the way of representing the road disparity projections with a linear model problematic . Therefore, the effects caused by the non-zero roll angle have to be eliminated before performing road region segmentation .
In recent years, many researchers have proposed to estimate the roll angle from the disparity map. For example, in  and , the road surface was assumed to be planar, and the disparities in a selected small area were modeled by a plane , where and are the horizontal and vertical disparity coordinates, respectively. The roll angle was then calculated as . However, finding a proper small area for plane modeling is always challenging, because such an area may contain an obstacle . This can severely affect the plane modeling precision . In this regard, an iterative roll angle estimation algorithm was proposed in , where multiple small areas were selected for plane modeling. The roll angle was then estimated from the optimum selected area. However, the aforementioned algorithms can only work for planar road surfaces. Therefore, in our previous work , the projections of road disparities were represented by a parabola, and the roll angle estimation was formulated as an energy minimization problem. To achieve greater processing efficiency, golden section search (GSS) algorithm  was utilized to reduce the roll angle search range. However, this method still takes extensive iterations to converge to the minimum. Therefore, the algorithm proposed in this paper solves the energy minimization problem using gradient descent (GD) , which greatly reduces the computational complexity of roll angle estimation.
The remainder of this paper is structured as follows: Section II presents the proposed roll angle estimation algorithm. The experimental results for performance evaluation are illustrated in Section III. Finally, Section IV summarizes the paper and provides recommendations for future work.
The input of our proposed algorithm is a dense disparity map, which was obtained from a stereo road image pair (see Fig. 2 and 2) through our previously published disparity estimation algorithm . The estimated disparity map is shown in Fig. 2, and its v-disparity map is shown in Fig. 2. To rotate the disparity map around a given angle , each set of original coordinates is transformed to a set of new coordinates using the following equations:
The parabola representing the road disparity projections can, therefore, be written as follows:
The coefficients of can, therefore, be estimated by solving an energy minimization problem as follows:
the column vectorstores the disparity values, the column vector stores the coefficients of , and
According to , the desirable roll angle corresponds to the minimum . Therefore, the proposed roll angle estimation algorithm is equivalent to the following energy minimization problem:
However, deriving the numeric solution of (10) is significantly challenging . Therefore, the proposed algorithm utilizes GD to approximate by formulating (10) as an iterative optimization problem as follows :
is an identity matrix, andis the learning rate. If is too high, (11) may overshoot the minimum. On the other hand, if is set to a relatively low value, the convergence of (11) may take too many iterations . Therefore, selecting a proper is always essential for GD. Instead of fixing the learning rate with a constant, we utilize backtracking line search to produce an adaptive learning rate:
The initial learning rate is set to 26. We will discuss the selection of in Section III. The initial approximation is set to , because in practical experiments the absolute value of the roll angle is usually not that high. The optimization iterates until the absolute difference between and is smaller than a preset threshold . The performance of the proposed roll angle estimation algorithm will be discussed in Section III.
Iii Experimental Results
In this section, we evaluate the performance of the proposed roll angle estimation algorithm both qualitatively and quantitatively. The proposed algorithm was implemented in Matlab 2019a on an Intel Core i7-8700K CPU (3.7 GHz) using a single thread.
Firstly, we discuss the selection of . In our experiments, we utilized a ZED stereo camera111https://www.stereolabs.com/ to capture stereo road image pairs. The disparity maps estimated from the stereo image pairs were then utilized to estimate the roll angles. Some experimental results of our pothole datasets222www.ruirangerfan.com are shown in Fig. 3, where readers can observe that the disparities on each row in the rotated disparity maps have similar values. Therefore, the proposed algorithm can effectively estimate the roll angle from a given disparity map.
Furthermore, we select a range of and record the number of iterations that GD takes to converge with respect to different , as shown in Fig. 4, where readers can see that is the optimum value for different . When rad (), GD takes three iterations to converge, while GD iterates four times, when is set to rad (), rad () or rad ().
Moreover, we compare the computational efficiency of the proposed algorithm with . As discussed in , the computational complexity of GSS is , where is the golden section ratio. The comparison of the computational efficiency between GSS and GD is shown in Table I, where we can see that GSS takes more iterations to converge with the decrease of , while the performance of GD largely remains the same. Therefore, the proposed roll angle estimation algorithm performs more robustly than  with respect to different thresholds .
However, GD involves the computations of both and , which are more complex than the computation of . Therefore, we also compare the runtime of GSS and GD with respect to different thresholds , as shown in Table II. Readers can observe that the runtime of GSS goes up gradually with the decrease of , while the runtime of GD changes slightly with respect to different . Overall, the proposed algorithm performs much faster than .
To quantify the accuracy of the proposed roll angle estimation algorithm, we utilize a synthesized stereo dataset from EISATS333https://ccv.wordpress.fos.auckland.ac.nz/ [26, 27], where the roll angle is perfectly zero. We manually rotate the disparity maps around a given angle, and then estimate the roll angles from the rotated disparity maps. Examples of the EISATS stereo datasets are shown in Fig. 5, where readers can see that the effects caused by the roll angle are successfully eliminated by performing the proposed algorithm. Furthermore, we compare the accuracy of the estimated roll angles between the proposed method and  with respect to different . The comparison results are shown in Fig. 6, where represents the average difference between the actual and estimated roll angles. From Fig. 6, we can find that GD performs more accurately than GSS when is set to a lower value. Additionally, the computational efficiency of GD is also greater than that of GSS. Therefore, GD is a better solution than GSS in terms of solving the energy minimization problem in roll angle estimation.
Iv Conclusion and Future Work
In this paper, a robust GD-based algorithm was proposed to estimate the roll angle from disparity maps. The proposed algorithm was developed based on our previously published work, where roll angle estimation was formulated as an energy minimization problem, which was then solved using GSS. The experimental results demonstrated that the proposed algorithm outperforms our previous method in terms of both accuracy and efficiency, and that it can successfully eliminate the effects caused by the non-zero roll angle. In the future, we aim to design a neural network to estimate the roll angle with an unsupervised learning strategy.
This work is supported by grants from the Research Grants Council of the Hong Kong SAR Government, China (No. 11210017, No. 16212815, No. 21202816, NSFC U1713211) awarded to Prof. Ming Liu. This work is also supported by grants from the Shenzhen Science, Technology and Innovation Commission (JCYJ20170818153518789), National Natural Science Foundation of China (No. 61603376) and Guangdong Innovation and Technology Fund (No. 2018B050502009) awarded to Dr. Lujia Wang. This work is also supported by funding awarded to Prof. Ioannis Pitas received from the European Union Horizon 2020 research and innovation programme, under grant agreement No. 731667 (Multidrone).
-  R. Fan, “Real-time computer stereo vision for automotive applications,” Ph.D. dissertation, University of Bristol, 2018.
E. Trucco and A. Verri,
Introductory techniques for 3-D computer vision. Prentice Hall Englewood Cliffs, 1998, vol. 201.
-  R. Fan, Y. Liu, M. J. Bocus, and M. Liu, “Real-time subpixel fast bilateral stereo,” arXiv preprint arXiv:1807.02044, 2018.
-  N. Qian, “Binocular disparity and the perception of depth,” Neuron, vol. 18, no. 3, pp. 359–368, 1997.
-  R. Fan and N. Dahnoun, “Real-time implementation of stereo vision based on optimised normalised cross-correlation and propagated search range on a gpu,” in 2017 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, 2017, pp. 1–6.
-  R. Fan, Y. Liu, X. Yang, M. J. Bocus, N. Dahnoun, and S. Tancock, “Real-time stereo vision for road surface 3-d reconstruction,” in 2018 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, 2018, pp. 1–6.
-  R. Fan, J. Jiao, J. Pan, H. Huang, S. Shen, and M. Liu, “Real-time dense stereo embedded in a uav for road inspection,” arXiv preprint arXiv:1904.06017.
-  R. Labayrade, D. Aubert, and J.-P. Tarel, “Real time obstacle detection in stereovision on non flat road geometry through” v-disparity” representation,” in Intelligent Vehicle Symposium, 2002. IEEE, vol. 2. IEEE, 2002, pp. 646–651.
-  N. Soquet, D. Aubert, and N. Hautiere, “Road segmentation supervised by an extended v-disparity algorithm for autonomous navigation,” in 2007 IEEE Intelligent Vehicles Symposium. IEEE, 2007, pp. 160–165.
-  R. Fan and N. Dahnoun, “Real-time stereo vision-based lane detection system,” Measurement Science and Technology, vol. 29, no. 7, p. 074005, 2018.
-  D. Yiruo, W. Wenjia, and K. Yukihiro, “Complex ground plane detection based on v-disparity map in off-road environment,” in 2013 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2013, pp. 1137–1142.
A. Broggi, C. Caraffi, R. I. Fedriga, and P. Grisleri, “Obstacle
detection with stereo vision for off-road vehicle navigation,” in
Proc. IEEE Computer Society Conf. Computer Vision and Pattern Recognition (CVPR’05) - Workshops, Sep. 2005, p. 65.
-  Z. Hu, F. Lamosa, and K. Uchimura, “A complete uv-disparity study for stereovision based 3d driving environment analysis,” in 3-D Digital Imaging and Modeling, 2005. 3DIM 2005. Fifth International Conference on, 2005, pp. 204–211.
T. Chen, B. Chen, X. Zhang, and Z. Yuan, “Free space detection using stereo confidence metrics and obstacle position probability maps,” inProc. 14th IEEE Int. Conf. Signal Processing (ICSP), Aug. 2018, pp. 1071–1075.
-  F. Oniga, E. Sarkozi, and S. Nedevschi, “Fast obstacle detection using u-disparity maps with stereo vision,” in Proc. IEEE Int. Conf. Intelligent Computer Communication and Processing (ICCP), Sep. 2015, pp. 203–207.
-  W. Song, M. Fu, Y. Yang, M. Wang, X. Wang, and A. Kornhauser, “Real-time lane detection and forward collision warning system based on stereo vision,” in Proc. IEEE Intelligent Vehicles Symp. (IV), Jun. 2017, pp. 493–498.
-  H. Ma, Y. Ma, J. Jiao, M. U. M. Bhutta, M. J. Bocus, L. Wang, M. Liu, and R. Fan, “Multiple lane detection algorithm based on optimised dense disparity map estimation,” in 2018 IEEE International Conference on Imaging Systems and Techniques (IST). IEEE, 2018, pp. 1–5.
-  R. Fan, M. J. Bocus, and N. Dahnoun, “A novel disparity transformation algorithm for road segmentation,” Information Processing Letters, vol. 140, pp. 18–24, 2018.
-  U. Ozgunalp, R. Fan, X. Ai, and N. Dahnoun, “Multiple lane detection algorithm based on novel dense vanishing point estimation,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 3, pp. 621–632, 2017.
-  R. Fan, X. Ai, and N. Dahnoun, “Road surface 3D reconstruction based on dense subpixel disparity map estimation,” IEEE Transactions on Image Processing, vol. PP, no. 99, p. 1, 2018.
-  M. Evans, R. Fan, and N. Dahnoun, “Iterative roll angle estimation from dense disparity map,” in 2018 7th Mediterranean Conference on Embedded Computing (MECO). IEEE, 2018.
-  P. Pedregal, Introduction to optimization. Springer Science & Business Media, 2006, vol. 46.
-  J. C. Meza, S. Profile, J. C. Meza, R. A. Oliva, E. O. L. Berkeley, P. D. Hough, and P. J. Williams, “Nonlinear optimization.”
-  E. W. Weisstein, “Least squares fitting,” 2002.
-  S. S. Rao and S. S. Rao, Engineering Optimization: Theory and Practice. Wiley, 2009.
-  T. Vaudrey, C. Rabe, R. Klette, and J. Milburn, “Differences between stereo and motion behaviour on synthetic and real-world stereo sequences,” in Image and Vision Computing New Zealand, 2008. IVCNZ 2008. 23rd International Conference. IEEE, 2008, pp. 1–6.
-  A. Wedel, C. Rabe, T. Vaudrey, T. Brox, U. Franke, and D. Cremers, “Efficient dense scene flow from sparse or dense stereo data,” in European conference on computer vision. Springer, 2008, pp. 739–751.