THE deployment of autonomous vehicles has been increasing rapidly since Google first launched their self-driving car project in 2009 . In recent years, with a number of technology breakthroughs being witnessed in the world where science fiction inventions are now becoming a reality, the competition to commercialise driver-less vehicles by companies like GM, Waymo and Daimler-Bosch is fiercer than ever . For instance, Volvo conducted a series of self-driving experiments involving around 100 cars in China . The 5G network is also utilised in self-driving vehicles to help them communicate with each other . Furthermore, the computer stereo vision technique has also been prevalently used in prototype vehicle road tests to provide the depth information for various automotive applications, e.g., road condition assessment [4, 5], lane and obstacle detection [6, 3], and vehicle state estimation [7, 8].
The state-of-the-art lane detection algorithms can mainly be classified as feature-based and model-based. The former extracts local and useful information of an image, such as edges, texture and colour, for the segmentation of lanes and road boundaries . On the other hand, the model-based algorithms aim to represent the lanes with mathematical equations, e.g., linear, parabolic, linear-parabolic and spline, based upon some common road geometry assumptions . The linear model works well for lanes with a low curvature , but a more flexible road model is inevitable when lanes with a higher curvature are considered. Therefore, some algorithms [11, 12, 13, 14] use a parabolic model to represent the lanes with a constant curvature. To address some more complex cases, Jung et al. proposed a combination of linear and parabolic lane models, where the far lanes are modelled as parabolas whereas the nearby lanes are represented as linear models 
. In addition to the models mentioned above, the spline model is an alternative method whereby the lane pixels are interpolated into an arbitrary shape[10, 16]. However, the more parameters introduced into a flexible model, the higher will be the computational complexity of the algorithm. Therefore, some authors resort to additional important properties of 3-D imaging techniques, such as disparity estimation , instead of being limited to only 2-D information.
One of the most prevalently used 3-D based methods is Inverse Perspective Mapping (IPM). With the assumption that two lanes are parallel to each other in the World Coordinate System (WCS), IPM can map a 3-D scenery into a 2-D bird’s eye view . In addition, some authors [18, 19, 20, 21] proposed to use the vanishing point to model lane markings and road boundaries, where and denote the vertical and horizontal coordinates of the vanishing point, respectively. An example of vanishing point is illustrated in Fig. 1, where and are two parallel straight lines. Two 3-D points and in the WCS are projected on the image plane and their corresponding points in the Image Coordinate System (ICS) are and , respectively. Therefore, the projections of and in the image are two straight lines and they intersect at the vanishing point . However, their algorithms work well only if the road surface is assumed to be flat or the camera parameters are known. Hence, some researchers pay closer attention to the disparity information which can be obtained using either active sensors, e.g., radar and laser, or passive sensors, e.g., stereo cameras . Since Labayrade et al. proposed the concept of “v-disparity” in 2002 , disparity information has been widely used to enhance the robustness of the lane detection systems. The work presented in  shows a particular instance where the disparity information is successfully combined with a lane detection algorithm to estimate for both flat and non-flat road surfaces. At the same time, the obstacles containing a lot of redundant information can be eliminated by comparing the actual and fitted disparity values. However, the estimation of
is greatly affected by the outliers when performing the Least Squares Fitting (LSF), and the lanes are sometimes wrongly detected because the selection of plus-minus peaks is not always effective. Moreover, achieving real-time performance is still challenging in because of the intensive computational complexity of the algorithm. Hence in this work, we propose a more efficient lane detection algorithm where the disparity information acquired at time is utilised to improve the process of the disparity estimation at time . The flow chart of the proposed lane detection algorithm is depicted in Fig. 2.
Firstly, we create a vectorto store the parameters of the road model. The keypoints and are then extracted and matched using Binary Robust Invariant Scalable Keypoints (BRISK), where and denote the matched keypoints in the left and right images, respectively. and are then utilised to create a sparse v-disparity histogram from which the road model can be estimated. The search range for the disparity estimation at time is subsequently controlled according to , where the search range at row is limited to . The lanes are then detected using our previously published method in . As for the disparity estimation at time (), is estimated by creating a dense v-disparity map using the disparity information acquired at time () and then fitting a parabola to the best path extracted from the dense v-disparity map. This not only improves the accuracy of the disparity estimation at time but also boosts its processing speed.
The remainder of the paper is structured as follows: Section II presents the proposed lane detection algorithm. In Section III, the experimental results are illustrated and the evaluation of the proposed system is carried out. Finally, Section IV summaries the paper and provides some recommendations for future work.
Ii Algorithm Description
Ii-a Disparity Estimation
The proposed disparity estimation approach in this paper is developed based on our previously published algorithm , where the search range at the position of is propagated from three estimated disparities at the positions of , and . In this subsection, we optimise the process of the disparity estimation by utilising the disparity information acquired at time to control the search range at time , where is a positive integer.
As for the stereo matching at time , the parameter vector is first initialised as . Then, is estimated by solving a least squares problem with a set of reliable correspondences and . In this paper, we use BRISK to detect and match and . It allows a faster execution to achieve approximately the same number of correspondences as SURF (Speeded-Up Robust Features) and SIFT (Scale-Invariant Feature Transform) . An example of keypoints detection and matching is shown in Fig. 3. The correspondences are then utilised to create a sparse v-disparity histogram as shown in Fig. 3. The values of , and can then be obtained by fitting a parabola to the non-zero candidates in the sparse v-disparity map. Since the outliers can severely affect the accuracy of the LSF, Random Sample Consensus (RANSAC) is utilised to iteratively update until the ratio reaches a maximum value, where and represent the number of inliers and outliers, respectively. Then, the search range at row is limited to , where is a threshold set to . The estimated disparity map at time is shown in Fig. 4.
Afterwards, the disparity information acquired at time is utilised to create a dense v-disparity map. The latter is then optimised using Dynamic Programming (DP) and the optimum solution corresponds to the vertical road profile. More details are provided in subsection II-C. The optimum solution is then interpolated into a quadratic polynomial and the values of , and at time can be obtained. The parameter vector at time is then used to control the search range at time according to the strategy mentioned above. The estimated disparity maps at time and are illustrated in Fig. 4 and Fig. 4, respectively.
Ii-B Image Segmentation and Edge Detection
Ii-B1 Bilateral Filtering
The input image is always noisy, making various edge detectors such as Sobel and Canny very sensitive to the blobs . Therefore, we first perform bilateral filtering on the input image to reduce redundant information while still preserving the edges. The filtered image is shown in Fig. 5.
Ii-B2 Road Model Estimation
As discussed in subsection II-C, the boundary between the road and sky can be obtained by solving a quadratic function . Furthermore, the road area can be extracted by computing the difference between the actual and fitted disparity values, i.e., and , and then finding the pixels that meet the requirement of , where is a threshold set to remove the pixels on the obstacles .
Ii-B3 Edge Detection
Ii-C Lane Detection
DP is an optimisation method which aims at solving a complicated problem by breaking it down into a series of simpler sub-problems . According to our previously published algorithm in , DP is performed to minimise the energy function in Eq. 1, where represents the total energy and the path with the minimum is selected as the optimum solution. depends on the total votes each accumulator gets and with a smoothness constant , penalises the changes in .
In this section, DP is first utilised to optimise the v-disparity map by minimising the energy function in Eq. 2. The blue path in the v-disparity map is the optimum solution, as shown in Fig. 6. The optimum solution is then interpolated into a quadratic polynomial: , and with respect to each row can thus be computed.
Then, DP is used to optimise the dense accumulator by minimising the energy function in Eq. 3. The blue path in the dense accumulator map is the optimum solution, as shown in Fig 6. with respect to each row can then be estimated by interpolating the optimum solution into a quadruplicate polynomial: .
provides the tangential direction and the curvature information of lanes, which can help to validate the lane positions. In our previous paper , a likelihood function is formed for each edge point and the plus-minus peak pairs are selected for lane visualisation, where is the angle between the -axis and the orientation of the edge point , and is the angle between the -axis and the radial from an edge pixel to . More details are provided in [24, 21]. The lanes can thus be visualised using , and . Some examples of lane detection results are shown in Fig. 7.
Iii Experimental Results
In this section, we use the KITTI dataset to quantify the robustness of the proposed algorithm. Some experimental results are shown in Fig. 7, where the green regions are the estimated road areas and the lines in red are the detected lanes.
Currently, it is impossible to access a satisfying ground truth dataset for the evaluation of lane detection algorithms because accepted test protocols do not usually exist . Therefore, many publications related to lane detection only compare the quality of their experimental results with the ones obtained using some other published algorithms . For this reason, we compare the performance of the proposed system with . Table I and Table II details the successful detection rates of the proposed algorithm and the algorithm presented in , respectively. The overall successful detection rate is around and it is increased by around .
In addition, the proposed algorithm is implemented in C programming language. The runtime of the algorithm for a single frame with a resolution of is 0.21 seconds (using a single thread of Intel Core I7-8700K CPU). By controlling the search range at time using the disparity information acquired at time , the runtime of the disparity estimation is reduced by around . Although the algorithm does not perform in real time, we believe that the execution speed of the implementation can be further accelerated by highly exploiting the parallel computing architecture.
Iv Conclusion and Future Work
This paper presented a multiple lane detection algorithm. The process of the disparity estimation is optimised by using the disparity information acquired at time to suggest the search range at time (). This not only improved the accuracy of the estimated disparities but also reduced the runtime of the algorithm. The runtime of the disparity estimation is reduced by approximately . The lanes were detected using our previously published algorithm in [21, 6]. The experimental results illustrated that the proposed system works robustly and precisely for both highway and urban scenes and a successful detection rate was achieved when processing the KITTI dataset.
However, some actual road conditions may result in failed detections. Therefore, we plan to train a deep neural network for dense vanishing point estimation. Furthermore, we plan to implement the proposed algorithm on some state-of-the-art embedded systems, such as Jetson TX2 for real-time purposes.
This work is supported by the Research Grant Council of Hong Kong SAR Government, China, under Project No. 11210017, No. 16212815 and No. 21202816, NSFC U1713211 and Shenzhen Science, Technology and Innovation Commission (SZSTI) JCYJ20160428154842603, awarded to Prof. Ming Liu; Shenzhen Science, Technology and Innovation Commission (SZSTI) JCYJ20170818153518789 and National Natural Science Foundation of China No. 61603376, awarded to Dr. Lujia Wang.
-  James A Brink, Ronald L Arenson, Thomas M Grist, Jonathan S Lewin, and Dieter Enzmann, “Bits and bytes: the future of radiology lies in informatics and information technology,” European Radiology, pp. 1–5, 2017.
-  Rui Fan and Naim Dahnoun, “Real-time implementation of stereo vision based on optimised normalised cross-correlation and propagated search range on a gpu,” in Imaging Systems and Techniques (IST), 2017 IEEE International Conference on. IEEE, 2017, pp. 241–246.
-  Rui Fan, Victor Prokhorov, and Naim Dahnoun, “Faster-than-real-time linear lane detection implementation using soc dsp tms320c6678,” in Imaging Systems and Techniques (IST), 2016 IEEE International Conference on. IEEE, 2016, pp. 306–311.
-  R. Fan, X. Ai, and N. Dahnoun, “Road surface 3D reconstruction based on dense subpixel disparity map estimation,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 3025–3035, June 2018.
-  Rui Fan, Yanan Liu, Xingrui Yang, Mohammud Junaid Bocus, Naim Dahnoun, and Scott Tancock, “Real-time stereo vision for road surface 3-d reconstruction,” arXiv preprint arXiv:1807.07433.
-  Rui Fan and Naim Dahnoun, “Real-time stereo vision-based lane detection system,” Measurement Science and Technology, vol. 29, no. 7, 2018.
-  Meghan Evans, Rui Fan, and Naim Dahnoun, “Iterative roll angle estimation from dense disparity map,” in 2018 7th Mediterranean Conference on Embedded Computing (MECO). IEEE, 2018.
-  Rui Fan, Mohammud Junaid Bocus, and Naim Dahnoun, “A novel disparity transformation algorithm for road segmentation,” Information Processing Letters, vol. 140, pp. 18–24, 2018.
-  Massimo Bertozzi and Alberto Broggi, “Gold: A parallel real-time stereo vision system for generic obstacle and lane detection,” IEEE transactions on image processing, vol. 7, no. 1, pp. 62–81, 1998.
-  Yue Wang, Eam Khwang Teoh, and Dinggang Shen, “Lane detection and tracking using b-snake,” Image and Vision computing, vol. 22, no. 4, pp. 269–280, 2004.
-  Karl Kluge and Sridhar Lakshmanan, “A deformable-template approach to lane detection,” in Intelligent Vehicles’ 95 Symposium., Proceedings of the. IEEE, 1995, pp. 54–59.
-  Yan Wang, Li Bai, and Michael Fairhurst, “Robust road modeling and tracking using condensation,” IEEE Transactions on Intelligent Transportation Systems, vol. 9, no. 4, pp. 570–579, 2008.
Yong Zhou, Rong Xu, Xiaofeng Hu, and Qingtai Ye,
“A robust lane detection and tracking method based on computer vision,”Measurement science and technology, vol. 17, no. 4, pp. 736, 2006.
-  Chris Kreucher and Sridhar Lakshmanan, “Lana: a lane extraction algorithm that uses frequency domain features,” IEEE Transactions on Robotics and automation, vol. 15, no. 2, pp. 343–350, 1999.
-  Cláudio Rosito Jung and Christian Roberto Kelber, “An improved linear-parabolic model for lane following and curve detection,” in Computer Graphics and Image Processing, 2005. SIBGRAPI 2005. 18th Brazilian Symposium on. IEEE, 2005, pp. 131–138.
-  Yue Wang, Dinggang Shen, and Eam Khwang Teoh, “Lane detection using spline model,” Pattern Recognition Letters, vol. 21, no. 8, pp. 677–689, 2000.
-  Marcos Nieto, Luis Salgado, Fernando Jaureguizar, and Julian Cabrera, “Stabilization of inverse perspective mapping images based on robust vanishing point estimation,” in Intelligent Vehicles Symposium, 2007 IEEE. IEEE, 2007, pp. 315–320.
-  David Schreiber, Bram Alefs, and Markus Clabian, “Single camera lane detection and tracking,” in Intelligent Transportation Systems, 2005. Proceedings. 2005 IEEE. IEEE, 2005, pp. 302–307.
-  David Hanwell and Majid Mirmehdi, “Detection of lane departure on high-speed roads.,” in ICPRAM (2), 2012, pp. 529–536.
-  Basel Fardi and Gerd Wanielik, “Hough transformation based approach for road border detection in infrared images,” in Intelligent Vehicles Symposium, 2004 IEEE. IEEE, 2004, pp. 549–554.
-  Umar Ozgunalp, Rui Fan, Xiao Ai, and Naim Dahnoun, “Multiple lane detection algorithm based on novel dense vanishing point estimation,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 3, pp. 621–632, 2017.
-  Raphael Labayrade, Didier Aubert, and J-P Tarel, “Real time obstacle detection in stereovision on non flat road geometry through” v-disparity” representation,” in Intelligent Vehicle Symposium, 2002. IEEE. IEEE, 2002, vol. 2, pp. 646–651.
-  Stefan Leutenegger, Margarita Chli, and Roland Y Siegwart, “Brisk: Binary robust invariant scalable keypoints,” in 2011 International conference on computer vision. IEEE, 2011, pp. 2548–2555.
-  Rui Fan, Yanan Liu, Mohammud Junaid Bocus, Lujia Wang, and Ming Liu, “Real-time subpixel fast bilateral stereo,” arXiv preprint arXiv:1807.02044.
-  Aharon Bar Hillel, Ronen Lerner, Dan Levi, and Guy Raz, “Recent progress in road and lane detection: a survey,” Machine vision and applications, vol. 25, no. 3, pp. 727–745, 2014.