Autonomous vehicles are required to perceive their surrounding environment and know their location in the world before they can plan a path to safely navigate to a desired location.
Indeed, accurate lateral localisation is crucial for many adas such as ldw, lka, and Parking Assist systems. However, these systems are susceptible to cluttered environments, where traffic and other obstructions are likely to disrupt the robust performance required for safe operation of the vehicle.
To this end, in previous work, we presented road segmentation [SuleymanovIROS2016], road boundary detection [Suleymanov2018Curb, Suleymanov2019Curb]
, and scene understanding[2018ITSC_kunze], which were about perceiving the environment of autonomous vehicles. Now, we take this perceived information – in this case detected road boundaries – and apply it to solve one of the fundamental tasks of autonomous driving and adas: localisation, or more specifically, lateral localisation. Figure 1 shows an overview of our pipeline, while Figure 2 shows an overview of the vrbd and orbi models that enable this localisation system.
This paper proceeds by reviewing related literature in Section II. Section III describes our approach to include invisible or occluded road boundaries in lateral localisation systems. We describe in Section IV details for implementation, evaluation, and our dataset. Section V discusses results from such an evaluation. Sections VII and VI summarise the findings and suggest further avenues for investigation.
Ii Related Work
Mapping and localisation techniques can be classified into two groups: global and relative[Thrun2001RobustMonteCarlo].
Global techniques for localisation of a robot can be achieved using gnss, but it is not precise enough for autonomous driving as the accuracy is worse than 2-3 metres in an open-sky environment [Ghallabi2018LiDARBasedLaneMarking]. Indeed, consider Figure 3. We observe that there are many gaps along the route when samples from the Oxford RobotCar Dataset [RobotCarDatasetIJRR] are overlaid on a digital map using gps coordinates. Relative techniques based on slam [grisetti2010tutorial]
or ego-motion estimation[borenstein1996measurement] do not employ a global frame of reference but are less susceptible to drift and scale better with vast environments.
Depending on the type of input sensor these approaches can be further categorised by their use of cameras (passive sensors) and lidar or radars (active sensors). Camera-based techniques, such as vo [nister2004visual], are sensitive to lighting conditions, shadows, illumination, under- and overexposure [mcmanus2014shady]. Using dense maps in lidar-based approaches, such as lo [zhang2014loam], is usually very accurate but computationally expensive for running in real-time. There is burgeoning interest in radar-based techniques, including ro [cen2018precise, 2019ITSC_aldera, 2019ICRA_aldera] and localisation [KidnappedRadarArXiv]. As of yet, these techniques do not offer a rich enough understanding of scene semantics to be employed for road boundary problems.
Finally, regardless of the sensor used, feature matching is often used to match inputs from the sensors to maps. Indeed, Lane markings, traffic signs, feature points or road boundaries can be used as features for localisation [sun20193d]. The long and continuous shape of road boundaries makes them stable and robust features for localisation in the lateral direction as they capture the structure of roads.
In this paper we use sparse lidar-based detected road boundaries for computational ease and infer occluded road boundaries to meet the challenge of partially or fully occlusion by obstacles or other road users.
Our cross-track localisation approach, the workflow for which is shown in Figure 1, is designed to:
Demonstrate usability of the outputs of a lidar-based road boundary detection approach for lateral localisation, and
Demonstrate a gain in performance that inferring of occluded road boundaries could bring.
This section details (in Sections III-C and III-B) the two subsystems inherent to our pipeline. As to be detailed in Section IV-A, the system acts between a map and a live trajectory, each taken from a different dataset (or collection of sensory records captured during a foray by the vehicle). We begin by describing a preprocessing step for all map and live lidar scans.
Iii-a lidar ipm
Labels and 3D lidar scans are transformed into 2D bird’s eye view images – or ipm – to obtain input images and their road boundary masks. Note that the 3D lidar scans are trimmed to keep only the points that are close to the road surface before transforming them into ipm images. Our test vehicle has two 3D lidar and we generate ipm images for both of them before combining them in a single frame, accounting for the 6dof extrinsic offset in sensor mount positions.
Iii-B Visible and occluded road boundary detection
We adopt our lidar-based vrbd and orbi models presented in [Suleymanov2018Curb, Suleymanov2019Curb] to run the inference over the map dataset with the roi of 24x24 squared metre, where 1 px correspond to in the real world. Outputs of the detected road boundaries are stored in the map with corresponding timestamps. Figure 2 shows and describes this in more detail. Figures 6 and 5 show examples of visible-only road boundary detection as well as inference of the position of occluded road boundaries.
Iii-C Map matching
The second dataset is used as a live input and the detected road boundaries from the second dataset are matched against the map dataset to perform lateral localisation. We assume that the initial guess of the location of the vehicle in the map are provided by a vision-based localiser [LinegarICRA2015]. Note that the vision-based localiser only provides timestamps of corresponding images from the map without providing initial pose. We use the icp algorithm [Besl1992ICP] to perform the matching process and estimate the transformations between the live inputs and the map. icp is a well-known algorithm that is used for matching point clouds, where the algorithm iteratively updates the transformation between two point clouds to minimise the distance between them. We adopt icp to estimate the transformation between live and map samples. The detected road boundary masks are binarised and converted into point clouds and then matched with icp as shown in Figure 4.
Iv Experimental Setup
This section details our experimental design in obtaining the results to follow in Section V.
Iv-a Dataset curation
To run our experiments we use a pair of datasets – one dataset as a map and the other as a live input. Our experiments cover two pairs of datasets in central Oxford, one taken from [RobotCarDatasetIJRR] and the other from [RadarRobotCarDatasetArXiv]. The pair taken from [RobotCarDatasetIJRR] are collected in April, 2018 and January 2019, respectively, and we refer to this experiment as Experiment 1. The pair taken from [RadarRobotCarDatasetArXiv] are both collected in January, 2019 and we refer to this experiment as Experiment 2.
Iv-B Data annotation
Fine-grained hand annotation of road boundaries from images would be a very time consuming process and it would be impossible to exactly annotate position of occluded road boundaries. To avoid that we annotate 3D point clouds that are collected using 2D laser. We then accumulate subsequent vertical laser scans in a coherent coordinate frame using vo to estimate vehicle’s motion in order to compute transformations between subsequent scans.
Iv-C Performance metrics
Our results are relative measures of improved lidar lateral localisation when including detections of occluded road boundaries over systems which do not have access to those detections and only use visible road boundaries. We compare the agreement of the returned poses to the pose available from a state-of-the-art vision-based localiser [LinegarICRA2015].
This section presents instrumentation of the metrics discussed in Section IV over both experiments: Experiment 1 and Experiment 2.
We present in Figure 7 qualitative examples of road boundary based icp matching for localisation. These examples demonstrate that icp accurately estimates the transformations between samples irrespective of the structure of the detected road boundaries. Small amounts of noise in the detection does not change the overall estimation of the transformations given that the road boundaries are detected in a balanced way over the sections of the true boundaries. icp can accurately estimate the transformation in the presence of detected road boundaries on one side of the road.
Consider Figure 8. In these illustrated cases icp fails to match the road boundaries accurately to generate transformations between samples. This happens when the detected road boundaries between live and map inputs are unbalanced, which forces icp to rotate the live inputs as keeping them parallel is more costly. This can be fixed using more sophisticated matching techniques. Consider, however, Figure 9 in which we include in the matching detections of occluded road boundaries as proposed in this paper. Without more sophisticated matching techniques, we are now able to obtain sane transformations.
|Visible only||Visible and occluded|
|Error range||Number of samples||Percentage||Number of samples||Percentage|
|Within 0.1 metre||20389||65.56%||21934||70.53%|
|Within 0.3 metre||27479||88.36%||28349||91.15%|
|Within 0.5 metre||29005||93.26%||29617||95.23%|
|Within 1 metre||30124||96.86%||30504||98.08%|
|Matching method||Map dataset||Live dataset||Visible only||Visible and occluded|
|ICP||2018-04-30||2019-01-18||18.95 cm||14.72 cm|
|ICP||2019-01-10-11||2019-01-10-12||18.54 cm||12.18 cm|
|ICP with worst rejection||2019-01-10-11||2019-01-10-12||9.33 cm||7.51 cm|
the estimated road boundary based lateral localisation results with the lateral localisation of the vision-based localiser. Note that the ipm images that are used for estimating the lateral localisation are interpolated to match the timestamps of the camera images that the vision-based localiser uses.
Here, we calculate average lateral error (mean absolute error) and yaw error for the lateral localisation based on:
Visible road boundaries only, and
A combination of all road boundaries.
For Experiment 1, our results show that the average lateral localisation error based on visible road boundaries only is . Including the inferred occluded road boundaries decreases the error by to . Similarly, the yaw error decreases from to . Similar boons to performance for Experiment 2 can be read from Table II.
We analyse in Tables I and 10 the output results by counting the number of samples that have a lateral error within 1, 0.5, 0.3, and 0.1 metres. Using the inferred occluded road boundaries increases the percentage of the number of samples within 1 metre from 96.86% to 98.08% Similar gains are achieved for the number of samples within 0.5, 0.3, and 0.1 metres. Overall, 70.53% of samples have lateral error less than 10 cm in contrast with the vision-based localiser. To visualise the lateral error of the samples we will plot them as histograms.
The lateral error of all samples are displayed in the timeline of Figure 11 to show that the majority of the samples (98.08%) has an error less than one metre and there are only a small number of peaks where localisation fails. We also plot a zoomed in section of the timeline to show that the large number of samples (70.53%) has a lateral error less than 10 cm.
Another timeline plotted in Figure 12 contains only 1000 samples and shows the importance of occluded road boundaries. The blue points in the figure represent the lateral error based on only visible road boundaries, where the red points are the localisation errors with all road boundaries. Overall, we observe that the blue points (only visible road boundaries) have higher values than the red ones (visible and occluded road boundaries).
Finally, Figure 13 shows distributions of the magnitude of localisation failure for Experiment 2
. These can be understood as the probability that a localisation failure (measured as a lateral deviation) will occur on a given localisation run. We observe for all ranges of failures the inclusion of inferred positions of occluded road boundaries is beneficial to the robustness of a lateral localisation system.
In this paper we presented a road boundary based lateral localisation method that provides accurate results using lidar data. The proposed system leverages new methods for detecting occluded road boundaries in sparse lidar data to improve lidar-based lateral localisation beyond that which is currently possible by naïve visible road boundary detection based methods. This is particularly evident and relevant in environments which are challenged by dense traffic conditions. Additionally, the proposed system is innovative in comparison to systems which may rely on gps initialisation of the lateral localisation system by fusing the initial pose hint from onboard sensors such as cameras (although the method is agnostic to how this hint is sourced), which is particularly important in gps-denied environments. This is also beneficial in comparison to existing systems which perform along-path localisation with lidar and rely in that sense on the detection of objects in the scene (e.g. traffic signs), which may also be occluded (or missing) in particularly challenging environments. We demonstrated that using inferred occluded road boundaries improves performance in environments which are challenged by the presence of obstructions.
Vii Future Work
In the future we plan to integrate the system on the all-weather platform described in [kyberd2019] in challenging, unstructured environments, requiring a new labelling regime to identify the boundaries of driveable paths.
Matthew Gadd is supported by Innovate UK under CAV2 – Stream 1 CRD (DRIVEN). Lars Kunze is supported by the Assuring Autonomy International Programme, a partnership between Lloyd’s Register Foundation and the University of York. Paul Newman is supported by UK EPSRC programme grant EP/M019918/1.