LiDAR Lateral Localisation Despite Challenging Occlusion from Traffic

03/10/2020 ∙ by Tarlan Suleymanov, et al. ∙ University of Oxford 0

This paper presents a system for improving the robustness of LiDAR lateral localisation systems. This is made possible by including detections of road boundaries which are invisible to the sensor (due to occlusion, e.g. traffic) but can be located by our Occluded Road Boundary Inference Deep Neural Network. We show an example application in which fusion of a camera stream is used to initialise the lateral localisation. We demonstrate over four driven forays through central Oxford - totalling 40 km of driving - a gain in performance that inferring of occluded road boundaries brings.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Autonomous vehicles are required to perceive their surrounding environment and know their location in the world before they can plan a path to safely navigate to a desired location.

Indeed, accurate lateral localisation is crucial for many adas such as ldw, lka, and Parking Assist systems. However, these systems are susceptible to cluttered environments, where traffic and other obstructions are likely to disrupt the robust performance required for safe operation of the vehicle.

To this end, in previous work, we presented road segmentation [SuleymanovIROS2016], road boundary detection [Suleymanov2018Curb, Suleymanov2019Curb]

, and scene understanding 

[2018ITSC_kunze], which were about perceiving the environment of autonomous vehicles. Now, we take this perceived information – in this case detected road boundaries – and apply it to solve one of the fundamental tasks of autonomous driving and adas: localisation, or more specifically, lateral localisation. Figure 1 shows an overview of our pipeline, while Figure 2 shows an overview of the vrbd and orbi models that enable this localisation system.

This paper proceeds by reviewing related literature in Section II. Section III describes our approach to include invisible or occluded road boundaries in lateral localisation systems. We describe in Section IV details for implementation, evaluation, and our dataset. Section V discusses results from such an evaluation. Sections VII and VI summarise the findings and suggest further avenues for investigation.

Figure 1: On overview of the pipeline proposed in this paper. After localisation is coarsely initialised, the live lidar scan is passed through our vrbd and orbi models. This gives us not only visible road boundaries but also inferred locations for the occluded parts of those road boundaries. These are then matched to a map which has been similarly processed for visible and occluded road boundaries. Note that localisation is coarsely initialised by a camera stream but this part of the system is interchangeable with e.g. lidar place recognition systems such as [kim20191].

Ii Related Work

Mapping and localisation techniques can be classified into two groups: global and relative 

[Thrun2001RobustMonteCarlo].

Global techniques for localisation of a robot can be achieved using gnss, but it is not precise enough for autonomous driving as the accuracy is worse than 2-3 metres in an open-sky environment [Ghallabi2018LiDARBasedLaneMarking]. Indeed, consider Figure 3. We observe that there are many gaps along the route when samples from the Oxford RobotCar Dataset [RobotCarDatasetIJRR] are overlaid on a digital map using gps coordinates. Relative techniques based on slam [grisetti2010tutorial]

or ego-motion estimation 

[borenstein1996measurement] do not employ a global frame of reference but are less susceptible to drift and scale better with vast environments.

Figure 2: Our LIDAR-based coupled approach for road boundary detection. Given a pair of ipm images from left and right LIDARs, the fully convolutional VRBD model detects visible road boundaries and then passes to the ORBI model for inference of occluded road boundaries. The second model contains 3 base layers, intra-layer convolutions and 3 layers of parameterised multi-scale predictions at the end.

Depending on the type of input sensor these approaches can be further categorised by their use of cameras (passive sensors) and lidar or radars (active sensors). Camera-based techniques, such as vo [nister2004visual], are sensitive to lighting conditions, shadows, illumination, under- and overexposure [mcmanus2014shady]. Using dense maps in lidar-based approaches, such as lo [zhang2014loam], is usually very accurate but computationally expensive for running in real-time. There is burgeoning interest in radar-based techniques, including ro [cen2018precise, 2019ITSC_aldera, 2019ICRA_aldera] and localisation [KidnappedRadarArXiv]. As of yet, these techniques do not offer a rich enough understanding of scene semantics to be employed for road boundary problems.

Figure 3: Samples of vehicle position are overlaid on a digital map using gps coordinates. Although the dataset is a complete loop, we observe many gaps along the route.

Finally, regardless of the sensor used, feature matching is often used to match inputs from the sensors to maps. Indeed, Lane markings, traffic signs, feature points or road boundaries can be used as features for localisation [sun20193d]. The long and continuous shape of road boundaries makes them stable and robust features for localisation in the lateral direction as they capture the structure of roads.

In this paper we use sparse lidar-based detected road boundaries for computational ease and infer occluded road boundaries to meet the challenge of partially or fully occlusion by obstacles or other road users.

Iii Methodology

Figure 4: Given the detected road boundaries mask of a scan, it is binarised and transformed into a point cloud to match with a scan from the map. icp is used to match the road boundaries and estimate the transformations between the frames. This example shows the accurate estimation of the transformation despite the undetected section of the road boundaries in the map frame.

Our cross-track localisation approach, the workflow for which is shown in Figure 1, is designed to:

  1. Demonstrate usability of the outputs of a lidar-based road boundary detection approach for lateral localisation, and

  2. Demonstrate a gain in performance that inferring of occluded road boundaries could bring.

This section details (in Sections III-C and III-B) the two subsystems inherent to our pipeline. As to be detailed in Section IV-A, the system acts between a map and a live trajectory, each taken from a different dataset (or collection of sensory records captured during a foray by the vehicle). We begin by describing a preprocessing step for all map and live lidar scans.

Iii-a lidar ipm

Labels and 3D lidar scans are transformed into 2D bird’s eye view images – or ipm – to obtain input images and their road boundary masks. Note that the 3D lidar scans are trimmed to keep only the points that are close to the road surface before transforming them into ipm images. Our test vehicle has two 3D lidar and we generate ipm images for both of them before combining them in a single frame, accounting for the 6dof extrinsic offset in sensor mount positions.

Figure 5: Output samples of detected visible road boundaries by the VRBD model with a ROI of 24x24 squared metre.
Figure 6: Output samples of detected visible and inferred occluded road boundaries by the VRBD and ORBI models with a ROI of 24x24 squared metre.

Iii-B Visible and occluded road boundary detection

We adopt our lidar-based vrbd and orbi models presented in [Suleymanov2018Curb, Suleymanov2019Curb] to run the inference over the map dataset with the roi of 24x24 squared metre, where 1 px correspond to in the real world. Outputs of the detected road boundaries are stored in the map with corresponding timestamps. Figure 2 shows and describes this in more detail. Figures 6 and 5 show examples of visible-only road boundary detection as well as inference of the position of occluded road boundaries.

Iii-C Map matching

The second dataset is used as a live input and the detected road boundaries from the second dataset are matched against the map dataset to perform lateral localisation. We assume that the initial guess of the location of the vehicle in the map are provided by a vision-based localiser [LinegarICRA2015]. Note that the vision-based localiser only provides timestamps of corresponding images from the map without providing initial pose. We use the icp algorithm [Besl1992ICP] to perform the matching process and estimate the transformations between the live inputs and the map. icp is a well-known algorithm that is used for matching point clouds, where the algorithm iteratively updates the transformation between two point clouds to minimise the distance between them. We adopt icp to estimate the transformation between live and map samples. The detected road boundary masks are binarised and converted into point clouds and then matched with icp as shown in Figure 4.

Iv Experimental Setup

This section details our experimental design in obtaining the results to follow in Section V.

Iv-a Dataset curation

To run our experiments we use a pair of datasets – one dataset as a map and the other as a live input. Our experiments cover two pairs of datasets in central Oxford, one taken from [RobotCarDatasetIJRR] and the other from [RadarRobotCarDatasetArXiv]. The pair taken from [RobotCarDatasetIJRR] are collected in April, 2018 and January 2019, respectively, and we refer to this experiment as Experiment 1. The pair taken from [RadarRobotCarDatasetArXiv] are both collected in January, 2019 and we refer to this experiment as Experiment 2.

Iv-B Data annotation

Fine-grained hand annotation of road boundaries from images would be a very time consuming process and it would be impossible to exactly annotate position of occluded road boundaries. To avoid that we annotate 3D point clouds that are collected using 2D laser. We then accumulate subsequent vertical laser scans in a coherent coordinate frame using vo to estimate vehicle’s motion in order to compute transformations between subsequent scans.

Iv-C Performance metrics

Our results are relative measures of improved lidar lateral localisation when including detections of occluded road boundaries over systems which do not have access to those detections and only use visible road boundaries. We compare the agreement of the returned poses to the pose available from a state-of-the-art vision-based localiser [LinegarICRA2015].

V Results

This section presents instrumentation of the metrics discussed in Section IV over both experiments: Experiment 1 and Experiment 2.

Figure 7: Examples of road boundary based icp matching for localisation. These examples demonstrate that icp accurately estimates the transformations between samples irrespective of the structure of the detected road boundaries since the detected road boundaries between samples are balanced over the sections of the true boundaries. Results obtained by processing the Experiment 1 dataset pair.

We present in Figure 7 qualitative examples of road boundary based icp matching for localisation. These examples demonstrate that icp accurately estimates the transformations between samples irrespective of the structure of the detected road boundaries. Small amounts of noise in the detection does not change the overall estimation of the transformations given that the road boundaries are detected in a balanced way over the sections of the true boundaries. icp can accurately estimate the transformation in the presence of detected road boundaries on one side of the road.

Figure 8: ICP matching failure examples, where the detected road boundaries on the right hand side of the road are unbalanced between samples. Results obtained by processing the Experiment 1 dataset pair.

Consider Figure 8. In these illustrated cases icp fails to match the road boundaries accurately to generate transformations between samples. This happens when the detected road boundaries between live and map inputs are unbalanced, which forces icp to rotate the live inputs as keeping them parallel is more costly. This can be fixed using more sophisticated matching techniques. Consider, however, Figure 9 in which we include in the matching detections of occluded road boundaries as proposed in this paper. Without more sophisticated matching techniques, we are now able to obtain sane transformations.

Figure 9: Comparison: icp matching based on detected visible road boundaries only (top) and based on all road boundaries (bottom). Results obtained by processing the Experiment 1 dataset pair.
Visible only Visible and occluded
Error range Number of samples Percentage Number of samples Percentage
Within 0.1 metre 20389 65.56% 21934 70.53%
Within 0.3 metre 27479 88.36% 28349 91.15%
Within 0.5 metre 29005 93.26% 29617 95.23%
Within 1 metre 30124 96.86% 30504 98.08%
Table I: Road boundary based lateral localisation results, comparing localisation based on visible road boundaries only with localisation based on a combination of visible and occluded road boundaries. The results show that using the inferred occluded road boundaries always improves performance. Results obtained by processing the Experiment 1 dataset pair.
Matching method Map dataset Live dataset Visible only Visible and occluded
ICP 2018-04-30 2019-01-18 18.95 cm 14.72 cm
ICP 2019-01-10-11 2019-01-10-12 18.54 cm 12.18 cm
ICP with worst rejection 2019-01-10-11 2019-01-10-12 9.33 cm 7.51 cm
Table II: Average lateral localisation error for both the Experiment 1 and Experiment 2 dataset pairs.

We compare in Figures 13, 12, 11, 10, II and I

the estimated road boundary based lateral localisation results with the lateral localisation of the vision-based localiser. Note that the ipm images that are used for estimating the lateral localisation are interpolated to match the timestamps of the camera images that the vision-based localiser uses.

Here, we calculate average lateral error (mean absolute error) and yaw error for the lateral localisation based on:

  1. Visible road boundaries only, and

  2. A combination of all road boundaries.

For Experiment 1, our results show that the average lateral localisation error based on visible road boundaries only is . Including the inferred occluded road boundaries decreases the error by to . Similarly, the yaw error decreases from to . Similar boons to performance for Experiment 2 can be read from Table II.

Figure 10: Histograms of road boundary based lateral error. The top left histogram includes all samples from the dataset, while the remaining histograms progressively narrow the displayed error range (horizontal axis). We observe that the majority of the samples (70.53%) have a maximum lateral error smaller than 10 cm. Results obtained by processing the Experiment 1 dataset pair.

We analyse in Tables I and 10 the output results by counting the number of samples that have a lateral error within 1, 0.5, 0.3, and 0.1 metres. Using the inferred occluded road boundaries increases the percentage of the number of samples within 1 metre from 96.86% to 98.08% Similar gains are achieved for the number of samples within 0.5, 0.3, and 0.1 metres. Overall, 70.53% of samples have lateral error less than 10 cm in contrast with the vision-based localiser. To visualise the lateral error of the samples we will plot them as histograms.

Figure 11: Lateral error of all samples displayed in a timeline (top), where we observe that there are only a small number of peaks where localisation failed. Progressively narrowing the displayed error range (vertical axis) shows that the majority of the samples (70.53%) have a maximum lateral error within 10 cm. Results obtained by processing the Experiment 1 dataset pair.

The lateral error of all samples are displayed in the timeline of Figure 11 to show that the majority of the samples (98.08%) has an error less than one metre and there are only a small number of peaks where localisation fails. We also plot a zoomed in section of the timeline to show that the large number of samples (70.53%) has a lateral error less than 10 cm.

Figure 12: A timeline of 1000 samples, displaying lateral errors based on only visible road boundaries (blue points) and errors based on all road boundaries (red points). We observe that the blue points are generally larger than the red ones, indicating that using all road boundaries is better for performance. Results obtained by processing the Experiment 1 dataset pair.

Another timeline plotted in Figure 12 contains only 1000 samples and shows the importance of occluded road boundaries. The blue points in the figure represent the lateral error based on only visible road boundaries, where the red points are the localisation errors with all road boundaries. Overall, we observe that the blue points (only visible road boundaries) have higher values than the red ones (visible and occluded road boundaries).

Finally, Figure 13 shows distributions of the magnitude of localisation failure for Experiment 2

. These can be understood as the probability that a localisation failure (measured as a lateral deviation) will occur on a given localisation run. We observe for all ranges of failures the inclusion of inferred positions of occluded road boundaries is beneficial to the robustness of a lateral localisation system.

Figure 13: Probability that a localisation failure (measured as a lateral deviation in the -direction) will occur on a given localisation run. Results obtained by processing the Experiment 2 dataset pair.

Vi Conclusions

In this paper we presented a road boundary based lateral localisation method that provides accurate results using lidar data. The proposed system leverages new methods for detecting occluded road boundaries in sparse lidar data to improve lidar-based lateral localisation beyond that which is currently possible by naïve visible road boundary detection based methods. This is particularly evident and relevant in environments which are challenged by dense traffic conditions. Additionally, the proposed system is innovative in comparison to systems which may rely on gps initialisation of the lateral localisation system by fusing the initial pose hint from onboard sensors such as cameras (although the method is agnostic to how this hint is sourced), which is particularly important in gps-denied environments. This is also beneficial in comparison to existing systems which perform along-path localisation with lidar and rely in that sense on the detection of objects in the scene (e.g. traffic signs), which may also be occluded (or missing) in particularly challenging environments. We demonstrated that using inferred occluded road boundaries improves performance in environments which are challenged by the presence of obstructions.

Vii Future Work

In the future we plan to integrate the system on the all-weather platform described in [kyberd2019] in challenging, unstructured environments, requiring a new labelling regime to identify the boundaries of driveable paths.

Acknowledgment

Matthew Gadd is supported by Innovate UK under CAV2 – Stream 1 CRD (DRIVEN). Lars Kunze is supported by the Assuring Autonomy International Programme, a partnership between Lloyd’s Register Foundation and the University of York. Paul Newman is supported by UK EPSRC programme grant EP/M019918/1.

References