Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving

by   Wei Zhou, et al.

One of the fundamental challenges in the design of perception systems for autonomous vehicles is validating the performance of each algorithm under a comprehensive variety of operating conditions. In the case of vision-based semantic segmentation, there are known issues when encountering new scenarios that are sufficiently different to the training data. In addition, even small variations in environmental conditions such as illumination and precipitation can affect the classification performance of the segmentation model. Given the reliance on visual information, these effects often translate into poor semantic pixel classification which can potentially lead to catastrophic consequences when driving autonomously. This paper presents a novel method for analysing the robustness of semantic segmentation models and provides a number of metrics to evaluate the classification performance over a variety of environmental conditions. The process incorporates an additional sensor (lidar) to automate the process, eliminating the need for labour-intensive hand labelling of validation data. The system integrity can be monitored as the performance of the vision sensors are validated against a different sensor modality. This is necessary for detecting failures that are inherent to vision technology. Experimental results are presented based on multiple datasets collected at different times of the year with different environmental conditions. These results show that the semantic segmentation performance varies depending on the weather, camera parameters, existence of shadows, etc.. The results also demonstrate how the metrics can be used to compare and validate the performance after making improvements to a model, and compare the performance of different networks.


page 1

page 3

page 4

page 6

page 7

page 8

page 9

page 12


RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving

LiDAR has become a standard sensor for autonomous driving applications a...

Few-shot 3D LiDAR Semantic Segmentation for Autonomous Driving

In autonomous driving, the novel objects and lack of annotations challen...

Adapting Semantic Segmentation Models for Changes in Illumination and Camera Perspective

Semantic segmentation using deep neural networks has been widely explore...

On the Real-World Adversarial Robustness of Real-Time Semantic Segmentation Models for Autonomous Driving

The existence of real-world adversarial examples (commonly in the form o...

A Multimodal Vision Sensor for Autonomous Driving

This paper describes a multimodal vision sensor that integrates three ty...

White paper on Selected Environmental Parameters affecting Autonomous Vehicle (AV) Sensors

Autonomous Vehicles (AVs) being developed these days rely on various sen...

Capsule Neural Network based Height Classification using Low-Cost Automotive Ultrasonic Sensors

High performance ultrasonic sensor hardware is mainly used in medical ap...

Please sign up or login with your details

Forgot password? Click here to reset