Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components

by   Cumhur Erkan Tuncali, et al.

Autonomous vehicles are complex systems that are challenging to test and debug. A requirements-driven approach to the development process can decrease the resources required to design and test these systems, while simultaneously increasing the reliability. We present a testing framework that uses signal temporal logic (STL), which is a precise and unambiguous requirements language. Our framework evaluates test cases against the STL formulae and additionally uses the requirements to automatically identify test cases that fail to satisfy the requirements. One of the key features of our tool is the support for machine learning (ML) components in the system design, such as deep neural networks. The framework allows evaluation of the control algorithms, including the ML components, and it also includes models of CCD camera, lidar, and radar sensors, as well as the vehicle environment. We use multiple methods to generate test cases, including covering arrays, which is an efficient method to search discrete variable spaces. The resulting test cases can be used to debug the controller design by identifying controller behaviors that do not satisfy requirements. The test cases can also enhance the testing phase of development by identifying critical corner cases that correspond to the limits of the system's allowed behaviors. We present STL requirements for an autonomous vehicle system, which capture both component-level and system-level behaviors. Additionally, we present three driving scenarios and demonstrate how our requirements-driven testing framework can be used to identify critical system behaviors, which can be used to support the development process.



There are no comments yet.


page 1

page 6

page 8

page 9

page 10

page 14


Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

Many organizations are developing autonomous driving systems, which are ...

ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles

In this paper, we present ViSTA, a framework for Virtual Scenario-based ...

DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars

Recent advances in Deep Neural Networks (DNNs) have led to the developme...

Callisto: Entropy based test generation and data quality assessment for Machine Learning Systems

Machine Learning (ML) has seen massive progress in the last decade and a...

Automated Generation of Test Models from Semi-Structured Requirements

[Context:] Model-based testing is an instrument for automated generation...

RMT: Rule-based Metamorphic Testing for Autonomous Driving Models

Deep neural network models are widely used for perception and control in...

Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems

Systematically testing models learned from neural networks remains a cru...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.