A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

03/23/2022
by   Dmytro Humeniuk, et al.
0

Many modern cyber physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Testing such systems poses numerous challenges: not only should the system inputs be varied, but also the surrounding environment should be accounted for. A number of tools have been developed to test the system model for the possible inputs falsifying its requirements. However, they are not directly applicable to autonomous cyber physical systems, as the inputs to their models are generated while operating in a virtual environment. In this paper, we aim to design a search based framework, named AmbieGen, for generating diverse fault revealing test scenarios for autonomous cyber physical systems. The scenarios represent an environment in which an autonomous agent operates. The framework should be applicable to generating different types of environments. To generate the test scenarios, we leverage the NSGA II algorithm with two objectives. The first objective evaluates the deviation of the observed system behaviour from its expected behaviour. The second objective is the test case diversity, calculated as a Jaccard distance with a reference test case. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane keeping assist system. We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi objective, and random search. Both single and multi objective configurations outperform the random search. Multi objective configuration can find the individuals of the same quality as the single objective, producing more unique test scenarios in the same time budget.

READ FULL TEXT

page 10

page 11

research
06/04/2021

PSY-TaLiRo: A Python Toolbox for Search-Based Test Generation for Cyber-Physical Systems

In this paper, we present the Python package PSY-TaLiRo which is a toolb...
research
02/23/2021

Data Driven Testing of Cyber Physical Systems

Consumer grade cyber-physical systems (CPS) are becoming an integral par...
research
03/22/2022

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

This paper presents an extended version of Deeper, a search-based simula...
research
01/25/2021

Test and Evaluation Framework for Multi-Agent Systems of Autonomous Intelligent Agents

Test and evaluation is a necessary process for ensuring that engineered ...
research
02/19/2020

Interface Modeling for Quality and Resource Management

We develop an interface-modeling framework for quality and resource mana...
research
04/14/2022

Environment Imitation: Data-Driven Environment Model Generation Using Imitation Learning for Efficient CPS Goal Verification

Cyber-Physical Systems (CPS) continuously interact with their physical e...
research
10/20/2021

Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees

Requirements driven search-based testing (also known as falsification) h...

Please sign up or login with your details

Forgot password? Click here to reset