A Tutorial on Sim-ATAV: Simulation-based Adversarial Testing Framework for Autonomous Vehicles

03/26/2019 ∙ by Cumhur Erkan Tuncali, et al. ∙ Arizona State University ∙ 0 ∙

Testing autonomous vehicles in simulation environments is crucial. Sim-ATAV is an open-source framework developed for experimenting with different test generation techniques in simulation environments for research purposes. This document provides a tutorial on Sim-ATAV with a running example.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

POST REPLY

Authors

page 2

page 4

page 6

page 9

page 10

page 14

page 17

page 23
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 An Overview of

[6, 7, 8] is a tool developed for experimenting with different test generation techniques in simulation environments for research purposes as described in [8]. It is mainly developed in Python and it uses the open-source robotics toolbox Webots [2] for 3D scene generation, vehicle and sensor modeling and simulation. can be interfaced with covering array generation tools like ACTS [4] and with falsification tools like [3], which is a toolbox. is publicly available as an open-source project [6].

1 provides a high-level overview of the framework. The main functionality of is provided by Simulation configurator and Simulation Supervisor blocks. Simulation configurator block represents the API for the user script to create a simulation and to receive the results. Simulation Supervisor block represents the part of the framework that executes inside the robotics simulation toolbox Webots. It uses the Webots API to (1) modify the simulation environment, e.g., add/configure vehicles, roads, pedestrians, provide vehicle controllers, (2) execute the simulation, and (3) collect information, e.g., the evolution of vehicle states over the simulation time. Simulation Supervisor receives the requested simulation configuration from the Simulation configurator over socket communication. When the simulation environment is set up, Simulation Supervisor requests Webots to start the simulation. User-provided controllers control the motion of the vehicles and pedestrians. At the end of the simulation, Simulation Supervisor sends the collected simulation trace to Simulation configurator.

External tools like ACTS can be used to generate combinatorial tests with covering arrays. provides functions to read covering array test scenarios from csv files. For falsification, is used to iteratively sample new configurations until a failure case is detected. Optionally, initial samples for the configurations can be read by from the covering array csv files. The sampled configurations in are passed as parameters to test generation functions using the Python interface provided in , by calling Python functions directly inside . The test generation functions return the simulation trace back to (and to ) as return values. More detail on falsification methods is available in [8, 3]. A running example of test generation is provided in the upcoming sections.

Figure 1: An Overview of Sim-ATAV.

Although they are not essential for the test generation purposes, also comes with some vehicle controller implementations as well as some basic perception system, sensor fusion, path planning and control algorithms that can be utilized by the user for Ego or Agent vehicle control.

2 Installation Instructions

requires Python 3.7 and Webots for basic functionality. For some controllers and test generation approaches, there are other requirements like , , TensorFlow, etc. The framework has been tested in Windows

® 10 with specific versions of the required packages.

Firstly, should be either downloaded or cloned using a git client 111: https://cpslab.assembla.com/spaces/sim-atav. For Windows®, the preferred installation approach is to use setup_for_windows.bat. Once executed, it will guide the user through the installation process and automate the process as much as possible. This script is not advanced and may fail for some systems. If the script fails, please try the steps below for a manual installation.

Below are the steps for the manual installation. All the paths are given relative to the root folder for the distribution. In case any problems are experienced during the installation of the packages, most of the packages can also be found in Christoph Gohlke’s website222Christoph Gohlke’s website: https://www.lfd.uci.edu/ gohlke/pythonlibs/:

  • Install Python 3.7-64 Bit

  • Install Webots r2019a.

  • (optional) If the system has CUDA-enabled GPU and it will be utilized for an increased performance:

    • Install CUDA Toolkit 10.0

    • Install CUDNN 7.3.1

  • Download Python_Dependencies from http://www.public.asu.edu/ etuncali/downloads/ and unzip it next to this installation script. The Python wheel (.whl) files should be directly under ./Python_Dependencies/

  • Install commonly used Python packages:

    • Install Numpy+MKL 1.14.6 either from Python_Dependencies folder or from Christoph Gohlke’s website:

      ⬇
      pip3 install –upgrade  Python_Dependencies/numpy-1.14.6+mkl-cp37-cp37m-win_amd64.whl

    • Install scipy 1.2.0:

      ⬇
      pip3 install scipy==1.2.

    • Install scikit-learn:

      ⬇
      pip3 install scikit-learn

    • Install pandas:

      ⬇
      pip3 install pandas

    • Install Absl Py:

      ⬇
      pip3 install absl-py

    • Install matplotlib:

      ⬇
      pip3 install matplotlib

    • Install pykalman:

      ⬇
      pip3 install pykalman

    • Install Shapely:

      ⬇
      pip3 install Shapely

      • If any problems are experienced during the installation of Shapely, it can be installed from Python_Dependencies folder:

        ⬇
        pip3 install Python_Dependencies/Shapely-1.6.4.post1-cp37-cp37m-win_amd64.whl

    • Install dubins:

      ⬇
      pip3 install dubins

      • If any problems are experienced when installing pydubins:

        One option is to go into the folder Python_Dependencies/pydubins and execute python setup.py install. Another option is the following:

        • Download pydubins from github.com/AndrewWalker/pydubins.

        • Do the following changes in dubins/src/dubins.c:

          ⬇
          #ifndef M_PI
          #define M_PI 3.14159265358979323846
          #endif

        • Call python setup.py install inside pydubins folder.

  • For controllers with DNN (Deep Neural Network) object detection:

    ! Currently, Python 3.7 support for Tensorflow is provided by a 3rd party (only for Windows). Installation wheels are provided under Python_Dependencies folder.

    Check if the system CPU supports AVX2 (for increased performance) 333A list of CPUs with AVX2 is available at:
    https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2    .

    • If the system GPU has CUDA cores, CUDNN is installed and the system CPU supports AVX 2: Install Tensorflow-gpu with AVX2 support.

      ⬇
      pip3 install –upgrade Python_Dependencies/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl

    • If the system GPU has CUDA cores, CUDNN is installed and the system CPU does NOT support AVX 2: Install Tensorflow-gpu without AVX2 support.

      ⬇
      pip3 install –upgrade Python_Dependencies/sse2/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl

    • If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU supports AVX 2: Install Tensorflow with AVX2 support.

      ⬇
      pip3 install –user –upgrade Python_Dependencies/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl

    • If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU does NOT support AVX 2: Install Tensorflow without AVX2 support.

      ⬇
      pip3 install –user –upgrade Python_Dependencies/sse2/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl

  • Install Python Dependencies of SqueezeDet (if the existing controllers that use will be used). There is no need to install separately, as it is provided in the framework.

    • Install joblib:

      ⬇
      pip3 install –upgrade joblib

    • Install opencv:

      ⬇
      pip3 install –upgrade opencv-contrib-python

    • Install pillow:

      ⬇
      pip3 install –upgrade Pillow

    • Install easydict:

      ⬇
      pip3 install –upgrade easydict==1.7

      ! If any problems are experienced while installing easydict, try the following:

      ⬇
      cd Python_Dependencies/easydict-1.7\
      python setup.py install
      cd ../..

  • To design Covering Array Tests: Please request a copy and install ACTS from NIST444ACTS tool can be requested from
    https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software/downloadable-tools    .

  • To do robustness-guided falsification, and are needed:

  • After the installation is finished, the Python package wheel files that are under the folder named Python_Dependencies can be deleted to save some disk space.

Setting to Utilize GPU:

If the system has a CUDA-enabled GPU, and CUDA Toolkit, CUDNN are installed, the variable has_gpu should be set to True in the following file to make the experiments use the system GPU for :

Sim_ATAV/classifier/classifier_interface/gpu_check.py.

3 Reference Manual

3.1 Simulation Entities

starts building a testing scenario from an existing Webots world file provided by the user, which we will call as base world file. The user has the option to have all or some of the simulation entities, roads, vehicles, etc., saved in the base world file before the test generation time. The user can use the functionality provided by to add more simulation entities to the world at the time of test generation. This functionality is especially useful when the search space for the tests contain some parameters of the simulation entities such as road width, number of lanes, positions of the vehicles.

This section describes the most commonly used simulation entities that can be programmatically added into the simulation world at the time of test generation. Note that may not provide the functionality to add all possible types of simulation objects that are supported by Webots. In this section, the class names and properties with their default values for the simulation entities supported by are provided. Test Generation Script which is developed by the user typically creates instances of required simulation entities and uses the provided functions to add those entities into the test scenario. For modifying the simulation environment beyond the capabilities of , the user can do the changes manually and save in the base world file or modify the source code of to add or change the capabilities as needed. For a deeper understanding of the possibilities, the reader is advised to get familiar with the Webots simulation environment and details of available simulation entities [2].

The class properties for the simulation entities are provided in the tables below. The first row of each table gives the class name. Other rows start with the property name, gives the default value of the property (the value used if not explicitly changed by the user), and a brief description of the property. For each class, a related source code snippet from a running example is provided. The original source code for the running example can be found as tests/tutorial_example_1.py in the distribution.

2 is an image from the scenario described in the running example. A yellow Ego vehicle is behind a pedestrian walking in the middle of a 3-lane road, and an agent vehicle is approaching from the opposite direction in the next lane. There is a bumpy road surface for a short distance, and a stop sign placed on the right side of the road. This is only a simple example to illustrate how to use .

Figure 2: A view from the generated scenario for the running example.
General information:

In Webots, all simulation entities are kept in a simulation tree and they can be easily accessed using their DEF_NAME property. Position fields are 3D arrays as , keeping the value of the position at each axis in Webots coordinate system, where is typically the vertical axis to the ground (in practice, this depends on the Webots world file provided by the user). Rotation fields are 4D arrays as , where

represents a rotation vector and

represents the rotation around this vector in clockwise direction.

3.1.1 Simulation Environment

Once an object is created for a simulation entity, the user can utilize the corresponding function provided by to add the entity to the scenario. An easier alternative is to utilize the SimEnvironment class provided by . An object of this class can be populated with the necessary simulation entities and passed to the simulation environment with a single function call. 1 summarizes the SimEnvironment class.

Class Name: SimEnvironment
Property Default Description
fog None Keeps fog object.
heart_beat_config None The heartbeat configuration.
view_follow_config None The viewpoint configuration.
ego_vehicles_list [] The list of Ego vehicles.
agent_vehicles_list [] The list of agent vehicles.
pedestrians_list [] The list of pedestrians.
road_list [] The list of roads.
road_disturbances_list [] The list of road disturbances.
generic_sim_objects_list [] The list of generic simulation objects.
control_params_list [] The list of controller parameters that will be set in the run time.
initial_state_config_list [] The list of initial state configurations.
data_log_description_list [] The list of data log descriptions.
data_log_period_ms None Data log period (ms).
Table 1: Simulation Environment Class.

For the running example, we start with an empty simulation environment. Listing 1 creates an empty SimEnvironment object that will later keep the required simulation entities.

⬇
1from Sim_ATAV.simulation_configurator.sim_environment \
2    import SimEnvironment
3
4sim_environment = SimEnvironment()
Listing 1: Source code for creating a simulation environment.

3.1.2 Road

A user-configurable road structure to use in the simulation environment is given in 2.

Class Name: WebotsRoad
Property Default Description
def_name “STRROAD” Name as it appears in the simulation tree.
road_type “StraightRoadSegment” Road type name as used by Webots.
rotation [0, 1, 0, math.pi/2] Rotation of the road
position [0, 0.02, 0] Starting position.
number_of_lanes 2 Number of lanes.
width number_of_lanes * 3.5 Road width (m).
length 1000 Road length (m).
Table 2: Simulation Entity: Road.

Listing 2 provides a source code snippet that creates a 3-lane straight road segment lying between 1000m and -1000m along the x-axis.

⬇
1from Sim_ATAV.simulation_control.webots_road import WebotsRoad
2
3road = WebotsRoad(number_of_lanes=3)
4road.rotation = [0, 1, 0, -math.pi / 2]
5road.position = [1000, 0.02, 0]
6road.length = 2000.0
7
8# Add the road into simulation environment object:
9sim_environment.road_list.append(road)
Listing 2: Source code for creating a road.

3.1.3 Vehicle

A user-configurable vehicle class to use in the simulation environment in 3.

Class Name: WebotsVehicle
Property Default Description
def_name “” Name as it appears in Webots simulation tree.
vhc_id 0 Integer ID for referencing to the vehicle.
vehicle_model “AckermannVehicle” Vehicle model can be any model name available in Webots.
rotation [0, 1, 0, 0] Rotation of the object.
current_position [0, 0.3, 0] x,y,z values of the position.
color [1, 1, 1] R,G,B values of the color in the range [0,1].
controller “void” Name of the vehicle controller.
is_controller_name_absolute False Indicates where to find the vehicle controller. Find the details below.
vehicle_parameters [] Additional parameters for the vehicle object.
controller_parameters [] Parameters that will be sent to the vehicle controller.
controller_arguments True Arguments passed to the vehicle controller executable.
sensor_array [] List of sensors on the vehicle (WebotsSensor objects).
Table 3: Simulation Entity: Vehicle.

The vehicle_model field of a vehicle object should match with the models available in Webots (or any custom models added to Webots by the user). Webots r2019a version provides vehicle model options ToyotaPrius, CitroenCZero, BmwX5, RangeRoverSportSVR, LincolnMKZ, TeslaModel3 as well as truck, motorcycle and tractor models. When is_controller_name_absolute is set to true, Webots will load the given controller from Webots_Projects/controllers folder, otherwise, it will load the controller named vehicle_controller which is located under the same folder but will take the controller name as an argument.

We can now create vehicles and place them on the road that was created above. Listing 3 provides an example source code snippet that creates an Ego vehicle at the position , , and an agent vehicle at the position , , vehicles facing toward each other. The controllers for the vehicles are set and the controller arguments are provided. The arguments accepted are controller-specific. The vehicle controllers used in this example can be found under the folder Webots_Projects/controllers.

⬇
1from Sim_ATAV.simulation_control.webots_vehicle import WebotsVehicle
2
3# Ego vehicle
4ego_x_pos = 20.0 # Setting the x position of the Ego vehicle in a variable.
5
6vhc_obj = WebotsVehicle()
7vhc_obj.current_position = [ego_x_pos, 0.35, 0.0]
8vhc_obj.current_orientation = math.pi/2
9vhc_obj.rotation = [0.0, 1.0, 0.0, vhc_obj.current_orientation]
10vhc_obj.vhc_id = 1
11vhc_obj.color = [1.0, 1.0, 0.0]
12vhc_obj.set_vehicle_model(’ToyotaPrius’)
13# Name of our controller python file is ’automated_driving_with_fusion2’:
14vhc_obj.controller = ’automated_driving_with_fusion2’
15# Controller will be found directly under controllers folder:
16vhc_obj.is_controller_name_absolute = True
17# Below is a list of arguments specific to this controller.
18# For reference, the arguments are: car_model, target_speed_kmh, target_lat_pos,
19# self_vhc_id, slow_at_intersection, has_gpu, processor_id
20vhc_obj.controller_arguments.append(’Toyota’)
21vhc_obj.controller_arguments.append(’70.0’)
22vhc_obj.controller_arguments.append(’0.0’)
23vhc_obj.controller_arguments.append(’1’)
24vhc_obj.controller_arguments.append(’True’)
25vhc_obj.controller_arguments.append(’False’)
26vhc_obj.controller_arguments.append(’0’)
27
28# Agent vehicle:
29vhc_obj2 = WebotsVehicle()
30vhc_obj2.current_position = [300.0, 0.35, 3.5]
31vhc_obj2.current_orientation = 0.0
32vhc_obj2.rotation = [0.0, 1.0, 0.0, -math.pi/2]
33vhc_obj2.vhc_id = 2
34vhc_obj2.set_vehicle_model(’TeslaModel3’)
35vhc_obj2.color = [1.0, 0.0,  0.0]
36vhc_obj2.controller = ’path_and_speed_follower’
37vhc_obj2.controller_arguments.append(’20.0’)
38vhc_obj2.controller_arguments.append(’True’)
39vhc_obj2.controller_arguments.append(’3.5’)
40vhc_obj2.controller_arguments.append(’2’)
41vhc_obj2.controller_arguments.append(’False’)
42vhc_obj2.controller_arguments.append(’False’)
43
44# Here, we don’t save the vehicles into simulation environment yet
45# because we will add some sensors on the vehicles below.
Listing 3: Source code for creating a vehicle.

3.1.4 Sensor

4 provides an overview of the WebotsSensor class that can be used to describe a sensor. Webots vehicle models have specific sensor slots on the vehicles. These are typically TOP, CENTER, FRONT, RIGHT, LEFT. The property sensor_type accepts the type of the sensor which should match the type used by Webots. The translation field of the sensor can be used to place the sensor to a different position relative to its corresponding sensor slot. As sensors can vary a lot in terms or parameters, sensor_fields property is provided to accept names and values of the desired parameters as a list of WebotsSensorField objects for flexible configuration of sensors.

Class Name: WebotsSensor
Property Default Description
sensor_type “” Type of the sensor as defined in Webots.
sensor_location FRONT Sensor slot enumeration.
FRONT, CENTER, LEFT, RIGHT, TOP
sensor_fields [] List of WebotsSensorField objects to customize the sensor.
Class Name: WebotsSensorField
field_name “” Name of the field that will be set.
field_val “” Value of the field.
Table 4: Simulation Entity: Sensor.

An example source code of adding sensors to vehicles is provided in Listing 4. In this example, we add a compass and a GPS device that are used by the controllers for path following. The receiver device added to the vehicles is used by the controllers to receive new commands from Simulation Supervisor. We add the receivers because we will later need them to update target paths of the vehicles. Note that although the necessary infrastructure for this approach is provided by , it is implementation specific and not mandated. In this example, we also add a radar device to Ego vehicle for collision avoidance.

⬇
1from Sim_ATAV.simulation_control.webots_sensor import WebotsSensor
2
3# Add a radio receiver to the center sensor slot
4# with the name field set to ’receiver’.
5# This is optional and will be used to communicate with the controller at the run-time.
6vhc_obj.sensor_array.append(WebotsSensor())
7vhc_obj.sensor_array[-1].sensor_location = WebotsSensor.CENTER
8vhc_obj.sensor_array[-1].sensor_type = ’Receiver’
9vhc_obj.sensor_array[-1].add_sensor_field(’name’, ’”receiver”’)
10
11# Add a compass to the center slot with the name field set to ’compass’.
12vhc_obj.sensor_array.append(WebotsSensor())
13vhc_obj.sensor_array[-1].sensor_location = WebotsSensor.CENTER
14vhc_obj.sensor_array[-1].sensor_type = ’Compass’
15vhc_obj.sensor_array[-1].add_sensor_field(’name’, ’”compass”’)
16
17# Add a GPS receiver to the center sensor slot.
18vhc_obj.sensor_array.append(WebotsSensor())
19vhc_obj.sensor_array[-1].sensor_location = WebotsSensor.CENTER
20vhc_obj.sensor_array[-1].sensor_type = ’GPS’
21
22# Add a Radar to the front sensor slot with the name field set to ’radar’.
23vhc_obj.sensor_array.append(WebotsSensor())
24vhc_obj.sensor_array[-1].sensor_type = ’Radar’
25vhc_obj.sensor_array[-1].sensor_location = WebotsSensor.FRONT
26vhc_obj.sensor_array[-1].add_sensor_field(’name’, ’”radar”’)
27
28# Finally, add the vehicle to the simulation environment as an Ego vehicle.
29sim_environment.ego_vehicles_list.append(vhc_obj)
30
31# Similar for the agent vehicle:
32vhc_obj2.sensor_array.append(WebotsSensor())
33vhc_obj2.sensor_array[-1].sensor_location = WebotsSensor.CENTER
34vhc_obj2.sensor_array[-1].sensor_type = ’Receiver’
35vhc_obj2.sensor_array[-1].add_sensor_field(’name’, ’”receiver”’)
36
37vhc_obj2.sensor_array.append(WebotsSensor())
38vhc_obj2.sensor_array[-1].sensor_location = WebotsSensor.CENTER
39vhc_obj2.sensor_array[-1].sensor_type = ’Compass’
40vhc_obj2.sensor_array[-1].add_sensor_field(’name’, ’”compass”’)
41
42vhc_obj2.sensor_array.append(WebotsSensor())
43vhc_obj2.sensor_array[-1].sensor_location = WebotsSensor.CENTER
44vhc_obj2.sensor_array[-1].sensor_type = ’GPS’
45
46# Add the agent vehicle to the simulation environment
47sim_environment.agent_vehicles_list.append(vhc_obj2)
Listing 4: Source code for adding sensor to a vehicle.

3.1.5 Pedestrian

The user-configurable pedestrian class, WebotsPedestrian, to use in the simulation environment is described in 5. Target speed and target path (trajectory) of the pedestrian are automatically passed as arguments to the given controller.

Class Name: WebotsPedestrian
Property Default Description
def_name “PEDESTRIAN” Name as it appears in Webots simulation tree.
ped_id 0 Integer ID for referencing to the pedestrian.
rotation [0, 1, 0, math.pi/2.0] Rotation of the object.
current_position [0, 0, 0] x,y,z values of the position.
shirt_color [0.25, 0.55, 0.2] R,G,B values of the shirt color in the range [0, 1].
pants_color [0.24, 0.25, 0.5] R,G,B values of the pants color in the range [0, 1].
shoes_color [0.28, 0.15, 0.06] R,G,B values of the shoes color in the range [0, 1].
controller “void” Name of the pedestrian controller.
target_speed 0.0 Walking speed of the pedestrian.
trajectory [] Walking path of the pedestrian.
Table 5: Simulation Entity: Pedestrian.

Listing 5 provides an example code snippet to create a pedestrian object, and provide a target speed and a path to define the motion of the pedestrian.

⬇
1from Sim_ATAV.simulation_control.webots_pedestrian import WebotsPedestrian
2
3pedestrian_speed = 3.0 # Setting the pedestrian walking speed in a variable.
4
5pedestrian = WebotsPedestrian()
6pedestrian.ped_id = 1
7pedestrian.current_position = [50.0, 1.3, 0.0]
8pedestrian.shirt_color = [0.0, 0.0, 0.0]
9pedestrian.pants_color = [0.0, 0.0, 1.0]
10pedestrian.target_speed = pedestrian_speed
11# Pedestrian trajectory as a list of x1, y1, x2, y2, …
12pedestrian.trajectory = [50.0, 0.0, 80.0, -3.0, 200.0, 0.0]
13pedestrian.controller = ’pedestrian_control’
14
15# Add the pedestrian into the simulation environment.
16sim_environment.pedestrians_list.append(pedestrian)
Listing 5: Source code for creating a pedestrian.

3.1.6 Fog

User-configurable fog class to use in the simulation environment is summarized in 6.

Class Name: WebotsFog
Property Default Description
def_name “FOG” Name as it appears in Webots simulation tree.
fog_type “LINEAR” Defines the type of the fog gradient.
color [0.93, 0.96, 1.0] R,G,B values of the fog color in the range [0, 1].
visibility_range 1000 Visibility range of the fog (m).
Table 6: Simulation Entity: Fog.

No camera is involved in this scenario, hence fog will not impact the performance of the controller. However, an example source code snippet is provided in Listing 6 for reference.

⬇
1from Sim_ATAV.simulation_control.webots_fog import WebotsFog
2
3# Creating fog with 700m visibility and adding it to the simulation environment.
4sim_environment.fog = WebotsFog()
5sim_environment.fog.visibility_range = 700.
Listing 6: Source code for creating fog.

3.1.7 Road Disturbance

Road disturbance objects are solid triangular objects placed on the road surface to emulate a bumpy road surface. WebotsRoadDisturbance, which is described in 7, contains the properties to describe how the solid objects will be placed to create a bumpy road surface.

Class Name: WebotsRoadDisturbance
Property Default Description
disturbance_id 1 Object ID for later reference.
disturbance_type INTERLEAVED Enumerated type of the disturbance.
INTERLEAVED,
FULL_LANE_LENGTH,
ONLY_LEFT, ONLY_RIGHT
rotation [0, 1, 0, 0] Rotation of the object.
position [0, 0, 0] x,y,z values of the position.
length 100 Length of the bumpy road surface (m).
width 3.5 Width of the corresponding lane.
height 0.06 Height of the disturbance (m).
surface_height 0.02 Height of the corresponding road surface (m).
inter_object_spacing 1.0 Distance between repeating solid objects on the road (m).
Table 7: Simulation Entity: Road Disturbance Object.

An example road disturbance object creation is given in Listing 7.

⬇
1from Sim_ATAV.simulation_control.webots_road_disturbance import WebotsRoadDisturbance
2
3# Create bumpy road for 3m where there are road disturbances on both side of the lane
4# of height 4cm, each separated with 0.5m.
5road_disturbance = WebotsRoadDisturbance()
6road_disturbance.disturbance_type = WebotsRoadDisturbance.TRIANGLE_DOUBLE_SIDED  #i.e., INTERLEAVED
7road_disturbance.rotation = [0, 1, 0, -math.pi / 2.0]  # Same as the road
8road_disturbance.position = [40, 0, 0]
9road_disturbance.width = 3.5
10road_disturbance.length = 3
11road_disturbance.height = 0.04
12road_disturbance.inter_object_spacing = 0.5
13
14# Add road disturbance into the simulation environment object.
15sim_environment.road_disturbances_list.append(road_disturbance)
Listing 7: Source code for creating road disturbance.

3.1.8 Generic Simulation Object

Generic simulation object is for adding any type of object into the simulation which is not covered above. For these objects, there are no checks performed or there are no limitations on the field values. The user can manually create any possible Webots object by setting all of its non-default field values.

Class Name: WebotsSimObject
Property Default Description
def_name “” Name as it appears in Webots simulation tree.
object_name “Tree” Type name of the object. Must be same as the name used by Webots.
object_parameters [] List of (field name, field value) tuples as strings. Names must be same as the field names used by Webots.
Table 8: Simulation Entity: Generic Simulation Object.

In Listing 8, although it is not expected to have an impact on the controller performance, a Stop Sign object is created as a generic simulation object example for reference.

⬇
1from Sim_ATAV.simulation_control.webots_sim_object import WebotsSimObject
2
3sim_obj = WebotsSimObject()
4sim_obj.object_name = ’StopSign’  # The name of the object as defined in Webots.
5# The field names and format as they are used by Webots.
6sim_obj.object_parameters.append((’translation’, ’40 0 6’))
7sim_obj.object_parameters.append((’rotation’, ’0 1 0 1.5708’))
8
9# Add the stop sign as a generic item into the simulation environment object.
10sim_environment.generic_sim_objects_list.append(sim_obj)
Listing 8: Source code for adding a stop sign to the simulation.

3.2 Configuring the Simulation Execution

3.2.1 Additional Controller Parameters

Depending on the application and implementation details, controller parameters can be directly given to the controllers or they can be sent in the run-time. To emulate runtime inputs, such as human commands, can transmit controller commands over virtual radio communication. The controller should be able to read those commands from a radio receiver and a receiver object should be added to one of the sensor slots of the vehicles. This is an optional approach and the user is free to use other approaches such as reading data from a file, using socket communications etc.

Class Name: WebotsControllerParameter
Property Default Description
vehicle_id None ID of the corresponding vehicle.
parameter_name “” Name of the parameter as string.
parameter_data [] Parameter data.
Table 9: Controller parameters.

An example controller parameter creation.

⬇
1from Sim_ATAV.simulation_control.webots_controller_parameter \
2  import WebotsControllerParameter
3
4# —– Controller Parameters:
5# Ego Target Path:
6target_pos_list = [[-1000.0, 0.0], [1000.0, 0.0]]
7
8# Add each target position as a controller parameter for Ego vehicle.
9for target_pos in target_pos_list:
10  sim_environment.controller_params_list.append(
11    WebotsControllerParameter(vehicle_id=1,
12      parameter_name=’target_position’,
13      parameter_data=target_pos))
14
15# Agent Target Path:
16target_pos_list = [[1000.0, 3.5], [145.0, 3.5],  [110.0, -3.5], [-1000.0, -3.5]]
17
18# Add each target position as a controller parameter for agent vehicle.
19for target_pos in target_pos_list:
20  sim_environment.controller_params_list.append(
21    WebotsControllerParameter(vehicle_id=2,
22      parameter_name=’target_position’,
23      parameter_data=target_pos))
Listing 9: Source code for creating controller parameters.

3.2.2 Heartbeat Configuration

Simulation Supervisor can periodically report the status of the simulation execution to Simulation Configurator with heartbeats. Simulation Configurator can further modify the simulation environment on the run time by responding to the heartbeats.

Class Name: WebotsSimObject
Property Default Description
sync_type NO_HEART_BEAT NO_HEART_BEAT: Do not report simulation status.
WITHOUT_SYNC: Report the status and continue simulation.
WITH_SYNC: Wait for new commands after each heartbeat.
period_ms 10 Period of the simulation status reporting.
Table 10: Heartbeat Configuration.

An example heartbeat configuration.

⬇
1from Sim_ATAV.simulation_control.heart_beat import HeartBeatConfig
2
3# Create a heartbeat configuration that will make Simulation Supervisor report
4# simulation status at every 2s and continue execution without waiting for a new
5# command.
6sim_environment.heart_beat_config = \
7  HeartBeatConfig(sync_type=HeartBeatConfig.WITHOUT_SYNC, period_ms=2000)
Listing 10: Source code for creating a heartbeat configuration.

3.2.3 Data Log Item

Data log items are the states that will be collected into the simulation trace by Simulation Supervisor.

Class Name: ItemDescription
Property Default Description
item_type None Type of the corresponding simulation entity.
TIME, VEHICLE, PEDESTRIAN
item_index None Index of the corresponding simulation entity.
item_state_index None Index of the state that will be recorded.
Table 11: Data Item Description.

An example list of data log items for simulation trajectory generation.

⬇
1 # —– Data Log Configurations:
2 # First entry in the simulation trace will be the simulation time:
3sim_environment.data_log_description_list.append(
4  ItemDescription(item_type=ItemDescription.ITEM_TYPE_TIME,
5    item_index=0, item_state_index=0))
6
7# For each vehicle in Ego and Agent vehicles list,
8# record x,y positions, orientation and speed:
9for vhc_ind in range(len(sim_environment.ego_vehicles_list) + len(sim_environment.agent_vehicles_list)):
10  sim_environment.data_log_description_list.append(
11    ItemDescription(item_type=ItemDescription.ITEM_TYPE_VEHICLE,
12      item_index=vhc_ind,
13      item_state_index=WebotsVehicle.STATE_ID_POSITION_X))
14  sim_environment.data_log_description_list.append(
15    ItemDescription(item_type=ItemDescription.ITEM_TYPE_VEHICLE,
16      item_index=vhc_ind,
17      item_state_index=WebotsVehicle.STATE_ID_POSITION_Y))
18  sim_environment.data_log_description_list.append(
19    ItemDescription(item_type=ItemDescription.ITEM_TYPE_VEHICLE,
20      item_index=vhc_ind,
21      item_state_index=WebotsVehicle.STATE_ID_ORIENTATION))
22  sim_environment.data_log_description_list.append(
23    ItemDescription(item_type=ItemDescription.ITEM_TYPE_VEHICLE,
24      item_index=vhc_ind,
25      item_state_index=WebotsVehicle.STATE_ID_SPEED))
26
27# For each pedestrian, record x and y positions:
28for ped_ind in range(len(sim_environment.pedestrians_list)):
29  sim_environment.data_log_description_list.append(
30    ItemDescription(item_type=ItemDescription.ITEM_TYPE_PEDESTRIAN,
31      item_index=ped_ind,
32      item_state_index=WebotsVehicle.STATE_ID_POSITION_X))
33  sim_environment.data_log_description_list.append(
34    ItemDescription(item_type=ItemDescription.ITEM_TYPE_PEDESTRIAN,
35      item_index=ped_ind,
36      item_state_index=WebotsVehicle.STATE_ID_POSITION_Y))
37
38# Set the period of data log collection from the simulation.
39sim_environment.data_log_period_ms = 10
40
41# Create Trajectory dictionary for later reference.
42# Dictionary will be used to relate received simulation trace to object states.
43sim_environment.populate_simulation_trace_dict()
Listing 11: Source code for creating data log items.

3.2.4 Initial State Configuration

Initial state configuration objects are for setting an initial state ofa simulation entity.

Class Name: InitialStateConfig
Property Default Description
item None Data item for the corresponding state as an ItemDescription object.
value None Initial value of the corresponding state.
Table 12: Initial State Configuration.

An example initial state configuration.

⬇
1from Sim_ATAV.simulation_control.initial_state_config import InitialStateConfig
2from Sim_ATAV.simulation_control.item_description import ItemDescription
3
4ego_init_speed_m_s = 10.0  # Keeping the Ego initial speed in a variable
5
6# Create and add an initial state configuration into simulation environment object.
7sim_environment.initial_state_config_list.append(
8  InitialStateConfig(item=ItemDescription(
9    item_type=ItemDescription.ITEM_TYPE_VEHICLE,  # State of a vehicle
10    item_index=0,  # Vehicle index 0 (first added vehicle)
11    item_state_index=WebotsVehicle.STATE_ID_VELOCITY_X),  # Speed along x-axis
12    value=ego_init_speed_m_s))
Listing 12: Source code for setting an initial state value.

3.2.5 Viewpoint Configuration

Webots viewpoint can automatically follow a simulation entity throughout the simulation. Viewpoint configuration is used to describe which object to follow if desired.

Class Name: ViewFollowConfig
Property Default Description
item_type None Type of the corresponding simulation entity.
TIME, VEHICLE, PEDESTRIAN
item_index None Index of the corresponding simulation entity.
position None values of the initial position of the viewpoint.
rotation None Rotation of the viewpoint.
Table 13: Viewpoint Configuration.

An example viewpoint configuration.

⬇
1from Sim_ATAV.simulation_configurator.view_follow_config import ViewFollowConfig
2
3# Create a viewpoint configuration to follow Ego vehicle (vehicle indexed as 0 as it
4# is the first added vehicle). Viewpoint will be positioned 15m behind and 3m above
5# the vehicle.
6sim_environment.view_follow_config = \
7  ViewFollowConfig(item_type=ItemDescription.ITEM_TYPE_VEHICLE,
8    item_index=0,
9    position=[sim_environment.ego_vehicles_list[0].current_position[0] - 15.0,
10    sim_environment.ego_vehicles_list[0].current_position[1] + 3.0,
11    sim_environment.ego_vehicles_list[0].current_position[2]],
12    rotation=[0.0, 1.0, 0.0, \
13      -sim_environment.ego_vehicles_list[0].current_orientation])
Listing 13: Source code for creating a viewpoint configuration.

3.2.6 Simulation Configuration

Simulation configuration provides the information necessary to execute a simulation through TCP/IP connection between Simulation Configurator and Simulation Supervisor.

Class Name: SimulationConfig
Property Default Description
world_file “../Webots_Projects/ Name of the base Webots
worlds/test_world_1.wbt” world file.
server_port 10021 Port number for connecting to Simulation Supervisor.
server_ip “127.0.0.1” Port number for connecting to Simulation Supervisor.
sim_duration_ms 50000 Simulation duration (ms).
sim_step_size 10 Simulation time step (ms).
run_config_arr [] List of run configurations for supporting multiple parallel simulation executions (RunConfig objects).
Class Name: RunConfig
Property Default Description
simulation_run_mode FAST_NO_GRAPHICS Webots run mode (simulation speed).
REAL_TIME:
Real-time speed,
FAST_RUN:
As fast as possible with visualization,
FAST_NO_GRAPHICS:
As fast as possible without visualization.
Table 14: Simulation Configuration.

An example simulation configuration creation.

⬇
1from Sim_ATAV.simulation_configurator import sim_config_tools
2
3sim_config = sim_config_tools.SimulationConfig(1)
4sim_config.run_config_arr.append(sim_config_tools.RunConfig())
5sim_config.run_config_arr[0].simulation_run_mode = SimData.SIM_TYPE_REAL_TIME
6sim_config.sim_duration_ms = 15000  # 15s simulation
7sim_config.sim_step_size = 10
8sim_config.world_file = ’../Webots_Projects/worlds/empty_world.wbt’
Listing 14: Source code for creating a simulation configuration.

3.3 Executing the Simulation

To start execution of a scenario, it is advised to first start Webots with the base world file manually. If Webots crashes or communication is lost, can restart Webots when necessary (only is Webots is executing in the same system). To start the simulation, Simulation Configurator should be used to connect to the Simulation Supervisor, send the simulation environment and configuration details, start the simulation, and finally collect the simulation trace. Below is an example source code for this step.

An example for execution of the simulation.

⬇
1from Sim_ATAV.simulation_configurator.sim_environment_configurator import SimEnvironmentConfigurator
2
3# Create a Simulation Configurator with the previously defined sim. configuration.
4sim_env_configurator = SimEnvironmentConfigurator(sim_config=sim_config)
5
6# Connect to the Simulation Supervisor.
7# Try maximum of 3 (max_connection_retry) times to connect.
8(is_connected, simulator_instance) = sim_env_configurator.connect(max_connection_retry=3)
9if not is_connected:
10    raise ValueError(’Could not connect!’)
11
12# Setup the scenario with previously populated Simulation Environment object.
13sim_env_configurator.setup_sim_environment(sim_environment)
14
15# Execute the simulation and get the simulation trace.
16trajectory = sim_env_configurator.run_simulation_get_trace()
Listing 15: Source code for executing the scenario.

4 Combinatorial Testing

provides basic functionality to read test scenarios from csv files, in which, each row represents a test case and each column holds the values of a test parameter. This feature can be used to automatically execute a set of predefined test cases. The user can create csv files that contain a set of test cases either manually or by using a tool. 15 provides a list of methods provided in covering_array_utilities.py.

load_experiment_data(file_name, header_line_count=6, index_col=None)
Description: Loads test cases from a csv file.
Arguments: file_name: Name of the csv file
header_line_count: Number of header lines to skip. (default: 6)
index_col: Index of the column that is used for data indexing. (default: None)
Return: A pandas data frame (a tabular data structure) containing all test cases.
get_experiment_all_fields(exp_data_frame, exp_ind)
Description: Returns all parameters for a test case (one row of the test table).
Arguments: exp_data_frame: Data frame that was read using
load_experiment_data
exp_ind: Index of the experiment
Return: A row from the test table that contains the requested test case.
get_field_value_for_current_experiment(cur_experiment, field_name)
Description: Returns the value of the requested test parameter.
Arguments: cur_experiment: Current test case that was read using
get_experiment_all_fields.
field_name: Name of the test parameter.
Return: Value of the requested parameter for the given test case.
Table 15: A list of Sim-ATAV methods for loading test cases from csv files.

For combinatorial testing, ACTS from NIST [4] can be used as a tool for generating covering arrays. Providing a complete guide on covering arrays and how to use ACTS is out of the scope of this chapter. The user of ACTS creates a system definition by defining the system parameters and providing the possible values for each parameter. The system definition is saved as an xml file. ACTS can generate a covering array of desired strength and export the outputs in a csv file. An example of a system definition and a corresponding 2-way covering array are provided in distribution in the files tests/examples/TutorialExampleSystemACTS.xml and
tests/examples/TutorialExample_CA_2way.csv, respectively. Below is an example of reading test cases from a csv file and running each test one by one. Note that, in the example, simulation trajectories are not evaluated. Outputs of each test case should be evaluated to decide the test result.

An example for running covering array test cases.

In this example, a 2-way covering array for the test parameters is generated in ACTS. 16 shows the original test parameter name from the file tutorial_example_1.py, the name used in ACTS, and the possible values for each parameter (corresponding ACTS file is TutorialExampleSystemACTS.xml).

Parameter Name in ACTS Possible Values
ego_init_speed_m_s ego_init_speed 0, 5, 10, 15
ego_x_pos ego_x_position 15, 20, 25
pedestrian_speed pedestrian_speed 2,3,4,5
Table 16: Describing test parameters for creating a covering array.

After entering the parameter descriptions in 16 to ACTS, a 2-way covering array is generated and exported as a csv file. Listing 16 shows the contents of the output csv file. The file is available as: tests/examples/TutorialExample_CA_2way.csv.

⬇
# ACTS Test Suite Generation: Mon Jan 14 22:46:33 MST 2019
#  ’*’ represents don’t care value
# Degree of interaction coverage: 2
# Number of parameters: 3
# Maximum number of values per parameter: 4
# Number of configurations: 16
ego_init_speed,ego_x_position,pedestrian_speed
0,20,2
0,25,3
0,15,4
0,20,5
5,25,2
5,15,3
5,20,4
5,25,5
10,15,2
10,20,3
10,25,4
10,15,5
15,20,2
15,25,3
15,15,4
15,15,5
Listing 16: CSV file containing a set of test cases generated by ACTS.

Below is the source code that is reading the test cases from the csv file using the functions listed in 15 and running each test case one by one.

⬇
1import time
2from Sim_ATAV.simulation_configurator import covering_array_utilities
3
4def run_test(ego_init_speed_m_s=10.0, ego_x_pos=20.0, pedestrian_speed=3.0):
5    ”””Runs a test with the given arguments”””
6    .
7    .
8    .
9    # This function is creating the test scenario, executing it and returning trajectory as described in previous section. Content is not repeated here for space considerations.
10    .
11    .
12    .
13    return trajectory
14
15def run_covering_array_tests():
16    ”””Runs all tests from the covering array csv file”””
17    exp_file_name = ’TutorialExample_CA_2way.csv’   # csv file containing the tests
18
19    # Read all experiment into a table:
20    exp_data_frame = covering_array_utilities.load_experiment_data(exp_file_name, header_line_count=6)
21
22    # Decide number of experiments based on the number of entries in the table.
23    num_of_experiments = len(exp_data_frame.index)
24
25    trajectories_dict = {}  # A dictionary data structure to keep simulation traces.
26
27    for exp_ind in range(num_of_experiments):  # For each test case
28        # Read the current test case
29        current_experiment = covering_array_utilities.get_experiment_all_fields(
30          exp_data_frame, exp_ind)
31
32        # Read the parameters from the current test case:
33        ego_init_speed = float(
34          covering_array_utilities.get_field_value_for_current_experiment(
35            current_experiment, ’ego_init_speed’))
36        ego_x_position = float(
37          covering_array_utilities.get_field_value_for_current_experiment(
38            current_experiment, ’ego_x_position’))
39        pedestrian_speed = float(
40          covering_array_utilities.get_field_value_for_current_experiment(
41            current_experiment, ’pedestrian_speed’))
42
43        # Execute the test case and record the resulting simulation trace:
44        trajectories_dict[exp_ind] = run_test(ego_init_speed_m_s=ego_init_speed,
45          ego_x_pos=ego_x_position, pedestrian_speed=pedestrian_speed)
46        time.delay(2)  # Give Webots some time to reload the world
47    return trajectories_dict
Listing 17: Source code for executing the covering array test cases.

5 Falsification / Search-based Testing

For performing falsification, a CPS falsification tool like [3] can be used. As is in , we need to call the test cases which are developed in Python from . In this section, first a simple approach is described to call the test cases from , then a simple setup is described to perform falsification. This chapter does not provide a complete guide to . Reader is referred to website for further details 666iwebsite: https://sites.google.com/a/asu.edu/s-taliro/s-taliro.

5.1 Connecting to

has built-in support to call Python functions. However, type conversions are required both in and in the code 777The user is free to use a different approach than what is described here.. provides basic functionality to convert simulation trace to a format (single dimensional list) that can be easily interpreted in . Listing 18 shows the necessary updates on the Python side. First, sim_duration argument is added to run_test function allow run different length of simulations. Next, for_matlab argument is added to tell the function is called from so that experiment_tools.npArray2Matlab method from is used to convert simulation trace to a single list for .

⬇
1def run_test(ego_init_speed_m_s=10.0, ego_x_pos=20.0, pedestrian_speed=3.0, sim_duration=15000, for_matlab=False):
2    ”””Runs a test with the given arguments”””
3    …
4    # This function is creating the test scenario, executing it and returning trajectory as described in previous section. Content is not repeated here for space considerations.
5    …
6    if for_matlab:
7        trajectory = experiment_tools.npArray2Matlab(trajectory)
8    return trajectory
Listing 18: Source code for executing the covering array test cases.

On the side, we need a wrapper function to call the function run_test. Listing 19 provides an example to such a wrapper function. This code can be found in distribution in the file /tests/examples/run_tutorial_example_from_matlab.m. Unused return parameters YT, LT, CLG, GRD and input arguments steptime, InpSignal are to maintain compatibility with function call format.

⬇
1function [T, XT, YT, LT, CLG, GRD] = run_tutorial_example_from_matlab(XPoint, sim_duration_s, steptime, InpSignal)
2%run_tutorial_example_from_matlab Run Webots simulation with the parameters in XPoint.
3% XPoint contains: [ego_init_speed_m_s, ego_x_pos, pedestrian_speed]
4
5% Run the simulation and receive the trajectory:
6traj = py.tutorial_example_1.run_test(XPoint(1), XPoint(2), XPoint(3), int32(sim_duration_s*1000.0), true);
7
8% Convert trajectory to matlab array
9mattraj = Core_py2matlab(traj);  % Core_py2matlab is from Matlab fileexchange, developed by Kyle Wayne Karhohs
10YT = [];
11LT = [];
12CLG = [];
13GRD = [];
14if isempty(mattraj)
15  T = [];
16  XT = [];
17else
18  % Separate time from the simulation trace:
19  T = mattraj(:,1)/1000.0;  % Also, convert time to s from ms
20  XT = mattraj(:, 2:end);  % Rest of the trace
21end
22end
Listing 19: Matlab code as a wrapper to execute Python test execution function.

5.2 Connecting to

To perform falsification with , an MTL specification should be defined, ranges for test parameters and the code, which is given in Listing 19, for running a simulation should be provided to as the model under test. Listing 20 gives a very simple example for falsification with an MTL requirement that checks x and y coordinates of Ego and agent vehicles to decide a collision. This code can be found in distribution in /tests/examples/run_falsification.m. The requirement is not to collide onto the agent vehicle. Note that, this requirement is very simplified to provide a simple example source code. A more complete check for collision would require incorporating further vehicle details and/or extracting collision information from the simulation.

⬇
1% Indices of states in simulation trace:
2cur_traj_ind = 1;
3EGO_X = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
4EGO_Y = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
5EGO_THETA = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
6EGO_V = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
7AGENT_X = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
8AGENT_Y = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
9AGENT_THETA = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
10AGENT_V = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
11PED_X = cur_traj_ind; cur_traj_ind = cur_traj_ind + 1;
12PED_Y = cur_traj_ind;
13NUM_ITEMS_IN_TRAJ = cur_traj_ind;
14
15% Predicates for MTL requirement:
16ii = 1;
17preds(ii).str=’y_check1’;
18preds(ii).A = zeros(1, NUM_ITEMS_IN_TRAJ);
19preds(ii).A(AGENT_Y) = 1;
20preds(ii).A(EGO_Y) = -1;
21preds(ii).b = 1.5;
22
23ii = ii+1;
24preds(ii).str=’y_check2’;
25preds(ii).A = zeros(1, NUM_ITEMS_IN_TRAJ);
26preds(ii).A(AGENT_Y) = -1;
27preds(ii).A(EGO_Y) = 1;
28preds(ii).b = 1.5;
29
30ii = ii+1;
31preds(ii).str=’x_check1’;
32preds(ii).A = zeros(1, NUM_ITEMS_IN_TRAJ);
33preds(ii).A(AGENT_X) = 1;
34preds(ii).A(EGO_X) = -1;
35preds(ii).b = 8;
36
37ii = ii+1;
38preds(ii).str=’x_check2’;
39preds(ii).A = zeros(1, NUM_ITEMS_IN_TRAJ);
40preds(ii).A(AGENT_X) = -1;
41preds(ii).A(EGO_X) = 1;
42preds(ii).b = 0;
43
44% Metric Temporal Logic Requirement:
45phi = ’[](!(y_check1 /\ y_check2 /\ x_check1 /\ x_check2))’;
46
47% Ranges for test parameters (ego_init_speed_m_s, ego_x_pos, pedestrian_speed):
48init_cond = [0.0, 15.0;
49            15.0, 25.0;
50             2.0, 5.0];
51
52% Provide our Matlab wrapper function for running the tests as the model.
53model = @run_tutorial_example_from_matlab;
54opt = staliro_options();
55opt.runs = 1;  % Do falsification only once.
56opt.black_box = 1;  % Because we use a custom Matlab function as the model.
57opt.SampTime = 0.010;  % Sample time. Same as Webots world time step.
58opt.spec_space = ’X’;  % Requirements are defined on state space.
59opt.optimization_solver = ’SA_Taliro’;  % Use Simulated Annealing
60opt.taliro = ’dp_taliro’;  % Use dp_taliro to compute robustness
61opt.map2line = 0;
62opt.falsification = 1;  % Stop when falsified
63opt.optim_params.n_tests = 100;  % maximum number of tries
64sim_duration = 15.0;
65
66disp([’Running S-TaLiRo … ’])
67[results, history] = staliro(model, init_cond, [], [], phi, preds, sim_duration, opt);
68
69res_filename = [’results_’, datestr(datetime(”now”), ’yyyy_mm_dd__HH_MM’), ’.mat’];
70save(res_filename)
71disp([”Results are saved to: ”, res_filename])
Listing 20: Matlab code for running falsification with S-TaLiRo.

6 Other Remarks

A user guide for is provided with a running example. The source code for the running example used in this chapter is available in distribution under tests/examples folder. As is a research tool that is not directly targeted for production-level systems special caution should be taken before using it for testing any critical functionality. is still evolving and it may contain a number of bugs or parts that are open to optimization. Here, only major functionality provided by is discussed. It provides further functionality like a number of sample vehicle controller subsystems, additional computations on simulation trajectory such as collision detection, and functionality toward evaluating perception-system performance. For a deeper understanding of ’s capabilities, the reader is encouraged to go through the source code for examples and experiments that are used as case studies for publications.

References

  • [1]
  • [2] Cyberbotics: Webots User Manual and Reference Manual. https://www.cyberbotics.com/support. Accessed: 2019-02-28.
  • [3] Georgios Fainekos, Sriram Sankaranarayanan, Koichi Ueda & Hakan Yazarel (2012): Verification of Automotive Control Applications using S-TaLiRo. In: Proceedings of the American Control Conference.
  • [4] D Richard Kuhn, Raghu N Kacker & Yu Lei (2013): Introduction to combinatorial testing. CRC press.
  • [5] Cumhur Erkan Tuncali (2019): Search-based Test Generation for Automated Driving Systems: From Perception to Control Logic. Ph.D. thesis, Arizona State University.
  • [6] Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito & James Kapinski (2018): Sim-ATAV: Open-Source Simulation-based Adversarial Test Generation Framework for Autonomous Vehicles. Available at https://cpslab.assembla.com/spaces/sim-atav/git/source.
  • [7] Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito & James Kapinski (2018): Sim-ATAV: Simulation-Based Adversarial Testing Framework for Autonomous Vehicles. In: Proceedings of the 21st International Conference on Hybrid Systems: Computation and Control (part of CPS Week), ACM, pp. 283–284.
  • [8] Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito & James Kapinski (2018):

    Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components

    .     In: IEEE Intelligent Vehicles Symposium (IV).

Sign up for DeepAI

Join one of the world's largest A.I. communities

Already have an account? Login here

Login to DeepAI

Don't have an account? Signup here

SHARE