share
1 An Overview of
[6, 7, 8] is a tool developed for experimenting with different test generation techniques in simulation environments for research purposes as described in [8]. It is mainly developed in Python and it uses the open-source robotics toolbox Webots [2] for 3D scene generation, vehicle and sensor modeling and simulation. can be interfaced with covering array generation tools like ACTS [4] and with falsification tools like [3], which is a toolbox. is publicly available as an open-source project [6].
1 provides a high-level overview of the framework. The main functionality of is provided by Simulation configurator and Simulation Supervisor blocks. Simulation configurator block represents the API for the user script to create a simulation and to receive the results. Simulation Supervisor block represents the part of the framework that executes inside the robotics simulation toolbox Webots. It uses the Webots API to (1) modify the simulation environment, e.g., add/configure vehicles, roads, pedestrians, provide vehicle controllers, (2) execute the simulation, and (3) collect information, e.g., the evolution of vehicle states over the simulation time. Simulation Supervisor receives the requested simulation configuration from the Simulation configurator over socket communication. When the simulation environment is set up, Simulation Supervisor requests Webots to start the simulation. User-provided controllers control the motion of the vehicles and pedestrians. At the end of the simulation, Simulation Supervisor sends the collected simulation trace to Simulation configurator.
External tools like ACTS can be used to generate combinatorial tests with covering arrays. provides functions to read covering array test scenarios from csv files. For falsification, is used to iteratively sample new configurations until a failure case is detected. Optionally, initial samples for the configurations can be read by from the covering array csv files. The sampled configurations in are passed as parameters to test generation functions using the Python interface provided in , by calling Python functions directly inside . The test generation functions return the simulation trace back to (and to ) as return values. More detail on falsification methods is available in [8, 3]. A running example of test generation is provided in the upcoming sections.
Although they are not essential for the test generation purposes, also comes with some vehicle controller implementations as well as some basic perception system, sensor fusion, path planning and control algorithms that can be utilized by the user for Ego or Agent vehicle control.
2 Installation Instructions
requires Python 3.7 and Webots for basic functionality. For some controllers and test generation approaches, there are other requirements like , , TensorFlow, etc. The framework has been tested in Windows
® 10 with specific versions of the required packages.Firstly, should be either downloaded or cloned using a git client 111: https://cpslab.assembla.com/spaces/sim-atav. For Windows®, the preferred installation approach is to use setup_for_windows.bat. Once executed, it will guide the user through the installation process and automate the process as much as possible. This script is not advanced and may fail for some systems. If the script fails, please try the steps below for a manual installation.
Below are the steps for the manual installation. All the paths are given relative to the root folder for the distribution. In case any problems are experienced during the installation of the packages, most of the packages can also be found in Christoph Gohlke’s website222Christoph Gohlke’s website: https://www.lfd.uci.edu/ gohlke/pythonlibs/:
-
Install Python 3.7-64 Bit
-
Install Webots r2019a.
-
(optional) If the system has CUDA-enabled GPU and it will be utilized for an increased performance:
-
Install CUDA Toolkit 10.0
-
Install CUDNN 7.3.1
-
-
Download Python_Dependencies from http://www.public.asu.edu/ etuncali/downloads/ and unzip it next to this installation script. The Python wheel (.whl) files should be directly under ./Python_Dependencies/
-
Install commonly used Python packages:
-
Install Numpy+MKL 1.14.6 either from Python_Dependencies folder or from Christoph Gohlke’s website:
pip3 install –upgrade Python_Dependencies/numpy-1.14.6+mkl-cp37-cp37m-win_amd64.whl -
Install scipy 1.2.0:
pip3 install scipy==1.2. -
Install scikit-learn:
pip3 install scikit-learn -
Install pandas:
pip3 install pandas -
Install Absl Py:
pip3 install absl-py -
Install matplotlib:
pip3 install matplotlib -
Install pykalman:
pip3 install pykalman -
Install dubins:
pip3 install dubins-
If any problems are experienced when installing pydubins:
One option is to go into the folder Python_Dependencies/pydubins and execute python setup.py install. Another option is the following:
-
Download pydubins from github.com/AndrewWalker/pydubins.
-
Do the following changes in dubins/src/dubins.c:
-
Call python setup.py install inside pydubins folder.
-
-
-
-
For controllers with DNN (Deep Neural Network) object detection:
! Currently, Python 3.7 support for Tensorflow is provided by a 3rd party (only for Windows). Installation wheels are provided under Python_Dependencies folder.
Check if the system CPU supports AVX2 (for increased performance) 333A list of CPUs with AVX2 is available at:
https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2.-
If the system GPU has CUDA cores, CUDNN is installed and the system CPU supports AVX 2: Install Tensorflow-gpu with AVX2 support.
pip3 install –upgrade Python_Dependencies/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl -
If the system GPU has CUDA cores, CUDNN is installed and the system CPU does NOT support AVX 2: Install Tensorflow-gpu without AVX2 support.
pip3 install –upgrade Python_Dependencies/sse2/tensorflow_gpu-1.12.0-cp37-cp37m-win_amd64.whl -
If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU supports AVX 2: Install Tensorflow with AVX2 support.
pip3 install –user –upgrade Python_Dependencies/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl -
If the system GPU does not have CUDA cores or CUDNN is not installed, and the system CPU does NOT support AVX 2: Install Tensorflow without AVX2 support.
pip3 install –user –upgrade Python_Dependencies/sse2/tensorflow-1.12.0-cp37-cp37m-win_amd64.whl
-
-
Install Python Dependencies of SqueezeDet (if the existing controllers that use will be used). There is no need to install separately, as it is provided in the framework.
-
Install joblib:
pip3 install –upgrade joblib -
Install opencv:
pip3 install –upgrade opencv-contrib-python -
Install pillow:
pip3 install –upgrade Pillow -
Install easydict:
pip3 install –upgrade easydict==1.7! If any problems are experienced while installing easydict, try the following:
-
-
To design Covering Array Tests: Please request a copy and install ACTS from NIST444ACTS tool can be requested from
https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software/downloadable-tools. -
To do robustness-guided falsification, and are needed:
-
Install from Mathworks(tested with r2017b).
-
Install 555is available at https://sites.google.com/a/asu.edu/s-taliro/s-taliro.
-
-
After the installation is finished, the Python package wheel files that are under the folder named Python_Dependencies can be deleted to save some disk space.
Setting to Utilize GPU:
If the system has a CUDA-enabled GPU, and CUDA Toolkit, CUDNN are installed, the variable has_gpu should be set to True in the following file to make the experiments use the system GPU for :
Sim_ATAV/classifier/classifier_interface/gpu_check.py.
3 Reference Manual
3.1 Simulation Entities
starts building a testing scenario from an existing Webots world file provided by the user, which we will call as base world file. The user has the option to have all or some of the simulation entities, roads, vehicles, etc., saved in the base world file before the test generation time. The user can use the functionality provided by to add more simulation entities to the world at the time of test generation. This functionality is especially useful when the search space for the tests contain some parameters of the simulation entities such as road width, number of lanes, positions of the vehicles.
This section describes the most commonly used simulation entities that can be programmatically added into the simulation world at the time of test generation. Note that may not provide the functionality to add all possible types of simulation objects that are supported by Webots. In this section, the class names and properties with their default values for the simulation entities supported by are provided. Test Generation Script which is developed by the user typically creates instances of required simulation entities and uses the provided functions to add those entities into the test scenario. For modifying the simulation environment beyond the capabilities of , the user can do the changes manually and save in the base world file or modify the source code of to add or change the capabilities as needed. For a deeper understanding of the possibilities, the reader is advised to get familiar with the Webots simulation environment and details of available simulation entities [2].
The class properties for the simulation entities are provided in the tables below. The first row of each table gives the class name. Other rows start with the property name, gives the default value of the property (the value used if not explicitly changed by the user), and a brief description of the property. For each class, a related source code snippet from a running example is provided. The original source code for the running example can be found as tests/tutorial_example_1.py in the distribution.
2 is an image from the scenario described in the running example. A yellow Ego vehicle is behind a pedestrian walking in the middle of a 3-lane road, and an agent vehicle is approaching from the opposite direction in the next lane. There is a bumpy road surface for a short distance, and a stop sign placed on the right side of the road. This is only a simple example to illustrate how to use .
General information:
In Webots, all simulation entities are kept in a simulation tree and they can be easily accessed using their DEF_NAME property. Position fields are 3D arrays as , keeping the value of the position at each axis in Webots coordinate system, where is typically the vertical axis to the ground (in practice, this depends on the Webots world file provided by the user). Rotation fields are 4D arrays as , where
represents a rotation vector and
represents the rotation around this vector in clockwise direction.3.1.1 Simulation Environment
Once an object is created for a simulation entity, the user can utilize the corresponding function provided by to add the entity to the scenario. An easier alternative is to utilize the SimEnvironment class provided by . An object of this class can be populated with the necessary simulation entities and passed to the simulation environment with a single function call. 1 summarizes the SimEnvironment class.
| Class Name: SimEnvironment | ||
|---|---|---|
| Property | Default | Description |
| fog | None | Keeps fog object. |
| heart_beat_config | None | The heartbeat configuration. |
| view_follow_config | None | The viewpoint configuration. |
| ego_vehicles_list | [] | The list of Ego vehicles. |
| agent_vehicles_list | [] | The list of agent vehicles. |
| pedestrians_list | [] | The list of pedestrians. |
| road_list | [] | The list of roads. |
| road_disturbances_list | [] | The list of road disturbances. |
| generic_sim_objects_list | [] | The list of generic simulation objects. |
| control_params_list | [] | The list of controller parameters that will be set in the run time. |
| initial_state_config_list | [] | The list of initial state configurations. |
| data_log_description_list | [] | The list of data log descriptions. |
| data_log_period_ms | None | Data log period (ms). |
For the running example, we start with an empty simulation environment. Listing 1 creates an empty SimEnvironment object that will later keep the required simulation entities.
3.1.2 Road
A user-configurable road structure to use in the simulation environment is given in 2.
| Class Name: WebotsRoad | ||
|---|---|---|
| Property | Default | Description |
| def_name | “STRROAD” | Name as it appears in the simulation tree. |
| road_type | “StraightRoadSegment” | Road type name as used by Webots. |
| rotation | [0, 1, 0, math.pi/2] | Rotation of the road |
| position | [0, 0.02, 0] | Starting position. |
| number_of_lanes | 2 | Number of lanes. |
| width | number_of_lanes * 3.5 | Road width (m). |
| length | 1000 | Road length (m). |
Listing 2 provides a source code snippet that creates a 3-lane straight road segment lying between 1000m and -1000m along the x-axis.
3.1.3 Vehicle
A user-configurable vehicle class to use in the simulation environment in 3.
| Class Name: WebotsVehicle | ||
|---|---|---|
| Property | Default | Description |
| def_name | “” | Name as it appears in Webots simulation tree. |
| vhc_id | 0 | Integer ID for referencing to the vehicle. |
| vehicle_model | “AckermannVehicle” | Vehicle model can be any model name available in Webots. |
| rotation | [0, 1, 0, 0] | Rotation of the object. |
| current_position | [0, 0.3, 0] | x,y,z values of the position. |
| color | [1, 1, 1] | R,G,B values of the color in the range [0,1]. |
| controller | “void” | Name of the vehicle controller. |
| is_controller_name_absolute | False | Indicates where to find the vehicle controller. Find the details below. |
| vehicle_parameters | [] | Additional parameters for the vehicle object. |
| controller_parameters | [] | Parameters that will be sent to the vehicle controller. |
| controller_arguments | True | Arguments passed to the vehicle controller executable. |
| sensor_array | [] | List of sensors on the vehicle (WebotsSensor objects). |
The vehicle_model field of a vehicle object should match with the models available in Webots (or any custom models added to Webots by the user). Webots r2019a version provides vehicle model options ToyotaPrius, CitroenCZero, BmwX5, RangeRoverSportSVR, LincolnMKZ, TeslaModel3 as well as truck, motorcycle and tractor models. When is_controller_name_absolute is set to true, Webots will load the given controller from Webots_Projects/controllers folder, otherwise, it will load the controller named vehicle_controller which is located under the same folder but will take the controller name as an argument.
We can now create vehicles and place them on the road that was created above. Listing 3 provides an example source code snippet that creates an Ego vehicle at the position , , and an agent vehicle at the position , , vehicles facing toward each other. The controllers for the vehicles are set and the controller arguments are provided. The arguments accepted are controller-specific. The vehicle controllers used in this example can be found under the folder Webots_Projects/controllers.
3.1.4 Sensor
4 provides an overview of the WebotsSensor class that can be used to describe a sensor. Webots vehicle models have specific sensor slots on the vehicles. These are typically TOP, CENTER, FRONT, RIGHT, LEFT. The property sensor_type accepts the type of the sensor which should match the type used by Webots. The translation field of the sensor can be used to place the sensor to a different position relative to its corresponding sensor slot. As sensors can vary a lot in terms or parameters, sensor_fields property is provided to accept names and values of the desired parameters as a list of WebotsSensorField objects for flexible configuration of sensors.
| Class Name: WebotsSensor | ||
| Property | Default | Description |
| sensor_type | “” | Type of the sensor as defined in Webots. |
| sensor_location | FRONT | Sensor slot enumeration. |
| FRONT, CENTER, LEFT, RIGHT, TOP | ||
| sensor_fields | [] | List of WebotsSensorField objects to customize the sensor. |
| Class Name: WebotsSensorField | ||
| field_name | “” | Name of the field that will be set. |
| field_val | “” | Value of the field. |
An example source code of adding sensors to vehicles is provided in Listing 4. In this example, we add a compass and a GPS device that are used by the controllers for path following. The receiver device added to the vehicles is used by the controllers to receive new commands from Simulation Supervisor. We add the receivers because we will later need them to update target paths of the vehicles. Note that although the necessary infrastructure for this approach is provided by , it is implementation specific and not mandated. In this example, we also add a radar device to Ego vehicle for collision avoidance.
3.1.5 Pedestrian
The user-configurable pedestrian class, WebotsPedestrian, to use in the simulation environment is described in 5. Target speed and target path (trajectory) of the pedestrian are automatically passed as arguments to the given controller.
| Class Name: WebotsPedestrian | ||
|---|---|---|
| Property | Default | Description |
| def_name | “PEDESTRIAN” | Name as it appears in Webots simulation tree. |
| ped_id | 0 | Integer ID for referencing to the pedestrian. |
| rotation | [0, 1, 0, math.pi/2.0] | Rotation of the object. |
| current_position | [0, 0, 0] | x,y,z values of the position. |
| shirt_color | [0.25, 0.55, 0.2] | R,G,B values of the shirt color in the range [0, 1]. |
| pants_color | [0.24, 0.25, 0.5] | R,G,B values of the pants color in the range [0, 1]. |
| shoes_color | [0.28, 0.15, 0.06] | R,G,B values of the shoes color in the range [0, 1]. |
| controller | “void” | Name of the pedestrian controller. |
| target_speed | 0.0 | Walking speed of the pedestrian. |
| trajectory | [] | Walking path of the pedestrian. |
Listing 5 provides an example code snippet to create a pedestrian object, and provide a target speed and a path to define the motion of the pedestrian.
3.1.6 Fog
User-configurable fog class to use in the simulation environment is summarized in 6.
| Class Name: WebotsFog | ||
|---|---|---|
| Property | Default | Description |
| def_name | “FOG” | Name as it appears in Webots simulation tree. |
| fog_type | “LINEAR” | Defines the type of the fog gradient. |
| color | [0.93, 0.96, 1.0] | R,G,B values of the fog color in the range [0, 1]. |
| visibility_range | 1000 | Visibility range of the fog (m). |
No camera is involved in this scenario, hence fog will not impact the performance of the controller. However, an example source code snippet is provided in Listing 6 for reference.
3.1.7 Road Disturbance
Road disturbance objects are solid triangular objects placed on the road surface to emulate a bumpy road surface. WebotsRoadDisturbance, which is described in 7, contains the properties to describe how the solid objects will be placed to create a bumpy road surface.
| Class Name: WebotsRoadDisturbance | ||
|---|---|---|
| Property | Default | Description |
| disturbance_id | 1 | Object ID for later reference. |
| disturbance_type | INTERLEAVED | Enumerated type of the disturbance. |
| INTERLEAVED, | ||
| FULL_LANE_LENGTH, | ||
| ONLY_LEFT, ONLY_RIGHT | ||
| rotation | [0, 1, 0, 0] | Rotation of the object. |
| position | [0, 0, 0] | x,y,z values of the position. |
| length | 100 | Length of the bumpy road surface (m). |
| width | 3.5 | Width of the corresponding lane. |
| height | 0.06 | Height of the disturbance (m). |
| surface_height | 0.02 | Height of the corresponding road surface (m). |
| inter_object_spacing | 1.0 | Distance between repeating solid objects on the road (m). |
An example road disturbance object creation is given in Listing 7.
3.1.8 Generic Simulation Object
Generic simulation object is for adding any type of object into the simulation which is not covered above. For these objects, there are no checks performed or there are no limitations on the field values. The user can manually create any possible Webots object by setting all of its non-default field values.
| Class Name: WebotsSimObject | ||
|---|---|---|
| Property | Default | Description |
| def_name | “” | Name as it appears in Webots simulation tree. |
| object_name | “Tree” | Type name of the object. Must be same as the name used by Webots. |
| object_parameters | [] | List of (field name, field value) tuples as strings. Names must be same as the field names used by Webots. |
In Listing 8, although it is not expected to have an impact on the controller performance, a Stop Sign object is created as a generic simulation object example for reference.
3.2 Configuring the Simulation Execution
3.2.1 Additional Controller Parameters
Depending on the application and implementation details, controller parameters can be directly given to the controllers or they can be sent in the run-time. To emulate runtime inputs, such as human commands, can transmit controller commands over virtual radio communication. The controller should be able to read those commands from a radio receiver and a receiver object should be added to one of the sensor slots of the vehicles. This is an optional approach and the user is free to use other approaches such as reading data from a file, using socket communications etc.
| Class Name: WebotsControllerParameter | ||
|---|---|---|
| Property | Default | Description |
| vehicle_id | None | ID of the corresponding vehicle. |
| parameter_name | “” | Name of the parameter as string. |
| parameter_data | [] | Parameter data. |
An example controller parameter creation.
3.2.2 Heartbeat Configuration
Simulation Supervisor can periodically report the status of the simulation execution to Simulation Configurator with heartbeats. Simulation Configurator can further modify the simulation environment on the run time by responding to the heartbeats.
| Class Name: WebotsSimObject | ||
|---|---|---|
| Property | Default | Description |
| sync_type | NO_HEART_BEAT | NO_HEART_BEAT: Do not report simulation status. |
| WITHOUT_SYNC: Report the status and continue simulation. | ||
| WITH_SYNC: Wait for new commands after each heartbeat. | ||
| period_ms | 10 | Period of the simulation status reporting. |
An example heartbeat configuration.
3.2.3 Data Log Item
Data log items are the states that will be collected into the simulation trace by Simulation Supervisor.
| Class Name: ItemDescription | ||
|---|---|---|
| Property | Default | Description |
| item_type | None | Type of the corresponding simulation entity. |
| TIME, VEHICLE, PEDESTRIAN | ||
| item_index | None | Index of the corresponding simulation entity. |
| item_state_index | None | Index of the state that will be recorded. |
An example list of data log items for simulation trajectory generation.
3.2.4 Initial State Configuration
Initial state configuration objects are for setting an initial state ofa simulation entity.
| Class Name: InitialStateConfig | ||
|---|---|---|
| Property | Default | Description |
| item | None | Data item for the corresponding state as an ItemDescription object. |
| value | None | Initial value of the corresponding state. |
An example initial state configuration.
3.2.5 Viewpoint Configuration
Webots viewpoint can automatically follow a simulation entity throughout the simulation. Viewpoint configuration is used to describe which object to follow if desired.
| Class Name: ViewFollowConfig | ||
|---|---|---|
| Property | Default | Description |
| item_type | None | Type of the corresponding simulation entity. |
| TIME, VEHICLE, PEDESTRIAN | ||
| item_index | None | Index of the corresponding simulation entity. |
| position | None | values of the initial position of the viewpoint. |
| rotation | None | Rotation of the viewpoint. |
An example viewpoint configuration.
3.2.6 Simulation Configuration
Simulation configuration provides the information necessary to execute a simulation through TCP/IP connection between Simulation Configurator and Simulation Supervisor.
| Class Name: SimulationConfig | ||
|---|---|---|
| Property | Default | Description |
| world_file | “../Webots_Projects/ | Name of the base Webots |
| worlds/test_world_1.wbt” | world file. | |
| server_port | 10021 | Port number for connecting to Simulation Supervisor. |
| server_ip | “127.0.0.1” | Port number for connecting to Simulation Supervisor. |
| sim_duration_ms | 50000 | Simulation duration (ms). |
| sim_step_size | 10 | Simulation time step (ms). |
| run_config_arr | [] | List of run configurations for supporting multiple parallel simulation executions (RunConfig objects). |
| Class Name: RunConfig | ||
| Property | Default | Description |
| simulation_run_mode | FAST_NO_GRAPHICS | Webots run mode (simulation speed). |
| REAL_TIME: | ||
| Real-time speed, | ||
| FAST_RUN: | ||
| As fast as possible with visualization, | ||
| FAST_NO_GRAPHICS: | ||
| As fast as possible without visualization. | ||
An example simulation configuration creation.
3.3 Executing the Simulation
To start execution of a scenario, it is advised to first start Webots with the base world file manually. If Webots crashes or communication is lost, can restart Webots when necessary (only is Webots is executing in the same system). To start the simulation, Simulation Configurator should be used to connect to the Simulation Supervisor, send the simulation environment and configuration details, start the simulation, and finally collect the simulation trace. Below is an example source code for this step.
An example for execution of the simulation.
4 Combinatorial Testing
provides basic functionality to read test scenarios from csv files, in which, each row represents a test case and each column holds the values of a test parameter. This feature can be used to automatically execute a set of predefined test cases. The user can create csv files that contain a set of test cases either manually or by using a tool. 15 provides a list of methods provided in covering_array_utilities.py.
| load_experiment_data(file_name, header_line_count=6, index_col=None) | |
|---|---|
| Description: | Loads test cases from a csv file. |
| Arguments: | file_name: Name of the csv file |
| header_line_count: Number of header lines to skip. (default: 6) | |
| index_col: Index of the column that is used for data indexing. (default: None) | |
| Return: | A pandas data frame (a tabular data structure) containing all test cases. |
| get_experiment_all_fields(exp_data_frame, exp_ind) | |
| Description: | Returns all parameters for a test case (one row of the test table). |
| Arguments: | exp_data_frame: Data frame that was read using |
| load_experiment_data | |
| exp_ind: Index of the experiment | |
| Return: | A row from the test table that contains the requested test case. |
| get_field_value_for_current_experiment(cur_experiment, field_name) | |
| Description: | Returns the value of the requested test parameter. |
| Arguments: | cur_experiment: Current test case that was read using |
| get_experiment_all_fields. | |
| field_name: Name of the test parameter. | |
| Return: | Value of the requested parameter for the given test case. |
For combinatorial testing, ACTS from NIST [4] can be used as a tool for generating covering arrays.
Providing a complete guide on covering arrays and how to use ACTS is out of the scope of this chapter.
The user of ACTS creates a system definition by defining the system parameters and providing the possible values for each parameter.
The system definition is saved as an xml file.
ACTS can generate a covering array of desired strength and export the outputs in a csv file.
An example of a system definition and a corresponding 2-way covering array are provided in distribution in the files tests/examples/TutorialExampleSystemACTS.xml and
tests/examples/TutorialExample_CA_2way.csv, respectively.
Below is an example of reading test cases from a csv file and running each test one by one.
Note that, in the example, simulation trajectories are not evaluated.
Outputs of each test case should be evaluated to decide the test result.
An example for running covering array test cases.
In this example, a 2-way covering array for the test parameters is generated in ACTS. 16 shows the original test parameter name from the file tutorial_example_1.py, the name used in ACTS, and the possible values for each parameter (corresponding ACTS file is TutorialExampleSystemACTS.xml).
| Parameter | Name in ACTS | Possible Values |
|---|---|---|
| ego_init_speed_m_s | ego_init_speed | 0, 5, 10, 15 |
| ego_x_pos | ego_x_position | 15, 20, 25 |
| pedestrian_speed | pedestrian_speed | 2,3,4,5 |
After entering the parameter descriptions in 16 to ACTS, a 2-way covering array is generated and exported as a csv file. Listing 16 shows the contents of the output csv file. The file is available as: tests/examples/TutorialExample_CA_2way.csv.
Below is the source code that is reading the test cases from the csv file using the functions listed in 15 and running each test case one by one.
5 Falsification / Search-based Testing
For performing falsification, a CPS falsification tool like [3] can be used. As is in , we need to call the test cases which are developed in Python from . In this section, first a simple approach is described to call the test cases from , then a simple setup is described to perform falsification. This chapter does not provide a complete guide to . Reader is referred to website for further details 666iwebsite: https://sites.google.com/a/asu.edu/s-taliro/s-taliro.
5.1 Connecting to
has built-in support to call Python functions. However, type conversions are required both in and in the code 777The user is free to use a different approach than what is described here.. provides basic functionality to convert simulation trace to a format (single dimensional list) that can be easily interpreted in . Listing 18 shows the necessary updates on the Python side. First, sim_duration argument is added to run_test function allow run different length of simulations. Next, for_matlab argument is added to tell the function is called from so that experiment_tools.npArray2Matlab method from is used to convert simulation trace to a single list for .
On the side, we need a wrapper function to call the function run_test. Listing 19 provides an example to such a wrapper function. This code can be found in distribution in the file /tests/examples/run_tutorial_example_from_matlab.m. Unused return parameters YT, LT, CLG, GRD and input arguments steptime, InpSignal are to maintain compatibility with function call format.
5.2 Connecting to
To perform falsification with , an MTL specification should be defined, ranges for test parameters and the code, which is given in Listing 19, for running a simulation should be provided to as the model under test. Listing 20 gives a very simple example for falsification with an MTL requirement that checks x and y coordinates of Ego and agent vehicles to decide a collision. This code can be found in distribution in /tests/examples/run_falsification.m. The requirement is not to collide onto the agent vehicle. Note that, this requirement is very simplified to provide a simple example source code. A more complete check for collision would require incorporating further vehicle details and/or extracting collision information from the simulation.
6 Other Remarks
A user guide for is provided with a running example. The source code for the running example used in this chapter is available in distribution under tests/examples folder. As is a research tool that is not directly targeted for production-level systems special caution should be taken before using it for testing any critical functionality. is still evolving and it may contain a number of bugs or parts that are open to optimization. Here, only major functionality provided by is discussed. It provides further functionality like a number of sample vehicle controller subsystems, additional computations on simulation trajectory such as collision detection, and functionality toward evaluating perception-system performance. For a deeper understanding of ’s capabilities, the reader is encouraged to go through the source code for examples and experiments that are used as case studies for publications.
References
- [1]
- [2] Cyberbotics: Webots User Manual and Reference Manual. https://www.cyberbotics.com/support. Accessed: 2019-02-28.
- [3] Georgios Fainekos, Sriram Sankaranarayanan, Koichi Ueda & Hakan Yazarel (2012): Verification of Automotive Control Applications using S-TaLiRo. In: Proceedings of the American Control Conference.
- [4] D Richard Kuhn, Raghu N Kacker & Yu Lei (2013): Introduction to combinatorial testing. CRC press.
- [5] Cumhur Erkan Tuncali (2019): Search-based Test Generation for Automated Driving Systems: From Perception to Control Logic. Ph.D. thesis, Arizona State University.
- [6] Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito & James Kapinski (2018): Sim-ATAV: Open-Source Simulation-based Adversarial Test Generation Framework for Autonomous Vehicles. Available at https://cpslab.assembla.com/spaces/sim-atav/git/source.
- [7] Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito & James Kapinski (2018): Sim-ATAV: Simulation-Based Adversarial Testing Framework for Autonomous Vehicles. In: Proceedings of the 21st International Conference on Hybrid Systems: Computation and Control (part of CPS Week), ACM, pp. 283–284.
-
[8]
Cumhur Erkan Tuncali,
Georgios Fainekos,
Hisahiro Ito &
James Kapinski
(2018):
Simulation-based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components
. In: IEEE Intelligent Vehicles Symposium (IV).
Comments
There are no comments yet.