Connected Autonomous Vehicles (CAVs) are enhanced daily with new autonomous features, leading gradually to fully autonomous means-of-transports [fully_autonomy]. Being part of the ecosystem of Intelligent Transportation Systems (ITSs), CAVs will require an agile interconnecting framework [broadband, TassiTVT], providing a constant service and optimal system behavior. To optimize and further enhance the performance of this framework, initial experimental evaluation of real-world trials and simulation results is required.
In this work, we aim to establish a connection between simulated and trial-based results for a vehicular network. Introducing the procedure followed, we will describe how inconsistencies during the experiments are identified and excluded from the evaluation. We will later present a calibration framework for the fine-tuning of the imperfect simulation results to enhance their behavior and achieve high-fidelity “real-world” results.
Simulations are approximated models of the physical world. However, they are easily and inexpensively conducted using an appropriate network simulator achieving near-perfect results. For example, the number of vehicles within a network can be easily scaled up to increase the network congestion (as in [dense_network]). Furthermore, they offer a high degree of flexibility. Using different configurations and isolating particular parameters, we can examine the behavior of a system under specific conditions. This is the case of [adaptive_beaconing] where the impact of different beacon intervals has been investigated with respect to the end-to-end delay, for different road networks and city-wide scenarios.
On the contrary, real-world trials are based on a “perfect” model. For example, authors in [experiments] experimentally analyzed the performance of a vehicular network based on data coming from off-the-shelf IEEE 802.11p devices. The disadvantages of this experimental evaluation are the cost, the required time and the inability to reproduce – since it is affected by physical parameters varying over time and it cannot be easily isolated and ignored.
A very complex real-world system is harder to model or might require increased resources to be simulated. Trial results can aid the design of a system by abstracting various parameters and introducing them as a priori knowledge in a simulation. For instance, authors in [gemv], designed a simulator for geometry-based efficient propagation models for Vehicle-to-Vehicle (V2V) communications. Their simulator was based on an extensive trial campaign, in order to identify parameters such as the path loss exponent and the small-scale signal deviation for different distances and environments. Trial-based results though may vary between different devices, being related with the quality of the equipment used and the software accompanying it.
For the above reasons, it is obvious that various approximations, random and systematic errors are introduced during a system performance evaluation. A direct comparison between the simulated and real-world results will end up in a performance difference. To increase the accuracy of this scientific evaluation, in this paper we will establish a framework where real-world trials and simulations co-exist. Sharing knowledge between them, we will fine-tune a full-stack network simulator, enhancing the accuracy of our results and giving us the leverage of more precise experimentation later. For our trials, an open-source testbed will be used, consisting of single-board devices equipped with different wireless Network Interface Controllers (NICs) designed to be IEEE 802.11p compliant. The simulated results will be acquired using the VEINS network simulator [veins]. This is a vehicular networking framework based on Omnet++ [omnetpp] and is compliant with the IEEE 802.11p/Wireless Access in Vehicular Environments (WAVE) standards.
The rest of the paper is organized as follows. In Sec. II, the hierarchical framework and the interrelationship between simulations and trials are described. The different entities of this framework, the relationship between them and the practical issues of the validation process are discussed. In Sec. III, the procedure to fine-tune VEINS is analyzed, starting from an initial calibration isolating various parameters and moving towards a full-stack system optimization. Individually analyzing each scenario, their fine-tuned performance evaluation is examined in Sec. LABEL:sec:results. Finally, Sec. LABEL:sec:conc concludes the paper and provides future research avenues.
Ii Hierarchical Framework: Trials and Simulations
Consider an ITS consisting of a number of CAVs and Road Side Units (RSUs) on a road network. Different kinds of data are exchanged with respect to the safety- or infotainment-related applications and services running on the ITS. For example, WAVE Short Messages (WSMs) [wave_standards], are safety-critical messages encapsulating core information about CAVs (e.g. position, velocity, size, etc.). These messages are either broadcast every or are triggered to announce road hazards. They are relatively short (~-) and a high delivery rate and a low one-hop end-to-end delay are regarded as their Quality-of-Service (QoS) constraints.
Apart from WSMs, safety-critical applications will be key in future ITSs. For example, video-assisted overtaking or traffic monitoring applications are tested on CAVs [video]. These applications require the transmission of video streams encapsulated within UDP packets. Increased data rate and low jitter are their main QoS requirements, with a more forgiving bit error rate (BER) performance due to the adoption of Forward Error Correction (FEC) codes and the new generation higher efficiency video encoders.
Following the content-related QoS requirements, each application behaves differently under various physical environments. Three different environments can be found in vehicular networks (urban, suburban, rural) [rician_downs]. For example, urban environments are affected by blockages from the buildings, significantly attenuating the signal. Urban canyon behaviors can be introduced, under specific circumstances, waveguiding the signal. On the other hand, foliage is the main form of blockage in rural areas while vehicles tend to move faster introducing a higher Doppler Shift effect. Each scenario should be approached differently when simulated, adapting accordingly the various channel model characteristics.
Ii-a Co-operation and Co-existence of Trials and Simulations
Cooperation between trials and simulations is mandatory to increase the accuracy of a system performance validation. Exchanging information between them can enhance the outcome, maximize the time utilization and minimize the cost. Establishing a framework between the reality-of-interest, i.e. the part of the real world (e.g. a city, a neighborhood or a road) that we are interested in investigating, the trials conducted and the simulation models used, will help us better understand the requirements and the limitations of each one. The interrelationship between these three entities can be found in Fig. 1.
The assessment of the simulation accuracy can be divided in two phases, the validation and the verification [verify_validate]. The verification (Fig. 1 – a) is the confirmation that a model is correctly implemented and reflects the real world. This is confirmed by conceptual models – abstractions of a system that characterize and standardize a network function, e.g. the OSI model (Fig. 1 – b). Verifying a model, the existing errors can be determined and fixed to assure that it matches specifications and assumptions with respect to the initial concept (Fig. 1 – c). For this work, we will not discuss the concept of the model verification for VEINS any further. For more, we kindly refer the reader to [veins_validation].
A trial-based evaluation is limited by the engineering constraints (Fig. 1 – d). For example, a NIC might support decreased transmission power compared to the standard specifications, limiting the operational range of a device. What is more, trials suffer from uncertainties (e.g. the attenuation of the signal with respect to the weather is unpredictable, not easily measured and environment-dependent) affecting the reliability and the validity of the experiment. The replicability of the experiment is also a big concern. All roads are not the same and the devices have different specifications, so replicating an experiment is difficult. This leads to incomplete results (Fig. 1 – e) as it is impossible to validate all possible combinations.
The validation of a theoretical model assesses the fidelity that a model reproduces the state and behavior of the real world (Fig. 1 – f). Verification usually precedes the validation of a model. The models can be validated with simple experiments isolating the external factors affecting the performance. To achieve a meaningful representation of the real world, a simulator should be fine-tuned at first (Fig. 1 – g) using inputs from measured results (e.g. path loss exponent) or applying weights at the output to minimize the divergence error. A direct comparison between the absolute values of the experimental results should be avoided. The validation process should focus on the trends of performance between the different experiments (e.g. both the simulated and trial-based results have a relative degradation when one parameter is changed).
Ii-B Hierarchical Validation of a Simulation Model
Differences between trial and simulated results can arise from various reasons. Typical examples are measurement errors (e.g. calibration errors, noise or data acquisition methods, etc.), formulation errors (e.g. incorrect channel models) or numerical errors (e.g. overflow of integers, subtraction of floating points, etc.). Two different error categories exist. Firstly, the random errors, affecting the relative precision of a model or a measurement. Secondly, the systematic errors affecting the absolute value of a result, being repeatable though and therefore, easily predicted.
In order to achieve the required level of accuracy, the validation procedure should be carried out throughout the entire development process following a hierarchical approach [validation]. Fragmenting the problem into smaller entities and solving them individually, the necessary level of precision can be achieved without increasing the complexity. Generally, the trials should test crucial features of the simulation models, such as the impact of the considered assumptions or the simplifications. On the other hand, simulations allow incremental validation towards a “real-world-like” system.
Iii Fine-Tuning VEINS Network Simulator
Designing a real-world ITS solution, all the above should be taken into account as well as the initial conditions, the boundaries and the trends of the performance. Isolating the characteristics that disruptively affect the performance we can isolate the systematic and the random errors approaching an ”ideal-like” system. Using a network simulator, we can further validate different scenarios based on our initial configuration. To do so, the fine-tuning process of VEINS requires a detailed study on the available hardware, the performance metrics used, an insight into how the simulator operates and what are the differences of the real-world. This study will be discussed in the next subsections.
Iii-a Description of the Experimental Setup
For our experimental validation, we prototyped an open-source IEEE 802.11p/DSRC testbed (Fig. 2) meeting the following requirements:
Low cost with the potential to be massively deploy later.
Open-source operating system providing enough flexibility for future developments.
Dual-operation as RSUs and On-Board Units (OBUs).
To meet these requirements, each device was equipped with a Mikrotik RB433 single-board computer (CPU , RAM, storage space, x3 Ethernets, x3 MiniPCI slots) [rb433]. Two IEEE 802.11a NICs were used for redundancy in the wireless links. The first one was a Mikrotik R52H [r52h], regarded in this work as a low-power (LP) NIC, with transmission power of up to and connected to a dipole antenna of gain at . The second model was a Mikrotik R5SHPn [R5SHPn], operating as the high-power (HP) transceiver in our experiments ( maximum transmission power). This NIC was connected to a -gain dipole antenna.
A low-latency OpenWRT Linux distribution was used111OpenWRT Barrier Breaker Release no. 14.07 - https://openwrt.org/ as the operating system. The different Atheros chipsets of each transceiver (AR5414 for the LP and AR9220 for the HP) required the use of two different Atheros drivers (ath5k for the LP and ath9k for the HP). Both were modified accordingly to enable IEEE 802.11p compatibility. The Linux kernel modules that we modified have been summarized in Fig. 3. The software modules cfg80211 and nl80211 act as interfaces between the user and kernel space, mac80211 is the general driver framework, and iw is the NIC configuration utility. Furthermore, cfg80211_ops and ieee80211_ops define the operations and the callbacks between the different blocks. The Outside the Context of a BSS (OCB) mode was enabled in the MAC layer, allowing the NICs to operate without being associated with an access point and the iw utility was modified accordingly to include the new commands for using OCB mode. Finally, the band was added in the regulatory domain.
Each device can be connected with a GPS dongle via its USB interface. Finally, a beaconing interface was developed that was able to acquire viable vehicle information (e.g. length, position, emissions, etc.) from an Engine Control Unit (ECU) and broadcast them to surrounding devices.
Iii-B Initial V2X Calibration Scenario
Consider a scenario with an ideal channel (no reflections, free-space path loss) between two ideal stationary vehicles (isotropic antennas, zero hardware attenuation). As this ideal system does not exist in the real world, the experimental setup was initially demonstrated inside an anechoic chamber () using both the HP and LP transceivers. A number of experiments was conducted at a distance of ~ for each transceiver and each MCS using two devices, one acting as a RSU and the other as an OBU. A UDP data stream, transmitted from the OBU to the RSU and generated using iPerf traffic generator222iPerf Traffic Generator - https://iperf.fr, and a periodic beacon every , were used to saturate the channel. As known, the network level performance is affected by the signal-to-interference-plus-noise ratio (SINR) and the sensitivity levels for each MCS. The SINR degrades based on the disruptive characteristics of the channel (e.g. distance attenuation, multipath, antenna misalignment, etc.) and the devices (e.g. thermal noise, etc.). Using an anechoic chamber, we conducted the experiments under near-optimal conditions (SINR greater than the sensitivity level) and therefore the optimal performance was achieved.
The same scenario was designed in VEINS as well. By partitioning the design process into smaller problems, we managed to achieve the required level of similarity. Firstly, we considered the IEEE 802.11p Physical Layer (PHY) frame, which consists of three fields [physical_layer]: (i) The Preamble marks the beginning of the PHY frame, is responsible for the appropriate antenna selection and corrects the timing and frequency offsets, (ii) the Signal field (SIG) specifies the frame rate and length and (iii) The Data
field consisting of the Physical layer Service Data Unit (PSDU) that encapsulates the MAC frame, the Physical Layer Convergence Procedure (PLCP) Service, and a Tail field. The Data field can also be padded with extra bits so its length is a multiple of the coded bits in an OFDM symbol. The above are transmitted using BPSKModulation and Coding Scheme (MCS). The length of each field can be found in Tab. LABEL:tab:parameters.
In VEINS, the duration of the Preamble and SIG is controlled by the parameter preambleDuration, while the bit length of the Data field is set equal to headerBitLength (see Fig. 4). The simulated PHY bitrates are governed by the wireless interface operational mode – namely, opMode that has been set to “p”, in this case. Each MCS should be manually configured for each individual simulation and matched with the appropriate PHY bitrate using pairs of the simulation parameters bitrate and modulation. Other PHY parameters that should be set within VEINS are the channel bandwidth, the carrierFrequency, the antennaType (ConstantAntennaGain in this case) and the gain.
The multi-channel operation introduced in the WAVE 1609.4 standard [wave_standards] was not considered in order to identify the maximum performance under saturation conditions. Furthermore, the RTS threshold – namely rtsThreshold, was set to a value greater than the frame size. This ensured that the RTS/CTS procedure was disabled. The MAC layer backoff times are drawn from a Contention Window (CW) starting from (cwMinData and cwMaxData). The values chosen for our setup (Tab. LABEL:tab:parameters) where proven to be optimal for vehicular communications [contention]. The length of the MAC TX queue size is capped by the driver (in the ath5k case), so the same value was considered in the simulation as well (maxQueueSize parameter within VEINS).
The VEINS parameter sentInterval sets the interval between the generation of two consecutive UDP packets. In order to saturate the channel, a very precise interval was chosen for each MCS. Suitable values were found with a trial and error method to fully utilize the channel without having packets discarded from the MAC TX queue. All the simulation and experimental parameters can be found in Tab. LABEL:tab:parameters and Sec. III-A.