Towards Safety-Aware Computing System Design in Autonomous Vehicles

05/21/2019 ∙ by Hengyu Zhao, et al. ∙ 0

Recently, autonomous driving development ignited competition among car makers and technical corporations. Low-level automation cars are already commercially available. But high automated vehicles where the vehicle drives by itself without human monitoring is still at infancy. Such autonomous vehicles (AVs) entirely rely on the computing system in the car to to interpret the environment and make driving decisions. Therefore, computing system design is essential particularly in enhancing the attainment of driving safety. However, to our knowledge, no clear guideline exists so far regarding safety-aware AV computing system and architecture design. To understand the safety requirement of AV computing system, we performed a field study by running industrial Level-4 autonomous driving fleets in various locations, road conditions, and traffic patterns. The field study indicates that traditional computing system performance metrics, such as tail latency, average latency, maximum latency, and timeout, cannot fully satisfy the safety requirement for AV computing system design. To address this issue, we propose a `safety score' as a primary metric for measuring the level of safety in AV computing system design. Furthermore, we propose a perception latency model, which helps architects estimate the safety score of given architecture and system design without physically testing them in an AV. We demonstrate the use of our safety score and latency model, by developing and evaluating a safety-aware AV computing system computation hardware resource management scheme.



There are no comments yet.


page 4

page 5

page 10

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

We are on the cusp of a transportation revolution where the autonomous vehicle (also called self-driving cars, uncrewed cars, or driverless cars) are likely to become an essential mobility option [42]. The time when people can sit back and tell their cars where to drive themselves is getting closer every day. Today, autonomous vehicles (AVs) incorporate sophisticated suites of sensors, such as cameras, light detecting and ranging (LiDAR), and radar (Figure 1); These sensors are backed by advanced computing system software and hardware that interpret massive streams of data in real-time. As such, autonomous driving promises new levels of efficiency and takes driver fatigue and human errors out of the safety equation.

However, the perceived technology will shift the burden of safety guarantee towards the vehicle system, and therefore, we need to ensure the safe operation of autonomous driving systems before releasing them to the public. As a consequence, the U.S. Department of Transportation recently released “A Vision for Safety” [7] to offer a path forward for the safe deployment of AVs and encourage ideas that deliver safer autonomous driving. Safety remains one of the most critical requirements for autonomous driving system design.

Figure 1: Top view of a typical autonomous vehicle with various sensors.

This paper focuses on nominal safety (discussed in § 2.2) of high automation vehicles, i.e., Level-4 autonomous vehicles. While traditional cars rely on human drivers to make driving decisions, future high automation vehicles will depend on the computing system for driving decision making (§ 2). Therefore, the computing system is one of the most critical components to ensure safety.

To investigate safety implications on the AV computing system design, we performed a field study (§ 3) by running’s industrial Level-4 autonomous driving fleets under testing in various locations, road conditions, and traffic patterns. Based on the knowledge learned from extensive data collected during our field study, we identify that traditional computing system metrics, such as tail latency, average latency, maximum latency, and timeout, do not accurately reflect the safety requirement of AV computing system design (§ 3.2). Instead, the level of AV safety is a non-linear function of accumulative instantaneous computing system response time and various external factors – including the AV’s acceleration, velocity, physical properties (e.g., braking distance), traffic, and road condition.

To guide safety-aware AV computing system design, we propose “safety score”, a safety metric that measures the level of AV safety based on computing system response time and the aforementioned external factors in a non-linear format (§ 4). Our safety score is rigorously derived from the published industrial formal AV safety model, Responsibility-Sensitive Safety (RSS) safety model [52]. RSS focuses on general rules for an AV to keep a safe distance with surrounding vehicles. Our safety score bridges such a conceptual model to AV computing system design with a quantitative approach. We hope the proposed safety score will be an essential contribution to the existing computer system metric portfolio. To facilitate the computation of safety scores, we also propose a latency model for offline computing system design evaluation. In summary, this paper makes the following contributions:

  • We provide a tutorial of state-of-the-art AV systems based on our survey and the practical experience on our AV systems. We offer a detailed discussion of automation levels, classification of safety requirements and standards, and computing system software and hardware organizations.

  • We perform a field study with our AV fleets. We present observations and safety-aware computing system design challenges based on the extensive data collected from our field study.

  • We propose a safety score, a metric that evaluates the level of safety in AV computing system design. By analyzing the safety score, we present a set of implications on safety-aware AV system design.

  • We propose a perception latency model, which allows AV computing system architects to estimate the safety score of a given computing system and architecture design in various driving conditions, without physically testing them in a car.

  • We demonstrate an example of utilizing our safety score and latency model to guide AV system computation resource management with a hardware and software co-design.

2 State-of-the-art AV System

High automation cars are under development and testing in both industry [3, 1, 5, 13] and academia [8, 11, 12]. To understand the current status of AV development, we examined the state-of-the-art AV design, AV safety, and hardware and software organizations.

2.1 Levels of Automation

The Society of Automotive Engineers (SAE) categorizes autonomous driving systems into six automation levels, which range from no automation (Level-0) to full automation (Level-5) [27]. Most commercialized autonomous driving systems are on partial automation (Level-2) and conditional automation (Level-3). At Level-2, the vehicle can steer, accelerate, and brake in certain circumstances; however, the driver needs to respond to traffic signals, lane changes, and scan for hazards. For example, Tesla’s latest Model S [3] is a Level-2 commercialized car. At Level-3, the car can manage most aspects of driving, including monitoring the environment; but the driver needs to be available to take over at any time when the vehicle encounters a scenario it cannot handle [4]. For example, NVIDIA recently announced a partnership with Audi to build Level-3 autonomous driving cars [1]. High automation (Level-4) AVs are under development. A Level-4, AV drives itself almost all the time without any human input, but might be programmed not to operate in unmapped areas or during severe weather. For instance, Waymo [5], originated as a project of Google, is developing a Level-4 autonomous driving system [45]. Full automation vehicles (Level-5), which can operate on any road and in any conditions that a human driver could manage, is not yet demonstrated. But this is a long-term goal of autonomous driving development.

This paper focuses on Level-4 AVs, which introduce much more critical safety requirements on the computing system than lower-level AVs. For example, Level-3 AVs treat human drivers as a backup of the AV computing system; but Level-4 AVs entirely rely on the computing system to make driving decisions in almost all scenarios.

2.2 AV Safety

In automotive industry, safety requirements are classified into functional safety and nominal safety 

[52]. Functional safety refers to the integrity of operation of vehicle’s electrical system, including hardware and software. For example, a hardware failure or a bug in software will both lead to a functional safety hazard. Functional safety of vehicles is standardized by ISO 26262 [6], which defines various automotive safety integrity levels that offer (a) failure-in-time targets for hardware and (b) systematic processes for software development and testing that conform with appropriate systems engineering practices.

However, unlike in a conventional vehicle, computing system plays an integral role in an AV. Even functionally safe, AV can still crash due to untimely or unsafe driving decision. As Riccardo Mariani, Intel Fellow and chief functional safety technologist in the internet of things group, pointed out, current ISO standard of functional safety is inadequate to ensure the safety of AV [2]. This falls in the domain of nominal safety. Nominal safety refers to whether the AV makes timely and safe driving decisions, assuming that the hardware and software are operating error free (i.e., functionally safe).

This paper focuses on nominal safety of AV. While software module algorithm design determines whether a driving decision (e.g., braking, acceleration, or steering) is safe, timely driving decision making heavily relies on the performance of computing system and architecture design. Two recent studies by Mobileye [52] and NVIDIA [47] defined formal nominal safety rules for AV. However, these studies focus on how to make planning and control decisions to keep a safe distance with surrounding vehicles. To our knowledge, no previous research investigates how to ensure timely driving decision making with safety-aware computing systems design. As such, this paper focuses on exploring safety-aware computing system design methodology that achieves timely driving decision making, by mapping formal safety requirements onto AV computing system design.

Note that it is impossible to guarantee that an AV will never be involved in an accident [52]. It is also difficult to predict or control the activity of other vehicles [47]. Therefore, we intend to guarantee that the AV will be sufficiently careful to avoid becoming a cause of an accident, the same as recent AV safety studies [52].

2.3 AV Computing System Architecture

The AV computing system performs similar tasks as a driver of a conventional car, including localization, perception, and planning and control tasks (described in § 2.4). In the rest of this paper, we call AV computing system as “AV system” for short.

While embedded or low-end processors may be sufficient to meet the computation requirement of Level-2 and Level-3 AVs, current Level-4 AV system designs typically adopt high-end heterogeneous architectures, which comprise sophisticated CPUs, GPUs, storage devices (e.g., terabytes of SSDs), FPGAs [9] and other accelerators, to accommodate the intensive computation demand. As such, high automation AV systems are fundamentally different from conventional real-time embedded systems [55]. Table 1 lists the heterogeneous architecture employed by our prototypes.

Implication: Although the computing system adopts server-grade processors, it is impossible to minimize the latency of all computation tasks due to the vast amount of interdependently executing software modules (discussed in § 2.5) and the finite hardware resources in the system. As such, it is critical to effectively utilize the limited hardware resources, while meeting given safety requirement.

CPU Intel Xeon processor
Main memory 16GB DDR4
Operating system Ubuntu
GPU NVIDIA Volta architecture
Device memory 12GB HBM2
Table 1: Computing system and architecture configurations.

2.4 Tasks of AV Computing System

The computing system interacts with various sensors and the car control system to perform three primary functions: localization, perception, and planning and control. Figure 2 illustrates the general working flow of the state-of-the-art Level-4 AV computing system.

Localization. Localization identifies the location of the vehicle on a high definition map, based on the information obtained from LiDAR, camera, and GPS. The high definition map stores static information on the way, such as lane lines, trees, guardrails, and the location of traffic lights and stop signs [20]. GPS offers a prediction of the current location of the vehicle on a global scale; but such prediction is not sufficiently accurate to allow the computing system to perform driving tasks, e.g., staying within a lane on the road [21]. Therefore, GPS information needs to be accompanied with LiDAR and camera “perception” (described below) of surrounding static objects, to interpret the local environment accurately.

Perception. Perception detects and interprets surrounding static (e.g., lane lines, trees, and traffic lights) and moving (e.g. other vehicles and pedestrians) objects with three types of sensors, LiDAR, cameras, and millimeter-wave radar, as shown in Figure 1. In the rest of the paper, we refer all the objects, which can be detected by perception, i.e., in the detection range, as “obstacles”. Localization typically only requires infrequent perception of static objects, while traffic interpretation requires frequent (e.g., per-frame – each frame of data collected by the sensors) perception of both static and moving objects.

Compared to camera and radar, LiDAR offers much higher accuracy on detecting the position, 3D geometry, speed, and moving direction of obstacles [50, 33]. Therefore, most of current Level-4 AVs highly rely on LiDAR to perform the perception task [31, 35, 26]. LiDAR continues to emit lasers across a 360 degree view. It receives reflected signals, whenever an obstacle blocks the laser. Based on the received signals, we can build a “point cloud” to represent the location and 3D geometry of the obstacle [49]; each point in the graph represents a received signal.

In most cases, the computing systems can identify obstacle properties based on the point clouds generated from LiDAR data, as shown in the left part of Figure 5. Cameras facilitate the obstacle interpretation by capturing color information (the right part of Figure 5). Radar is used to detect the radial speeds of moving obstacles [41]. After sensor data collection, the computing system performs a “perception fusion” to integrate the data from the three types of sensors.

Overall, perception obtains accurate information of location, geometry, speed, and moving direction of obstacles. Such information is essential for localization and making appropriate driving decisions to avoid collisions in planning/control.

Figure 2: AV computing system tasks.

Planning and control. The essential functions of planning and control include prediction, trajectory planning, and vehicle control. Their dependency relationship is shown in Figure 2. A motion planner employs localization and perception information to make driving control decisions, such as steering angle, acceleration, and braking of the vehicle. Functional safety is of critical importance to the motion planner. As such, a backup mechanism, commonly radar or sonar [43], is used to verify the conclusion informed by other components of the system. With the perception results, the prediction function tracks the behavior of obstacles at real-time and predicts their next movements. The computing system generates an optimal trajectory for the vehicle, based on the prediction and perception results. The control function will control the steering, acceleration, and deceleration of the vehicle based on the driving decision made by the planning function.

2.5 LiDAR Perception

As perception in most of the current Level-4 AV designs heavily relies on LiDAR, we present more details of LiDAR perception.

Dependency graph. To process LiDAR data, the computing system executes a considerable number of inter-dependent software modules (i.e., software programs) at real-time. To understand the dependency relationship among these software modules, we build a dependency graph based on our AV system. Figure 3 shows an example111We only show renamed high-level modules due to confidential reasons.. LiDAR perception consists of a set of main modules (shown in blue) and sub-modules (shown in orange). For example, the Labeling module first sorts out a rectangular region-of-interest (ROI), which restricts the range of LiDAR detection; then, it calls its four sub-modules to perform finer-grained labeling within the ROI. As another example, Segmentation extracts useful semantic and instance information (e.g., surrounding vehicles and pedestrians) from the background. The Merge module clusters the points, which are likely to belong to the same object, in a point cloud; the Filter modules process the point clouds. Some software modules, e.g., Segmentation, do not only impose long latency, but also have a strong dependency with other software modules.

Implication: Due to their long latency and high dependency, certain software modules contribute more to the overall perception latency and computing system response time than other modules. Therefore, these modules are typically more safety critical.

Figure 3: Dependency graph of LiDAR perception.

2.6 Types of Computing System Workloads

AV system workloads consist of both machine learning and non-machine-learning (non-ML) software modules. In our AVs, camera perception software modules are commonly deep-learning-based computer vision workloads; but most of the software programs in our AV system are non-ML. For example, some classification and segmentation modules can employ Support Vector Machine (SVM) 

[51] and fully connected networks; camera data perception may adopt ResNet [30]. Currently, lots of the software programs running in our AV system, even for the perception task, are non-ML workloads; many of these non-ML workloads are as time-consuming and compute-intensive as deep learning workloads. One such example is the Segmentation module shown in Figure 3.

Furthermore, many software modules can execute on both CPU and GPU with the same algorithms but different implementations. This introduces substantial flexibility in managing and allocating the various hardware resources among the software workloads.

3 Field Study and Observations

To explore realistic AV safety requirement and computing system hardware/software behaviors, we run a fleet of Level-4 AVs in various locations, road conditions, and traffic patterns over three contiguous months. Our field study yields over 200 hours and 2000 miles of traces with a considerable size of data. The data is collected at the granularity of one sensor frame, which is the smallest granularity that the computing system interprets and responses to obstacles. With each frame, we collect (1) the execution latency of each software module, (2) total computing system response time, (3) the utilization of computing system hardware resources (CPU, GPU, and memories), (4) the environment information obtained by the sensors (e.g., the distribution of static and moving obstacles), and (5) instantaneous velocity and acceleration of the AV.

Our field study is performed as part of the company’s routine road tests. The data we collected incorporates a large variety of scenarios, including near-accident scenes.

Figure 4: (a) Computing system response time breakdown. (b) The latency of LiDAR perception main modules.

3.1 Observations

Breakdown of computing system response time and the impact of architecture design. As shown in Figure 4(a), the majority (74%) of computing system response time is consumed by perception. LiDAR perception is the major contributor to perception latency. As nominal safety requires a timely response, it is critical to ensure timely LiDAR perception. In addition, we observe that most of the workloads running in our AV system are compute-intensive. Various software modules have a large variety of computation requirements and parallelism characteristics. As a result, native heterogeneous system hardware resource management easily results in unbalanced compute resource utilization, leading to significant delays in software execution.

Safety-critical software module analysis. We further analyze the execution latency of the main modules. Figure 4(b) shows an example with LiDAR perception modules by illustrating average, maximum and minimum, and tail latency of the ten main software modules across thousands of executions during our months of field study. We have two critical observations. First, maximum, average, and tail latency yields different criticality rankings on the modules. For example, specific modules with high maximum latency (e.g., Occupancy grid) have low average and tail latency. We observe that maximum latency is typically achieved in safety-challenging scenarios, such as heavy traffic and near-accident situations, e.g., Figure 5 explained below. Therefore, maximum latency better indicates safety-criticality of a software module than other formats of latency. Second, we observe that safety-critical modules also typically have strict inter-dependency with other modules in the dependency graph. Examples include Segmentation and Filter. Based on the two observations, safety-critical modules can be identified as those having long maximum latency and strict inter-dependency with other modules.

Figure 5: An emergency hardbrake case that we encountered.

Timely traffic perception is the most critical to AV safety. Due to the limited ranges of sensor detection, the computing system has limited time (typically within one or several sensor sampling intervals) to interpret an obstacle in the traffic after it enters the sensor-detectable region. Moreover, as perception consumes the majority of computing system response time, timely traffic perception is the most critical to AV safety. Specifically, LiDAR perception is the most safety critical in our AV system (and many other Level-4 AV implementations), because most of the perception task relies on LiDAR. In fact, we encountered several safety emergencies, when the AV needed to perform a hardbrake due to the delay of LiDAR perception. Figure 5 illustrates one of such safety emergency incidents we encountered. As shown in the right camera view, a white SUV is passing through an intersection, when our AV intends to enter the same intersection222The white SUV entered the intersection before the traffic light turned red, passing through at a high speed.. Our system recognized the traversing SUV within a LiDAR sampling interval and ended up performing a hardbrake as shown in Figure 5(d). But if the perception had taken slightly longer time, a collision would have happened.

Correlation between perception latency and obstacle distribution. We observe that perception latency depends on both computing system configuration and the distribution of obstacles (especially in traffic). With a given computing system hardware configuration, obstacles closer to the vehicle have a higher impact on perception latency than distant ones. Furthermore, obstacles that horizontally distribute in front of the vehicle have a higher impact on perception latency than those distributed in other directions. Finally, among the close obstacles, the denser the obstacle distribution, the longer the perception latency would be. The rationale underlying is that close, horizontally-distributed, and dense obstacles reflect more LiDAR signals than other obstacles, hence generate more data for LiDAR perception.

Figure 6: The relationship between LiDAR perception latency and computing system response time.

Latency accumulation effect. We observe a latency accumulation effect of certain software modules. Figure 6 illustrates an example of LiDAR perception modules: once LiDAR perception latency exceeds 100ms, the system response time increases at a much higher rate (in the region with a steeper slope in Figure 6) with the increase of LiDAR perception latency. The reason is two-fold. First, because LiDAR has a fixed sampling rate (with a 100ms interval in our AV), the perception, if not completed within the sampling interval, will further delay the processing of subsequent frames. Second, as software modules are inter-dependent, a delay of one software module will further delay its dependent software modules. The slopes and thresholds of the accumulation effect vary across different software modules, depending on individual characteristics and the inter-dependency relationship with other modules.

Impact of other external factors. Besides traffic, several other external factors also have a critical impact on the safety requirement of AV system design. These factors include the acceleration and velocity of the AV and vehicles around it, AV’s physical properties (e.g., braking distance), and road condition. Taking the scenario in Figure 5 as an example, if our AV had a higher velocity or longer braking distance, the collision would also have happened.

3.2 Safety-aware Design Challenges

As discussed in § 2.2, this paper focuses on nominal safety that ensures timely driving decision making by the computing system. It seems that timely driving decision making can be achieved by minimizing the computing system response time. However, this is challenging. To ensure achievable minimum response time within finite computing system hardware resources, performance optimization needs to be guided by specific metrics. Based on our field study, traditional computing system performance metrics, such as average latency, maximum latency, tail latency, and timeouts, do not accurately reflect the safety requirement of AV systems due to the following reasons.

First, AV safety is determined by instantaneous response time (Figure 5), instead of statistical patterns adopted in traditional performance metrics. Second, it is challenging to predict the response times due to the complex inter-dependent relationships between software modules and the latency accumulation effect. Third, AV safety is also determined by various external factors as discussed in our field study observations. As a result, the level of AV safety is not a simple weighted sum of the elements, but a non-linear function of response time as we identified in our study (§ 4). Finally, although timeout is widely used to measure real-time system performance [55], it is impractical to simply use timeout to evaluate the safety of AV system design: It is difficult to determine the threshold required to calculate the timeout of AV system response time, because each software module has a different threshold (e.g., different sampling interval with various sensors); External factors further lead to dynamically changing thresholds. Therefore, we need to rethink the fundamental metric used to guide safety-aware AV system design.

4 Safety Score

In this section, we propose safety score, a metric that measures the level of AV safety based on computing system response time and external factors to guide safety-aware AV system design.

4.1 Safety Score Description

Our safety score is rigorously derived from the published industrial formal AV safety model – Responsibility-Sensitive Safety (RSS) [52]

. RSS is developed by Mobileye, an Intel subsidiary that develops advanced driver-assistance systems (ADAS) to provide warnings for collision prevention and mitigation. RSS defines safe distances around an AV, formal mathematical models that identify the emergency moment, when the safe distances are compromised, and proper planning and control decisions an AV needs to perform during an emergency. RSS has been endorsed by various AV projects, such as Valeo 

[14] and Baidu’s Apollo Program [10].

Unfortunately, RSS does not provide many insights on safety-aware computing system design. To give a quantitative guideline for safety-aware computing system design, we map RSS to the safety score based on our field study observations. The safety score is defined as the following equation (check Appendix A for precise mathematical formulation of safety score):

Variable Description
Instantaneous computing system response time, defined by Equation 2.
Response time window.
, Indicating the velocity and acceleration of an AV and surrounding vehicles, defined by Equation 8 in Appendix A.
, Reward or penalty on the level of safety, when is lower or higher than , respectively.
Table 2: Variables in safety score.

The variables in the equation are described in Table 2. Here, is the “instantaneous” computing system response time, which we define as the response time to one frame of sensor data (or several frames depending on the perception granularity of an AV implementation). We calculate by the following equation:


Here, is the instantaneous latency of a safety-critical module on the critical path; is the total number of such modules. Safety critical modules are determined based on the inter-dependency relationship in the dependency graph and the maximum latency via the extensive pre-product road tests (§ 3). is an accumulation effect function, which reflects the contribution of to system response time . For each safe-critical module, we obtain a latency accumulation effect curve that can be described by a function . Figure 6 shows an example of such curves, where the x-axis is and y-axis is the value of . But curve shapes vary across different modules.

The rest variables in Equation 1 reflect the impact of external factors. The response time window is the minimum time that the AV would take to avoid hitting another vehicle that is certain distance away, if both vehicles keep moving at the same direction, velocity, and acceleration, i.e., neither of vehicles perform any response to a reducing distance. We allow AV developers to define the distance in various manners, e.g., as (1) a minimum safe distance allowed by traffic rules, which is adopted by our experiments as an example to make our evaluation more concrete or (2) a user-defined confident safe distance. Figure 16 in Appendix A provides a more detailed analysis of .

Variables and reflect the velocity and acceleration of the AV and the surrounding vehicle, calculated by Equation 8 in Appendix A. Safety reward and penalty are user-defined coefficients, which indicate the user’s confidence of safety. We allow users to adjust the reward and penalty based on their own road test experiences, such as the velocity of the vehicles in the traffic that the AV is likely to run into, road condition, and AV braking distance. We discuss more detailed implications of the reward and penalty in § 4.2. Appendix A provides more detailed discussion of these variables.

No clear boundary is defined between safe and unsafe in various AV safety models [52, 47]. As such, the safety score does not intend to give a threshold of safe versus unsafe either. Instead, it implies the level of safety – higher safety score (could be either positive or negative) indicates safer AV system design.

Figure 7: The non-linear relationship between safety score and instantaneous response time.

4.2 Implications to Computing System Design

Based on our practical experience, safety score provides the following key implications on the safety-aware AV system design.

Nonlinearity. The safety score is a non-linear function of instantaneous response time . Safety-aware computing system design approach highly relies on the shape of the curve. Figure 7 lists a few examples of safety score curves as a function of . With a given set of external factors, the shape of the curve is determined by response time window , safety reward , and safety penalty . Based on our road tests, the majority of safety emergency scenarios happen when . When , further optimizing the instantaneous response time does not significantly improve the level of safety. Therefore, in practice the shape of the safety score curve appears like the bold line (=0.05, =0.1) shown in Figure 7, with a lower reward than penalty. As such, computing system design needs to prioritize reducing the instantaneous response time , when .

Instantaneous latency. As we discussed before, the safety score is computed by instantaneous response time per sensor frame. Therefore, it is infeasible to adopt traditional performance metrics calculated based on statistical performance behaviors, such as tail, average, and maximum and minimum latency. § 7.1 quantitatively compares various metrics.

Safety impact across various software modules. Due to the complex inter-dependency relationship and latency accumulation effect of the software modules, the instantaneous latency of different software module has different impact on safety score. This is modeled by Equation 2, where each software module has a disparate form of the contribution to the instantaneous AV system response time.

Figure 8: Overview of the proposed perception latency model. (a) Sources of perception latency. (b) Latency model working flow.

5 Latency Model

The goal of our latency model is to estimate the instantaneous response time in safety score, given obstacle distribution and computation resource allocation. Our field study shows that perception latency is a significant contributor to system response time (§ 3). Therefore, we focus on modeling perception latency.

Our model is built based on two observations: first, closer and denser obstacles lead to higher perception latency (§ 3); second, latency also depends on computation resources allocated to run a software module333In our current AV system, such hardware resource allocation refers to whether executing a software module on CPU or GPU. Although our system does not adopt FPGA or other accelerators, our latency model applies to general heterogeneous architecture design with a variety of hardware resources. Figure 8 shows an overview of our latency model. The model input includes (1) obstacle density distribution and (2) computation resource allocation of each software module. The output is the estimated latency of each software module, i.e., in Equation 2. Our latency model estimates in two steps:

  • We first estimate a baseline latency (), which is the latency of executing a software module on a baseline computation resource (e.g., CPU in our AV system), using a baseline latency model presented by Equation 3.

  • Then, we calculate with latency conversion ratios based on given resource allocation plan.

In the following, we use LiDAR perception as an example to discuss our latency model in detail and can surely apply it to other perception modules, such as camera, radar, etc. § 7.2 evaluates the accuracy of our latency model.

5.1 Model Input

Obstacle density distribution vector. We design an obstacle count map to represent obstacle density distribution. The map divides the sensor ROI into regular grids, with each grid storing the number of obstacles in it. For example, our AV has a 64 50 ROI with a 2 2 grid size. This grid size captures most moving objects on the road, e.g., vehicles and bicycles. But large obstacles, which spread across multiple grids, are counted multiple times. To express more accurate obstacle density, we adopt a hierarchy of three levels of obstacle count maps with a grid size of 2 2, 810, and 64 50 (the whole ROI), respectively. With the map hierarchy, we generate an obstacle density distribution vector as illustrated in Figure 8(b).

Computation resource allocation vector. We represent computation resource allocation by a resource index vector , where each element in the vector indicates the resource allocation of a software module. For example, our current AV implementation only has two resource allocation options, running on CPU (index=0) or GPU (index=1). Then, indicates that software modules are executed on CPU, CPU, GPU, …, CPU.

5.2 Latency Model

Perception software modules vary in complexity. We take into account the diversity in the algorithm complexity and model the baseline latency of each software module () by the following equation,


Here, is a dimension obstacle density distribution vector, where and are the grid counts in the finest and second-level hierarchy, respectively. The operator is coefficient-wise vector product, and is the inner vector product.

This is a standard curve fitting problem. To solve for coefficients , , , , and

in the equation, we perform linear regression on massive road testing data, which is used to generate the model input data, i.e., obstacle density distribution and corresponding baseline latency.

The final latency of each perception module then can be estimated by


Here, is the ratio of executing the same module on a different computation resource than the baseline. This ratio is determined by exhaustive search of running a software module with various computation resource allocation configurations.

6 AV System Resource Management

We demonstrate an example of utilizing our safety score and latency model to guide the computing system design, by designing a heterogeneous computation resource management scheme. The goal of resource management is to determine the computation hardware resource allocation of each software module and the priority of software module execution to optimize safety score.

Overview. Our resource management has two phases, resource planning and resource scheduling. During the planning phase, we perform an exhaustive search of computation resource allocation and priority options of executing software modules in various obstacle distributions, to determine the options that maximize safety score given each obstacle distribution. This yields a resource management plan for each obstacle distribution. During the scheduling phase, we match the current obstacle distribution with resource management plans and determine the plan to use. To ensure sufficient resources to be scheduled for each software module, our AV system maintains redundant computation resources. Due to the large searching space, resource planning usually takes a long time. Therefore, we perform the planning phase offline, while the AV system only performs online scheduling. To further accelerate the online scheduling, offline planning groups obstacle distributions associated with the same resource management plan to clusters. As such, the scheduling phase only needs to match clusters.

6.1 Offline Planning

The offline planning phase analyzes the massive data collected by road tests to create a set of resource management plans with various obstacle distributions. A resource management plan includes (i) the computation resources allocated for software module execution (CPU or GPU in our current AV system design) and (ii) the priority of software modules, which is used to determine which module gets executed first, when multiple choices exist.

Algorithm 1 illustrates the offline planning procedure. For each field study sample, which comprises instantaneous computing system response time and the latency of each software module for an obstacle distribution, we perform an exhaustive search in the space of resource management plans. We categorize the samples, which achieve the highest safety score for the same resource management plan, in the same cluster. Based on our field study, samples in the same cluster have similar (1) obstacle density distribution and (2) perception timeout pattern, which is the number of continuous incidents where the perception latency exceeds the sensor sampling interval (e.g., a LiDAR sampling interval is 100ms). Therefore, we concatenate these two as the feature, and compute the feature vector of each cluster by averaging the feature of each sample in the cluster.

This phase only need to be done once, whenever the AV adapts to a new location or physical system. In practice, we found that the clustering and the associated resource management plans are stable over time, even though users can also periodically run the offline phase to re-calibrate.

8:for  do
11:for  do
12:     for  do
Algorithm 1 Offline planning.

6.2 Online Monitoring and Scheduling

The online phase adopts a software/hardware co-design approach to (1) monitor the computing system execution and (2) match the resource management plans. Figure 9 shows a system design overview. When the AV is driving, the computing system continuously (1) monitors instantaneous system response time and the pattern of perception timeout and (ii) determines when and which resource management plan to adopt.

A loop of three steps implements this: (Step-1) perception timeout pattern monitoring, (Step-2) cooperative cluster matching, and (Step-3) plan switching. The online phase triggers a hardware-based perception timeout pattern monitoring, whenever one of the safety critical perception software modules has a timeout (e.g. LiDAR); the monitoring hardware counts the number of continuous timeouts across the subsequent LiDAR sampling intervals to identify whether timeout happens accidentally or continuously, e.g., over a user-defined continuous timeouts. Based on our field study, LiDAR perception timeout happens either accidentally for a few times or continuously for more than 100 frames. Therefore, we set to be 100. If continuous timeout happens, the online phase performs the next two steps based on the following equations:


where . This is the current feature vector, which is the concatenation of current obstacle density distribution vector and current timeout pattern ; is the concatenation operator. is the feature vector of the cluster, which consists of the obstacle density distribution vector and timeout pattern of the cluster. Assuming road condition and traffic pattern do not dramatically change within a certain amount of time, the online resource management software will scan the features of all clusters and identify a matching cluster, by finding the cluster with the closest feature as the current one.

Figure 9: AV computing system design that performs safety-aware online monitoring and scheduling.

Hardware and software implementation. To meet the real-time requirement of online resource management, we implement Step-1 in CPU hardware by adding a set of comparators and counters as shown in Figure 9. The comparators track the timeouts; the counters accumulate the number of continuous timeouts. In our AV system, an 8-bit counter is sufficient to capture the continuous timeout patterns. Step-2 and Step-3 are triggered once Step-1 detects continuous timeouts. As a result, these Step-2 and Step-3 are performed infrequently. As such, we implement the primary functionality of these two steps by software as a load balancer [46, 40] in CPU; it acts as a reverse proxy of software module scheduling across CPU and GPU. The user-level implementation imposes negligible performance overhead demonstrated by our experiment results, due to the infrequent cluster matching and resource management plan switching. In our AV system, the software modules that can execute on both CPU and GPU are implemented and compiled with both versions. We store the indices generated during Step-2 in CPU main memory. Step-3 reads the memory to identify the resource management plan with the current feature.

7 Evaluation

Our evaluation employs two simulators. First, to evaluate safety score, we use an in-house AV simulator, which replays and analyzes road testing data444The simulator is also used by the company to analyze routine road testing data.. Second, we use an in-house computer architecture simulator to evaluate the effectiveness of the online resource monitoring and scheduling in our resource management scheme. We compare three computing system architecture configurations and resource management policies:

  • CPU – Executing all software modules on CPU;

  • CPU+GPU – The native heterogeneous AV system design. It employs tail latency to determine the priority of software module execution in resource management, with the GPU to accelerate all deep learning software modules.

  • Resource management – Heterogeneous AV computing system design with our resource management scheme. The results take into account the performance overhead of the online monitoring and scheduling phase.

7.1 The Need for Safety Score

Our safety-score, which is rigorously derived from the RSS model, measures the level of safety of AV system design. Furthermore, our AV simulator results show that the level of safety identified by our safety score is in line with the level of safety based on the probability of collisions.

To demonstrate that other traditional latency metrics do not accurately indicate the level of safety, we compare computing system resource management results guided by safety score, 95th percentile tail latency (99th percentile tail latency yields similar results), average latency, and maximum latency. We perform an exhaustive search of resource management plans for software modules. Due to space limitation, Figure 10 shows an example of five resource management plans. But the following observations hold with our exhaustive search. For fair comparison, we present the results obtained under the same set of obstacle distributions.

The results (the higher the better) show that safety-score-guided resource management leads to entirely different results than other metrics. For example, the safety score indicates that we should choose RM-1, whereas RM-2, RM-3, and RM-4 leads to the best tail latency, average latency, and maximum latency, respectively. Therefore, guiding the computing system design by these traditional latency metrics can be misleading; it is critical to adopt safety score.

Figure 10: Resource management (RM) guided by various metrics (y-axis is normalized to the maximum of each metric).

7.2 Latency Model Analysis

Pearson correlation coefficient. We adopt the Pearson correlation coefficient [19] to evaluate the correlation between LiDAR perception latency and obstacle density distribution. Figure 11 depicts a heat map of Pearson correlation coefficients distributed in an obstacle count map hierarchy, based on real driving scenarios during our field study. Instead of the count of obstacles, each grid in Figure 11 stores a Pearson correlation coefficient. The coefficient measures the correlation between obstacle density and LiDAR perception latency: the higher the coefficient, the larger the correlation between the two. Figure 11 uses colors to express Pearson correlation coefficient values: the darker the color, the larger the value. As such, in the areas with darker colors, LiDAR perception latency is more susceptible to obstacle density. We make three observations from Figure 11. First, LiDAR perception latency is more susceptible to nearby obstacles than those far away. This is in line with our field study observation. Second, both the top left and top right areas have high Pearson coefficient. This may be caused by heavy horizontal traffic through an intersection during rush hours. Finally, LiDAR perception latency is more sensitive to the density than the distribution of obstacles.

Model accuracy. We use mean squared error (MSE) [54] (the lower the better) to evaluate the accuracy of our perception latency model quantitatively. We investigate our latency model under a variety of driving scenarios, such as morning and afternoon, rush hours and non-rush hours, and local roads and express ways. The proposed latency model consistently provides high accuracy with an average MSE as low as .

Model coefficient. We borrow the above heat map methodology to analyze the coefficients in our perception latency model. Figure 12 shows the coefficients obtained based on our field study data. We make two observations. First, most coefficients are non-zero. Second, coefficients of different terms in Equation 3 have a significant variation on values. The coefficients of lower-order terms have larger values than those of higher-order terms. For instance, (Figure 12(a)), which are the coefficients of the highest order term in Equation 3, has the smallest values at a scale of .

Figure 11: Heat map of Pearson correlation coefficient for obstacle count map hierarchy.
Figure 12: Heat maps of perception latency model coefficients , , , and .
Figure 13: Safety score with various architecture configurations.
Figure 14: Energy efficiency of various AV computing system designs.
Figure 15: Average response time of various computing system designs.

7.3 Resource Management Analysis

Figure 13 compares the safety score of three different AV system architecture configurations and resource management policies. Compared with CPU and CPU+GPU, Resource Management achieves the highest safety score, which is and higher than CPU+GPU and CPU, respectively.

Beyond improving safety, Resource Management also improves computing system performance and energy efficiency. Figure 14 compares the energy-delay product (EDP) of various configurations. Resource Management leads to the highest EDP in both LiDAR perception alone and the whole computing system. Our hardware modifications to implement online resource management only impose less than 1% of total system energy consumption. Figure 15 compares the average response time of various configurations. Compared with CPU and GPU+CPU, Resource management reduces the response time averaged throughout our field study data by and 35%, respectively; LiDAR perception latency is reduced by and 45% on average.

8 Related Work

To our knowledge, this is the first paper to propose a safety metric and latency model used for guiding AV computing systems and architecture design, based on a large-scale field study of real industry Level-4 AVs. In this section, we discuss related works on AV nominal safety policies, safety-aware AV system design, and heterogeneous computing system design.

AV nominal safety policies. Two recent studies, RSS [52] and SFF [47], investigated AV nominal safety policies. Both studies define the high-level concept of AV nominal safety. Our paper is in line with this high-level safety concept. Unfortunately, neither of the studies investigates safety requirement on computing systems and architecture design. SFF [47] focuses on policies that ensure safe driving decisions, e.g., braking, acceleration, and steering, under given limitations, such as perception visibility and AV system response time. SFF guides the computing system software algorithm design to enforce safety policies. But the study does not provide policies on timely driving decision making, which is one of the essential goals of computing systems and architecture design. RSS [52] develops a set of general rules to ensure collision avoidance, by enforcing that the AV keeps a safe distance with surrounding vehicles. To this end, the safety rules treat the AV and surrounding vehicles as objects with given velocity and acceleration, without implications on the requirement of AV internal systems (including the computing system). Our study focuses on the nominal safety requirements of AV computing systems and architecture design.

Safety-aware AV system design. Most previous safety-aware AV system designs focus on developing software module algorithms [48, 38, 29, 28, 39, 15, 32, 23]. For instance, recent works improve the latency and precision of LiDAR- and camera-based perception, by leveraging video sensing, laser rangefinders, light-stripe range-finder to monitor all around the autonomous driving vehicle to improve safety with a map-based fusion system [16, 48, 38]. Several studies enhance the quality of driving decisions by optimizing perception and planning software algorithms [29, 28]. Previous works also investigate ways of effectively responding to uncertainty at real-time [39, 32, 15] and path planning to avoid static obstacles [23].

AV computing system design. Most previous studies on AV systems and architecture design focus on accelerator design and energy efficiency improvement of deep learning or computer vision software execution [42, 34]. The performance and energy optimizations are guided by standard performance metrics. Safety requirement is not adequately considered in most of these studies. As our evaluation demonstrates, safety-aware computing system design leads to different architecture and systems designs compared with guided by traditional design metrics. However, our metric and latency model can also be used to guide deep learning and computer vision accelerator design in AV systems.

Heterogeneous computing system design. AV computing systems adopt sophisticated heterogeneous architectures used in real-time scenarios. Prior works on real-time heterogeneous computing systems [53, 25, 22] focus on embedded processors, rather than sophisticated architectures. Previous studies that investigate server-grade CPU and GPU processor based systems focus on distributed and data center applications [36, 37, 18, 44, 24, 17]; in such cases, scalability is a more critical issue than single-system resource limitation. Furthermore, neither type of heterogeneous system use cases has the level of safety requirements as autonomous driving.

9 Conclusion

In summary, this paper proposes a safety score and perception latency model to guide safety-aware AV computing system design. Furthermore, we elaborated detailed AV computing system architecture design and workloads, as well as discussing a set of implications of safety-aware AV system design based on a field study with industrial AVs, and demonstrate the use of our safety score and latency model with a safety-aware heterogeneous computation resource management scheme. As high automation vehicles are still at the early stages of development, we believe our work only scratched the surface of safety-aware AV system design. Substantial follow-up work is required to enhance the computing system design towards safe autonomous driving.


  • [1] “Audi and NVIDIA automotive partners,”
  • [2] “AV safety ventures beyond ISO 26262,”
  • [3] “Model S,”
  • [4] “Self-driving cars: Understanding the 6 autonomous levels,”
  • [5] “Waymo,”
  • [6] “ISO, Road vehicles: Functional safety,” 2011.
  • [7] “NHTSA, Automated driving systems: A vision for safety,” 2017, 13069a-ads2.0_090617_v9a_tag.pdf.
  • [8] “Self-driving carts to make their debut on UC San Diego roads in January,” 2017. [Online]. Available:
  • [9] “Alphabet’s Waymo is now in the ’prove it’ stage of its self-driving journey,” 2018. [Online]. Available:
  • [10] “Baidu to integrate Mobileye’s responsibility sensitive safety model into Apollo program,” 2018. [Online]. Available:
  • [11] “Illinois to develop testing program for self-driving vehicles,” 2018. [Online]. Available:
  • [12] “Texas A&M launches autonomous shuttle project in downtown Bryan,” 2018. [Online]. Available:
  • [13] “Uber advanced technologies group,” 2019. [Online]. Available:
  • [14] “Valeo signs an agreement with Mobileye to develop a new autonomous vehicle safety standard,” 2019. [Online]. Available:
  • [15] M. Althoff, O. Stursberg, and M. Buss, “Model-based probabilistic collision detection in autonomous driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 10, pp. 299–310, 2009.
  • [16] R. Aufrère, J. Gowdy, C. Mertz, C. Thorpe, C.-C. Wang, and T. Yata, “Perception for collision avoidance and autonomous driving,” Mechatronics, vol. 13, no. 10, pp. 1149–1161, 2003.
  • [17]

    A. Beloglazov, J. Abawajy, and R. Buyya, “Energy-aware resource allocation heuristics for efficient management of data centers for cloud computing,”

    Future generation computer systems, vol. 28, no. 5, pp. 755–768, 2012.
  • [18] A. Beloglazov and R. Buyya, “Energy efficient resource management in virtualized cloud data centers,” in Proceedings of the 2010 10th IEEE/ACM international conference on cluster, cloud and grid computing.   IEEE Computer Society, 2010, pp. 826–831.
  • [19] J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” in Noise reduction in speech processing.   Springer, 2009, pp. 1–4.
  • [20] G. Bresson, Z. Alsayed, L. Yu, and S. Glaser, “Simultaneous localization and mapping: A survey of current trends in autonomous driving,” IEEE Transactions on Intelligent Vehicles, vol. 2, no. 3, pp. 194–220, 2017.
  • [21] W. Burgard, O. Brock, and C. Stachniss, “Map-based precision vehicle localization in urban environments,” in MIT Press, 2011, pp. 352–359.
  • [22] H. Chen, X. Zhu, H. Guo, J. Zhu, X. Qin, and J. Wu, “Towards energy-efficient scheduling for real-time tasks under uncertain cloud computing environment,” Journal of Systems and Software, vol. 99, pp. 20–35, 2015.
  • [23] K. Chu, M. Lee, and M. Sunwoo, “Local path planning for off-road autonomous driving with avoidance of static obstacles,” IEEE Transactions on Intelligent Transportation Systems, vol. 13, no. 4, pp. 1599–1616, 2012.
  • [24] K. Czajkowski, I. Foster, N. Karonis, C. Kesselman, S. Martin, W. Smith, and S. Tuecke, “A resource management architecture for metacomputing systems,” in Workshop on Job Scheduling Strategies for Parallel Processing.   Springer, 1998, pp. 62–82.
  • [25]

    Y. Dang, B. Wang, R. Brant, Z. Zhang, M. Alqallaf, and Z. Wu, “Anomaly detection for data streams in large-scale distributed heterogeneous computing environments,” in

    ICMLG2017 5th International Conference on Management Leadership and Governance.   Academic Conferences and publishing limited, 2017, p. 121.
  • [26] L. Delobel, C. Aynaud, R. Aufrere, C. Debain, R. Chapuis, T. Chateau, and C. Bernay-Angeletti, “Robust localization using a top-down approach with several lidar sensors,” in 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO).   IEEE, 2015, pp. 2371–2376.
  • [27] A. Driving, “Levels of driving automation are defined in new SAE international standard J3016: 2014,” SAE International. Google Scholar.
  • [28] A. Furda and L. Vlacic, “Towards increased road safety: Real-time decision making for driverless city vehicles,” in Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on.   IEEE, 2009, pp. 2421–2426.
  • [29] A. Furda and L. Vlacic, “Enabling safe autonomous driving in real-world city traffic using multiple criteria decision making,” IEEE Intelligent Transportation Systems Magazine, vol. 3, no. 1, pp. 4–17, 2011.
  • [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in

    Proceedings of the IEEE conference on computer vision and pattern recognition

    , 2016, pp. 770–778.
  • [31] J. Hecht, “Lidar for self-driving cars,” Optics and Photonics News, vol. 29, no. 1, pp. 26–33, 2018.
  • [32] T. Helldin, G. Falkman, M. Riveiro, and S. Davidsson, “Presenting system uncertainty in automotive uis for supporting trust calibration in autonomous driving,” in Proceedings of the 5th international conference on automotive user interfaces and interactive vehicular applications.   ACM, 2013, pp. 210–217.
  • [33] M. Himmelsbach, A. Mueller, T. Lüttel, and H.-J. Wünsche, “Lidar-based 3D object perception,” in Proceedings of 1st international workshop on cognition for technical systems, vol. 1, 2008.
  • [34] K. Jo, J. Kim, D. Kim, C. Jang, and M. Sunwoo, “Development of autonomous car part II: A case study on the implementation of an autonomous driving system based on distributed architecture,” IEEE Transactions on Industrial Electronics, vol. 62, no. 8, pp. 5119–5132, 2015.
  • [35] A. Joshi and M. R. James, “Generation of accurate lane-level maps from coarse prior maps and lidar,” IEEE Intelligent Transportation Systems Magazine, vol. 7, no. 1, pp. 19–29, 2015.
  • [36] A. Klimovic, H. Litz, and C. Kozyrakis, “Selecta: Heterogeneous cloud storage configuration for data analytics,” in Proceedings of the 2018 USENIX Conference on Usenix Annual Technical Conference, 2018, pp. 759–773.
  • [37] K. Krauter, R. Buyya, and M. Maheswaran, “A taxonomy and survey of grid resource management systems for distributed computing,” Software: Practice and Experience, vol. 32, no. 2, pp. 135–164, 2002.
  • [38] F. Kunz, D. Nuss, J. Wiest, H. Deusch, S. Reuter, F. Gritschneder, A. Scheel, M. Stübler, M. Bach, P. Hatzelmann et al., “Autonomous driving at Ulm university: A modular, robust, and sensor-independent fusion approach,” in Intelligent Vehicles Symposium (IV), 2015 IEEE.   IEEE, 2015, pp. 666–673.
  • [39] C. Laugier, I. Paromtchik, M. Perrollaz, Y. Mao, J.-D. Yoder, C. Tay, K. Mekhnacha, and A. Nègre, “Probabilistic analysis of dynamic scenes and collision risk assessment to improve driving safety,” Its Journal, vol. 3, no. 4, pp. 4–19, 2011.
  • [40] J. Lee, M. Samadi, Y. Park, and S. Mahlke, “Transparent CPU-GPU collaboration for data-parallel kernels on heterogeneous systems,” in Proceedings of the 22Nd International Conference on Parallel Architectures and Compilation Techniques, 2013, pp. 245–256.
  • [41] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt et al., “Towards fully autonomous driving: Systems and algorithms,” in Intelligent Vehicles Symposium (IV), 2011 IEEE.   IEEE, 2011, pp. 163–168.
  • [42] S.-C. Lin, Y. Zhang, C.-H. Hsu, M. Skach, M. E. Haque, L. Tang, and J. Mars, “The architectural implications of autonomous driving: Constraints and acceleration,” in Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems.   ACM, 2018, pp. 751–766.
  • [43] S. Liu, L. Li, J. Tang, S. Wu, and J.-L. Gaudiot, “Creating autonomous vehicle systems,” Synthesis Lectures on Computer Science, vol. 6, no. 1, pp. i–186, 2017.
  • [44] M. Maheswaran, S. Ali, H. Siegal, D. Hensgen, and R. F. Freund, “Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems,” in Heterogeneous Computing Workshop, 1999.(HCW’99) Proceedings. Eighth.   IEEE, 1999, pp. 30–44.
  • [45] B. Marr, “Key milestones of Waymo – Google’s self-driving cars,”
  • [46] S. Mittal and J. S. Vetter, “A survey of CPU-GPU heterogeneous computing techniques,” ACM Comput. Surv., vol. 47, no. 4, pp. 69:1–69:35, Jul. 2015.
  • [47] D. Nistér, H.-L. Lee, J. Ng, and Y. Wang, “The safety force field,” in NVIDIA White Paper, 2019.
  • [48] U. Ozguner, C. Stiller, and K. Redmill, “Systems for safety and autonomous behavior in cars: The darpa grand challenge experience,” Proceedings of the IEEE, vol. 95, no. 2, pp. 397–412, 2007.
  • [49] R. B. Rusu and S. Cousins, “3d is here: Point cloud library (pcl),” in Robotics and automation (ICRA), 2011 IEEE International Conference on.   IEEE, 2011, pp. 1–4.
  • [50] S. Schneider, M. Himmelsbach, T. Luettel, and H.-J. Wuensche, “Fusing vision and lidar-synchronization, correction and occlusion reasoning,” in 2010 IEEE Intelligent Vehicles Symposium.   IEEE, 2010, pp. 388–393.
  • [51] C. Schuldt, I. Laptev, and B. Caputo, “Recognizing human actions: a local SVM approach,” in Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, vol. 3.   IEEE, 2004, pp. 32–36.
  • [52] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” arXiv preprint arXiv:1708.06374, 2017.
  • [53] J. Song, J. Wang, L. Zhao, S. Huang, and G. Dissanayake, “MIS-SLAM: Real-time large scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing,” arXiv preprint arXiv:1803.02009, 2018.
  • [54] M. Tuchler, A. C. Singer, and R. Koetter, “Minimum mean squared error equalization using a priori information,” IEEE Transactions on Signal processing, vol. 50, no. 3, pp. 673–683, 2002.
  • [55] Q. Zhu, H. Zeng, W. Zheng, M. D. Natale, and A. Sangiovanni-Vincentelli, “Optimization of task allocation and priority assignment in hard real-time distributed systems,” ACM Transactions on Embedded Computing Systems (TECS), vol. 11, no. 4, p. 85, 2012.

Appendix A Safety Score Formulation

Responsibility-Sensitive Safety (RSS) [52] formalizes safety as assuring five common-sense rules, which AV driving policy should obey. Specifically, the policy will choose safe actions under various circumstances. The action is safe, if (1) given the current driving state and the actions and (2) when the surrounding vehicles are acting adversely but reasonably, a collision can be avoided.

More formally, a given driving state of the AV and the driving state of the vehicle that might have collision with the AV (referred to as “obstacle-of-attention”), together determine the minimum distance to avoid collision by Equation 7:


Here, and are velocities of the AV and the obstacle-of-attention, respectively; is the minimum/maximum acceleration of speeding up() or slowing down(). Let and be the response times of the AV and the obstacle-of-attention. All variables are non-negative: velocities and accelerations only represent the absolute value.

Parameters and are used to represent whether the two vehicles are driving in the opposite or the same directions. The values are scenario-dependent. For example, indicates that the two vehicles, the AV and the obstacle-of-attention, are driving towards each other, either along the lane direction or perpendicular to the lane direction. The reasonable conditions are the obstacle-of-attention takes some response time before it applies a brake; the most adverse condition is to use the minimum slowing down acceleration . When the two vehicles are driving towards the same direction, which is represented by , the most adverse condition is that the obstacle-of-attention instantly brakes () with highest possible acceleration .

Parameters depend on various environmental conditions and user’s safety preference. In practice, under bad weather or road conditions, we may adopt lower and than normal conditions, because vehicles tend to drive slower and apply cautious speed changes. is the minimum distance between the two vehicles, when they perform safe actions [52] and full stop. Users may specify larger to allow a larger safety margin or smaller to achieve higher driving flexibility. Higher enables the AV to react more agile, but may lead to the worse passenger experience. At any time, the AV speed is known. The AV sensors detect the speed of the obstacle-of-attention. Collapsing all the parameters and grouping by the only variable gives a quadratic form as Equation 8. Notice that and are non-negative.


According to RSS [52], if the current distance is larger than , the safety criteria is met. Without loss of generosity, a safety score function can be defined as Equation 9. is the reward specified by users to achieve extra margin from minimal safety distance; is the penalty of unit distance less than the minimum safety distance.


We employ Equation 8 to bridge the safety score with computing system response time. First, the current distance between the AV and the obstacle-of-attention will determine the response time window that will avoid the collision. The response time window is the time that the AV would take to hit the obstacle-of-attention, if both vehicles keep moving at the same velocity and acceleration – or more formally, if both vehicles conform the Duty of Care, i.e. an individual should exercise "reasonable care" while performing actions that might harm the safety of others [52]. For instance, the Duty of Care allows the front car to perform break, such that the distance to the rear car is reduced, but prohibits the front car to backup, when followed by another car. We determine by solving the following function:

Figure 16: Illustration of d- and t-.

Let be the AV computing system response time and determine . Figure 16 depicts the relationship between Equation 8 with the current distance and instantaneous computing system response time . Then the safety score function can be written as Equation 11.