Querying Labelled Data with Scenario Programs for Sim-to-Real Validation

by   Edward Kim, et al.

Simulation-based testing of autonomous vehicles (AVs) has become an essential complement to road testing to ensure safety. Consequently, substantial research has focused on searching for failure scenarios in simulation. However, a fundamental question remains: are AV failure scenarios identified in simulation meaningful in reality, i.e., are they reproducible on the real system? Due to the sim-to-real gap arising from discrepancies between simulated and real sensor data, a failure scenario identified in simulation can be either a spurious artifact of the synthetic sensor data or an actual failure that persists with real sensor data. An approach to validate simulated failure scenarios is to identify instances of the scenario in a corpus of real data, and check if the failure persists on the real data. To this end, we propose a formal definition of what it means for a labelled data item to match an abstract scenario, encoded as a scenario program using the SCENIC probabilistic programming language. Using this definition, we develop a querying algorithm which, given a scenario program and a labelled dataset, finds the subset of data matching the scenario. Experiments demonstrate that our algorithm is accurate and efficient on a variety of realistic traffic scenarios, and scales to a reasonable number of agents.


page 1

page 4

page 7


Addressing the IEEE AV Test Challenge with Scenic and VerifAI

This paper summarizes our formal approach to testing autonomous vehicles...

Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World

We present a new approach to automated scenario-based testing of the saf...

Insertion of real agents behaviors in CARLA autonomous driving simulator

The role of simulation in autonomous driving is becoming increasingly im...

Parameterisation of lane-change scenarios from real-world data

Recent Autonomous Vehicles (AV) technology includes machine learning and...

Automatic lane change scenario extraction and generation of scenarios in OpenX format from real-world data

Autonomous Vehicles (AV)'s wide-scale deployment appears imminent despit...

Scalable Autonomous Vehicle Safety Validation through Dynamic Programming and Scene Decomposition

An open question in autonomous driving is how best to use simulation to ...

Can we learn where people come from? Retracing of origins in merging situations

One crucial information for a pedestrian crowd simulation is the number ...

Please sign up or login with your details

Forgot password? Click here to reset