Recovering and Simulating Pedestrians in the Wild

by   Ze Yang, et al.

Sensor simulation is a key component for testing the performance of self-driving vehicles and for data augmentation to better train perception systems. Typical approaches rely on artists to create both 3D assets and their animations to generate a new scenario. This, however, does not scale. In contrast, we propose to recover the shape and motion of pedestrians from sensor readings captured in the wild by a self-driving car driving around. Towards this goal, we formulate the problem as energy minimization in a deep structured model that exploits human shape priors, reprojection consistency with 2D poses extracted from images, and a ray-caster that encourages the reconstructed mesh to agree with the LiDAR readings. Importantly, we do not require any ground-truth 3D scans or 3D pose annotations. We then incorporate the reconstructed pedestrian assets bank in a realistic LiDAR simulation system by performing motion retargeting, and show that the simulated LiDAR data can be used to significantly reduce the amount of annotated real-world data required for visual perception tasks.


page 2

page 5

page 7

page 8


Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction

We present a novel method for testing the safety of self-driving vehicle...

Simulating LIDAR Point Cloud for Autonomous Driving using Real-world Scenes and Traffic Flows

We present a LIDAR simulation framework that can automatically generate ...

AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles

As self-driving systems become better, simulating scenarios where the au...

Lidar based Detection and Classification of Pedestrians and Vehicles Using Machine Learning Methods

The goal of this paper is to classify objects mapped by LiDAR sensor int...

Sequential Joint Shape and Pose Estimation of Vehicles with Application to Automatic Amodal Segmentation Labeling

Shape and pose estimation is a critical perception problem for a self-dr...

Lidar-Monocular Surface Reconstruction Using Line Segments

Structure from Motion (SfM) often fails to estimate accurate poses in en...

Diverse Complexity Measures for Dataset Curation in Self-driving

Modern self-driving autonomy systems heavily rely on deep learning. As a...