nuScenes: A multimodal dataset for autonomous driving

by   Holger Caesar, et al.

Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image-based benchmark datasets have driven the development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We also define a new metric for 3D detection which consolidates the multiple aspects of the detection task: classification, localization, size, orientation, velocity and attribute estimation. We provide careful dataset analysis as well as baseline performance for lidar and image based detection methods. Data, development kit, and more information are available at


page 1

page 2

page 4

page 5

page 6


aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception

Autonomous driving is a popular research area within the computer vision...

Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection

Detecting vehicles and representing their position and orientation in th...

OSDaR23: Open Sensor Data for Rail 2023

For driverless train operation on mainline railways, several tasks need ...

Sensor Equivariance by LiDAR Projection Images

In this work, we propose an extension of conventional image data by an a...

Combined Image- and World-Space Tracking in Traffic Scenes

Tracking in urban street scenes plays a central role in autonomous syste...

Distant Vehicle Detection Using Radar and Vision

For autonomous vehicles to be able to operate successfully they need to ...

3D for Free: Crossmodal Transfer Learning using HD Maps

3D object detection is a core perceptual challenge for robotics and auto...

Please sign up or login with your details

Forgot password? Click here to reset