The Oxford Radar RobotCar Dataset: A Radar Extension to the Oxford RobotCar Dataset

09/03/2019 ∙ by Dan Barnes, et al. ∙ 0

In this paper we present The Oxford Radar RobotCar Dataset, a new dataset for researching scene understanding using Millimetre-Wave FMCW scanning radar data. The target application is autonomous vehicles where this modality remains unencumbered by environmental conditions such as fog, rain, snow, or lens flare, which typically challenge other sensor modalities such as vision and LIDAR. The data were gathered in January 2019 over thirty-two traversals of a central Oxford route spanning a total of 280 km of urban driving. It encompasses a variety of weather, traffic, and lighting conditions. This 4.7 TB dataset consists of over 240,000 scans from a Navtech CTS350-X radar and 2.4 million scans from two Velodyne HDL-32E 3D LIDARs; along with six cameras, two 2D LIDARs, and a GPS/INS receiver. In addition we release ground truth optimised radar odometry to provide an additional impetus to research in this domain. The full dataset is available for download at: ori.ox.ac.uk/datasets/radar-robotcar-dataset

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

page 6

page 7

Code Repositories

radar-robotcar-dataset-sdk

Additional Helpers for the Oxford Radar RobotCar Dataset https://ori.ox.ac.uk/datasets/radar-robotcar-dataset


view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

While many of the challenges in urban autonomy have been met successfully with lasers and cameras, radar offers the field of robotics a new level of robust sensing. The fmcw class of radar provides a -view of the scene and is capable of detecting targets at ranges far exceeding those of 3D LIDAR. These advantages are particularly valuable to autonomous vehicles which need to see further if they are to travel safely at higher speeds or to operate in wide open spaces where there is a dearth of distinct features. Moreover, these vehicles must function reliably in unstructured environments and require a sensor such as radar that thrives in all conditions – rain, snow, dust, fog, or direct sunlight.

This dataset builds upon the Oxford RobotCar Dataset [RobotcarDatasetIJRR], one of the the largest available datasets for autonomous driving research. The original dataset release consisted of over of vehicle-mounted monocular and stereo imagery, 2D and 3D LIDAR, as well as inertial and GPS data collected over a year of driving in Oxford, UK. More than traversals of a route were performed over this period to capture scene variation over a range of timescales, from the day/night illumination cycle to long-term seasonal variations. As a valuable resource for self-driving research, the vehicle software and mechatronics have been maintained since the original dataset was gathered and released: now configured with a millimetre-wave radar and two additional 3D LIDARs. The current appearance of the vehicle with these additional sensors can be seen in Figure 2. Along with the raw sensor recordings from all sensors, we provide an updated set of calibrations, ground truth trajectory for the radar sensor as well as MATLAB and Python development tools for utilising the data.

By sharing this large-scale radar dataset with researchers we aim to accelerate research into this promising modality for mobile robotics and autonomous vehicles of the future.

Fig. 1: The Oxford Radar RobotCar Dataset for complex and robust scene understanding with Millimetre-Wave fmcw scanning radar data. We collected traversals of a central Oxford route with the Oxford RobotCar platform during the month of January, 2019. Despite weather conditions such as rain, direct sunlight, and fog which are challenging for traditional modalities such as vision (left), radar (right) holds the promise of consistent sensor observations for mapping, localisation, and scene understanding. Sample pairs are taken from different locations of the driven route.

Ii Related Work

A number of LIDAR- and vision-based autonomous driving datasets, such as [pandey2011ford, geiger2013vision, blanco2014malaga, cordts2016cityscapes, yu2018bdd100k, nuscenes2019, lyft2019, huang2018apolloscape], are available to the community and were primarily collected in order to develop competencies in these modalities. This dataset release is meant to advocate the increased exploitation of fmcw radar for vehicle autonomy and thus presents radar data alongside the camera and LIDAR data typically appearing in datasets such as these; towards replicating and advancing these competencies with this promising sensor modality.

Some papers using similar, if not identical, fmcw radar for state estimation prior to the release of this dataset include 

[vivet2013localization, schuster2016landmark, 2018ICRA_cen, 2019ICRA_cen, 2019ICRA_aldera]. To this end, Section VI discusses the optimised ground truth radar odometry data released as part of this dataset to help further research in this area.

The Navtech radar dataset presented in [2019ICRA_park] is concurrent to this release. Although significantly smaller in size than our release, the comparable setups should provide a great opportunity for cross-validating approaches between datasets in different geographical locations. The Marulan datasets presented in [peynot2010marulan] also use fmcw radar, but only configured to a maximum range of . Additionally, while these datasets are collected under variable conditions, they represent fairly static outdoor scenes that are not representative of urban driving.

Iii The Radar RobotCar Platform

The dataset was collected using the same Oxford RobotCar platform as in [RobotcarDatasetIJRR], an autonomous-capable Nissan LEAF, illustrated with sensor layout in Figure 2. For this release, the RobotCar was equipped with the following sensors which were not in the original release:

Fig. 2: The Radar RobotCar platform (top) and sensor location diagram (bottom) with the Navtech CTS350-X radar mounted in the centre. Coordinate frames show the origin and direction of each sensor mounted on the vehicle with the convention: -forward (red), -right (green), -down (blue). Measurements shown are approximate; the development tools include exact extrinsic calibrations for all sensors.
  • 1 x Navtech CTS350-X Millimetre-Wave fmcw radar, , measurements per rotation, range, range resolution,  beamwidth

  • 2 x Velodyne HDL-32E 3D LIDAR,  HFoV,  VFoV, planes, , range, range resolution

In addition to the original sensors as in [RobotcarDatasetIJRR]:

  • 1 x Point Grey Bumblebee XB3 (BBX3-13S2C-38) trinocular stereo camera, , , " Sony ICX445 CCD, global shutter, lens,  HFoV, baseline

  • 3 x Point Grey Grasshopper2 (GS2-FW-14S5C-C) monocular camera, , , " Sony ICX285 CCD, global shutter, fisheye lens (Sunex DSL315B-650-F2.3),  HFoV

  • 2 x SICK LMS-151 2D LIDAR,  FoV, , range,  resolution

  • 1 x NovAtel SPAN-CPT ALIGN inertial and GPS navigation system, axis, , GPS/GLONASS, dual antenna

As the main focus of the release, the Navtech CTS350-X radar was mounted at the centre of the vehicle aligned to the vehicle axes. We used a pair of Velodyne HDL-32E 3D LIDARs instead of the LD-MRS 3D LIDAR used in [RobotcarDatasetIJRR] for drastically improved 3D scene understanding. In addition to providing twice the range and intensity returns, the Velodynes provide a full  HFoV with  VFoV for full coverage around the vehicle.

Sensor drivers for both the Navtech CTS350-X and Velodyne HDL-32E devices were developed internally to provide accurate synchronisation and timestamping with the other sensors. For further details on sensors from the original release, compute specifications, and data logging procedures please consult the original dataset paper [RobotcarDatasetIJRR].

Fig. 3: Example sensor data from the Navtech CTS350-X radar. Raw radar power power returns in polar form (left) for a full sweep with a maximum range of and the corresponding scan in cartesian form (right), with the vehicle in the center and maximum range set to . Tools required to parse the data and perform the polar-to-cartesian conversion are provided in the sdk discussed in Section VII.

Iv Radar Data

The Navtech CTS350-X is a fmcw scanning radar without doppler information, configured to return power readings at a range resolution of across zimuths at a frequency of (corresponding to a maximum range of and  azimuth resolution). Other configurations of the Navtech CTS350-X are able to provide range in excess of or higher rotation frequencies. However, for this dataset shorter range, high resolution data was deemed most useful in urban scenarios where straight line distances over are rare.

This type of radar rotates about its vertical axis while continuously transmitting and receiving frequency-modulated radio waves similar to a spinning LIDAR. The frequency shift between the transmitted and received waves is used to compute the range of an object, and the received power indicates the object’s reflectivity, size, and orientation relative to the receiver. One full rotation and its 2D power data can be represented by a matrix in which each row corresponds to an azimuth and each column to a range, as shown in Figure 3, where the intensity represents the highest power reflection within a range bin.

The radar operates at frequencies of to , ensuring consistent measurements through harsh local conditions such as dust, rain, and snow. The main beam spread is between points horizontally and vertically; with an additional cosec squared beam pattern up to below the horizontal which permits detection of objects beneath the main beam.

V Data Collection

Fig. 4: Random pairs of Bumblebee XB3 images (left) with the temporally closest Navtech CTS350-X radar scan (right) from the Oxford Radar RobotCar Dataset, showing the challenging diversity of weather, lighting, and traffic conditions encountered during the period of data collection in Oxford, UK in January 2019.

This dataset release follows the original Oxford RobotCar Dataset route in Oxford, UK and consists of traversals in different traffic, weather, and lighting conditions in January 2019 totalling of urban driving. The vehicle was driven manually throughout the period of data collection; no autonomous capabilities were used. The total download size of the dataset is . Figure 4 shows a random selection of images taken from the dataset, illustrating the variety of situations encountered. Table I lists summary statistics for the raw data collected through the entire month-long collection while Table II lists summary statistics for processed data which are also made available for download.

Every effort was made to follow the exact route for every traversal however this was not always possible and slight diversions were made infrequently. Additionally two partial traversals are included which do not cover the entire route. The GPS/INS data can be used to identify diversions. However, similarly to [RobotcarDatasetIJRR], the accuracy of the fused INS solution varied significantly during the course of data collection. Instead, we suggest using the optimised radar odometry shown in Figure 6 and discussed in Section VI as the best available solution of the underlying motion of the radar.

Sensor Type Count Size
Bumblebee XB3 Image 2,887,776 2.2 TB
Grasshopper 2 Image 2,963,601 1.6 TB
LMS-151 2D Scan 5,988,123 67.3 GB
SPAN-CPT GPS 3D Position 300,814 35.4 MB
SPAN-CPT INS 6DoF Position 3,008,085 491.7 MB
Navtech CTS350-X Radar Scan 240,088 106.1 GB
Velodyne Raw 3D Scan 2,405,785 91.0 GB
TABLE I: Summary statistics for collected data.
Sensor Type Count Size
Stereo vo 6DoF Position 961,487 89.0 MB
GT Radar Odometry 3DoF Position 240,024 28.6 MB
Velodyne Binary 3D Scan 2,405,785 774.3 GB
TABLE II: Summary statistics for processed data.

V-a Sensor Calibration

We provide a full set of extrinsic calibration data needed to utilise the additional Navtech and Velodyne sensors where the intrinsics and extrinsics of the sensors from [RobotcarDatasetIJRR] remain unchanged. Figure 2 illustrates the extrinsic configuration of sensors on the Radar RobotCar platform. The new LIDAR and radar sensors’ extrinsics were calibrated by manually taking measurements of the as-built positions of the sensors as a seed and then performing pose optimisation to minimise the error between laser and radar co-observations. Precise extrinsic calibrations for each sensor are included in the development tools to be discussed in Section VII. As per [RobotcarDatasetIJRR] the sensor extrinsics are not guaranteed to have remained constant throughout the lifetime of the vehicle. However, given the relatively short duration of this trial, little degradation is expected. Given the large overlap in observable environment and diversity of sensor modalities, this dataset provides an excellent test-bed for work on cross-modality calibration and we encourage using our estimates as initial seeds for further research.

Fig. 5: Directory layout for the Oxford Radar RobotCar Dataset. When downloading multiple zip archives from multiple traversals, extracting them all in the same directory will preserve the folder structure shown here.

V-B Data Formats

Figure 5 shows the typical directory structure for a single dataset. In contrast to [RobotcarDatasetIJRR] we do not chunk sensor data into smaller files. Therefore each zip file download corresponds to the complete sensor data for one dataset traversal (or processed sensor output such as stereo vo) with the folder structure inside the archive illustrated in Figure 5. The formats for each data type are as follows:

V-B1 Radar scans

are stored as lossless-compressed PNG files111https://www.w3.org/TR/PNG/ in polar form with each row representing the sensor reading at each azimuth and each column representing the raw power return at a particular range. The files are structured as <dataset>/radar/<timestamp>.png where <timestamp> is the starting UNIX timestamp of the capture, measured in microseconds. In the configuration used there are azimuths per sweep (rows) and range bins (columns).

To give users all the raw data they could need we also embed the following per azimuth metadata into the PNG image within the first columns as follows:

  • UNIX timestamp as int64 in cols 1-8.

  • Sweep counter as uint16 in cols 9-10; converted to angle in radians with:

     angle = sweep_counter / 2800 * 
  • Finally, a valid flag as uint8 in col 11.

The valid flag is included as there are a very small number of data packets carrying azimuth returns that are infrequently dropped. To this end, in order to simplify usage for users, we have interpolated adjacent returns so that each provided radar scan has azimuths (rows). If this is not desirable it is advised to simply drop any row which has the valid flag set to zero.

V-B2 3D Velodyne LIDAR scans

are provided in two formats, a raw form which encapsulates all the raw data recorded from the sensor for users to do with as they please, or in binary form representing the non-motion compensated pointcloud for a particular scan.

Raw scans: are released as lossless PNG files with each column representing the sensor reading at each azimuth. The files are structured <dataset>/<laser>/<timestamp>.png, where <laser> is velodyne_left or velodyne_right and <timestamp> is the starting UNIX timestamp of the capture, measured in microseconds. To give users all the raw data they could need we embed per azimuth metadata into the PNG within the following rows:

  • Raw intensities for each laser as uint8 in rows 1-32.

  • Raw ranges for each laser as uint16 in rows 33-96, converted to metres with:

     ranges (metres) = ranges_raw * 0.02
  • Sweep counter as uint16 in rows 97-98; converted to angle in radians with:

     angle = sweep_counter / 18000 * 
  • Finally, approximate UNIX timestamps as int64 in rows 99-106

Timestamps are received for each data packet from the Velodyne LIDAR which includes sets of readings for all lasers. We have linearly interpolated timestamps at each azimuth reading. However, the user is able extract the original received timestamps by simply taking every twelfth timestamp.

Binary scans: are released as single-precision floating point values packed into a binary file representing the non-motion compensated pointcloud generated from the corresponding raw scan, similar to the Velodyne scan format in [geiger2013vision]. The files are structured as <dataset>/<laser>/<timestamp>.bin, where <laser> is velodyne_left or velodyne_right and <timestamp> is the starting UNIX timestamp of the capture, measured in microseconds. Each scan consists of x values, where , , are the 3D Cartesian coordinates of the LIDAR return relative to the sensor (in metres), and is the measured intensity value.

V-B3 Ground Truth Radar Odometry

The files <dataset>/gt/radar_odometry.csv contain the relative pose solution as detailed in Section VI

, consisting of the source and destination frame UNIX timestamps (chosen to be in the middle of the corresponding radar scans), the six-vector Euler parameterisation (

, , , , , ) of the relative pose relating the two frames (where , , are all zero) and the starting source and destination frame UNIX timestamps of the corresponding radar scans which can be used as the <timestamp> to load the corresponding radar scan files.

Fig. 6: Optimised radar odometry plotted on OpenStreetMap [OpenStreetMap] for each of the dataset traversals, where each run is offset for visualisation purposes. The trajectories were generated by optimising robust vo [barnes2018driven], visual loop closures [cummins2008fab], and GPS/INS as constraints. Map data copyrighted OpenStreetMap contributors and available from openstreetmap.org.

Vi Ground Truth Radar Odometry

Alongside this dataset we provide ground truth radar odometry temporally aligned to the radar data to help further research using this modality for motion estimation, map building, and localisation. The poses were generated by performing a large-scale optimisation with Ceres Solver [ceres-solver] incorporating vo, visual loop closures, and GPS/INS constraints with the resulting trajectories shown in Figure 6.

Specifically, we include all dataset traversals and calculate robust vo using the approach proposed in [barnes2018driven]

, in which each image is masked with a neural network before generating odometry estimates using 

[churchill2012experience]. Visual loop closures are then found within and across each traversal using FAB-MAP [cummins2008fab]. For each traversal we optimise the vo, GPS/INS, and individual loop closures in the radar frame to obtain an approximately accurate global pose estimate. Finally, all pose chains are jointly optimised with all constraints before using slerp to create the ground truth, time-synchronised radar odometry.

Fig. 7: Example sensor data from the Velodyne HDL-32E 3D LIDAR. A raw Velodyne scan (left) stores intensities (top) and ranges (bottom) for each azimuth (columns) whereas a binary scan stores the cartesian pointcloud (right). Tools required to parse the data and perform the raw-to-pointcloud conversion are provided in the sdk mentioned in Section VII. Here the raw scan (left) is shown with invalid pixels set to black and stretched colourmap to improve visibility for the reader.

Vii Development Tools

We provide a set of MATLAB and Python development tools for easy access to and manipulation of the newly provided data formats; where tools for sensors from the original dataset, such as for imagery, remain unchanged. The new tools include simple functions to load and display radar and Velodyne scans as well as more complex functionality such as converting the polar radar data into cartesian form and converting raw Velodyne data into a pointcloud. To simplify usage these tools have been merged back into the original Oxford RobotCar Dataset sdk222github.com/ori-mrg/robotcar-dataset-sdk

. We also provide, and plan to extend, additional functionality useful to the community such as a batch downloader script for this dataset and deep learning dataloaders; for up to date information on these please refer to the dataset website.

Vii-a Radar Loading and Conversion to Cartesian

The MATLAB and Python functions LoadRadar.m and load_radar.py read a raw radar scan from a specified directory and at a specified timestamp, and return the per-azimuth UNIX timestamps (), azimuth angles (), and power returns () as well as the range resolution () as described previously. For this data release radar resolution will always equal .

The functions RadarPolarToCartesian.m and radar_polar_to_cartesian.py take the azimuth angles (), power returns () and radar range resolution () from a decoded radar scan and converts the polar scan into cartesian form according to a desired cartesian resolution () and cartesian size ().

The scripts PlayRadar.m and play_radar.py produce an animation of the available radar scans from a dataset directory as well as performing polar-to-cartesian conversion as shown in Figure 3; please consult this script and the individual functions for demo usage.

Vii-B Velodyne Loading and Conversion to Pointcloud

Similarly, the MATLAB and Python functions LoadVelodyneRaw.m and load_velodyne_raw.py read a raw Velodyne scan from a specified directory and at a specified timestamp, of the form <timestamp>.png, and return ranges (), intensities (), azimuth angles (), and approximate timestamps () as described previously.

The functions VelodyneRawToPointcloud.m and velodyne_raw_to_pointcloud.py take the ranges (), intensities (), and azimuth angles () from a decoded raw Velodyne scan and produce a pointcloud in cartesian form including per-point intensity values.

The functions LoadVelodyneBinary.m and load_velodyne_binary.py read a binary Velodyne scan from a specified directory and at a specified timestamp, of the form <timestamp>.bin, and returns a pointcloud in cartesian form including per-point intensity values.

Finally, the scripts PlayVelodyne.m and play_velodyne.py produce an animation of the available Velodyne scans from a dataset directory in different forms from the raw or binary formats, as shown in Figure 7; please consult this script and the individual functions for demo usage.

Viii Summary and Future Work

We have presented the The Oxford Radar RobotCar Dataset, a new large-scale dataset focused on further exploitation of of millimetre-wave radar scanners for large-scale and long-term vehicle autonomy and mobile robotics. Although this modality has received relatively little attention in this context, we anticipate that this release will help foster discussion of its uses within the community and encourage new and interesting areas of research not possible before.

In the future, we would like to continue to collect and share large-scale radar datasets in new and challenging conditions and more precisely fine-tune the current extrinsic calibration parameters, perhaps by using publicly available toolboxes designed for radar-LIDAR-camera systems such as [ICRA19_Domhof]. Additionally we would like to investigate semantic scene understanding in radar and show that it is a viable alternative for otherwise commonly used sensors like vision and LIDAR.

Ix Acknowledgements

The authors thank all the members of the ori who performed scheduled driving over the data collection period. We would also like to thank our partners at Navtech Radar, without whom this dataset release would not have been possible.

Dan Barnes is supported by the UK EPSRC Doctoral Training Partnership. Matthew Gadd is supported by Innovate UK under CAV2 – Stream 1 CRD (DRIVEN). Paul Newman and Ingmar Posner are supported by EPSRC Programme Grant EP/M019918/1.

References