Over the past fifty years, computer-vision research has been devoted to standard, frame-based cameras (i.e., rolling or global shutter cameras) and only in the last few years cameras have been successfully used in commercial autonomous mobile robots, such as cars, drones, and vacuum cleaners, just to mention a few. Despite the recent progress, we believe that the advent of event-based cameras is about to revolutionize the robot sensing landscape. Indeed, the performance of a mobile robot in tasks, such as navigation, depends on the accuracy and latency of perception. The latency depends on the frequency of the sensor data plus the time it takes to process the data. It is typical in current robot-sensing pipelines to have latencies in the order of – or more, which puts a hard bound on the maximum agility of the platform. An event-based camera virtually eliminates the latency: data is transmitted using events, which have a latency in the order of micro-seconds. Another advantage of event-based cameras is their very high dynamic range ( vs. of standard cameras), which makes them ideal in scenes characterized by large illumination changes. Other key properties of event-based cameras are low-bandwidth, low-storage, and low-power requirements. All these properties enable the design of a new class of algorithms for high-speed and high-dynamic-range robotics, where standard cameras are typically not ideal because of motion blur, image saturation, and high latency. However, the way that event-based cameras convey the information is completely different from that of traditional sensors, so that a paradigm shift is needed to deal with them.
1.1 Related Datasets
The first work is tailored for comparison of event-based optical flow estimation algorithms (Rueckauer and Delbruck, 2016)
. It contains both synthetic and real datasets under pure rotational (3 degrees of freedom (DOF)) motion on simple scenes with strong visual contrasts. Ground truth was acquired using the inertial measurement unit (IMU). In contrast, our datasets contain arbitrary, hand-held, 6-DOF motion in a variety of artificial and natural scenes with precise ground-truth camera poses from a motion-capture system.
A more similar work to ours is (Barranco et al., 2016). Their focus is to create a dataset that facilitates comparison of event-based and frame-based methods for 2D and 3D visual navigation tasks. To this end, a ground robot was equipped with a DAVIS and a Microsoft Kinect RGB-D sensor. The DAVIS was mounted on a pan-tilt unit, thus it could be excited in 5-DOF. The scene contains checkerboards, books, and a chair. Ground truth was acquired by the encoders of pan-tilt unit and the ground robot’s wheel odometry, and is therefore subject to drift. In contrast, our dataset contains hand-held, 6-DOF motion (slow- and high-speed) on a variety of scenes with precise ground-truth camera poses from a motion-capture system, which is not subject to drift.
2 The DAVIS Sensor
The Dynamic and Active-pixel Vision Sensor (DAVIS) (Brandli et al., 2014) (see Fig. 0(a)) is an event camera that transmits events in addition to frames. Events are pixel-level, relative-brightness changes that are detected in continuous time by specially-designed pixels111Video illustration: https://youtu.be/LauQ6LWTkxM. The events are timestamped with micro-second resolution and transmitted asynchronously at the time they occur. Each event is a tuple , where are the pixel coordinates of the event, is the timestamp of the event, and is the polarity of the event, which is the sign of the brightness change. This representation is sometimes also referred to as Address-Event Representation (AER). The DAVIS has a spatial resolution of pixels. A visualization of the event output is shown in Fig. 0(b). Both the events and frames are generated by the same physical pixels, hence there is no spatial offset between the events and the frames.
Due to its low latency and high temporal resolution, both in the range of micro-seconds, event-based cameras are very promising sensors for high-speed mobile robot applications. Since event cameras are data-driven (only brightness changes are transmitted), no redundant data is transmitted. The required bandwidth thus depends on the motion speed and the type of scene. An additional advantage for robotic applications is the high dynamic range of (compared to of expensive computer-vision cameras), which allows both indoor and outdoor operation without changing parameters. Since all pixels are independent, very large contrast changes can also take place within the same scene.
Over the course of the last seven years, several groups including ours have demonstrated the use of event-based sensors in a variety of tasks, such as SLAM in 2D (Weikersdorfer et al., 2013) and 3D (Kueng et al., 2016; Kim et al., 2016; Rebecq et al., 2016b), optical flow (Cook et al., 2011; Benosman et al., 2014; Bardow et al., 2016), visual odometry (Censi and Scaramuzza, 2014), 6-DOF localization for high-speed robotics (Mueggler et al., 2014), line detection and localization (Yuan and Ramalingam, 2016), 3D reconstruction (Rebecq et al., 2016a), image reconstruction and mosaicing (Kim et al., 2014; Reinbacher et al., 2016), orientation estimation (Gallego and Scaramuzza, 2016), and continuous-time trajectory estimation (Mueggler et al., 2015).
However, all these methods were evaluated on different, specific datasets and, therefore, cannot be compared against each other. The datasets we propose here are tailored to allow comparison of pose tracking, visual odometry, and SLAM algorithms. Since event-based cameras, such as the DAVIS, are currently still expensive ( USD), these data also allow researchers without equipment to use well-calibrated data for their research.
2.1 Davis Imu
In addition to the visual output (events and frames), the DAVIS includes an IMU that provides gyroscope and accelerometer data, thus enabling to design visual-inertial event-based algorithms. The DAVIS camera has the IMU mounted directly behind and centered under the image sensor pixel array center, at a distance of about from it, so that the IMU shares nearly the same position as the event sensor (i.e., the photoreceptor, not the optical center of the camera, since this is lens dependent; the camera-IMU calibration is discussed on page 5.3). The IMU axes are aligned with the visual sensor axes (see Fig. 0(a)). More specifically, the IMU is an InvenSense MPU-6150222IMU data sheet: https://store.invensense.com/ProductDetail/MPU6150-invensense/470090/, which integrates a three-axis gyroscope that can measure in the range and a three-axis accelerometer for the range . It integrates six 16-bit ADCs for digitizing the gyroscope and accelerometer outputs at sample rate.
3 DAVIS Simulator
Simulation offers a good baseline when working with new sensors, such as the DAVIS. Based on the operation principle of an ideal DAVIS pixel, we created a simulator that, given a virtual 3D scene and the trajectory of a moving DAVIS within it, generates the corresponding stream of events, intensity frames, and depth maps. We used the computer graphics software Blender333https://www.blender.org/ to generate thousands of rendered images along the specified trajectory, ensuring that the motion between consecutive images was smaller than pixel. For each pixel, we keep track of the time of the last event triggered at that location. This map of timestamps (also called surface of active events (Benosman et al., 2014)), combined with time interpolation of the rendered image intensities, allows determining brightness changes of predefined amount (given by the contrast threshold) in the time between images, thus effectively providing continuous timestamps, as if events were generated asynchronously. Time interpolation has an additional benefit: it solves the problem of having to generate millions of images for each second of a sequence, as it would have been required to deliver microsecond-resolution timestamps in the absence of interpolation.
More specifically, Fig. 2 illustrates the operation of the simulator for a single pixel . The continuous intensity signal at pixel , (black) is sampled at the times of the rendered images (blue markers). These samples are used to determine the times of the events: the data is linearly interpolated between consecutive samples and the crossings of the resulting lines (in red) with the levels given by multiples of the contrast threshold (i.e., horizontal lines) specify the timestamps of the events (red dots). As it can be observed, this simple interpolation scheme allows for higher resolution event time stamps than those of the rendered images, and the generation of multiple events between two samples if the corresponding intensity jump is larger than the contrast threshold.
The provided events are “perfect” measurements up to sampling and quantization; under this condition, an image can be reconstructed from the event stream at any point in time by accumulating events according to
where is the rendered image at time and selects the pixel to be updated on every event (pixel of is updated at time ). We used this scheme to check that the reconstructed image agreed with the rendered image at several points in time; specifically, the per-pixel intensity error was confined to the quantization interval .
Event generation operates on brightness pixels, which are computed from the rendered color images using the ITU-R Recommendation BT.601444https://www.itu.int/rec/R-REC-BT.601 for luma, i.e., according to formula , with RGB channels in linear color space to better resemble the operation of the DAVIS.
Because realistic event noise is extremely difficult to model due to the complex behavior of event sensors with respect to their bias settings and other factors, the provided simulation datasets do not include event noise. Nevertheless, the simulator, and the datasets created with it, are a useful tool for prototyping new event-based algorithms. Our implementation is available as open-source software.555https://github.com/uzh-rpg/rpg_davis_simulator
In this section, we describe the datasets that we provide. The datasets contain:
the asynchronous event stream,
intensity images at about ,
inertial measurements (3-axis gyroscope and 3-axis accelerometer) at ,
ground-truth camera poses from a motion-capture system666We use an OptiTrack system from NaturalPoint. with sub-millimeter precision at (for the indoor datasets),
the intrinsic camera matrix.
All information comes with precise timestamps. For datasets that were captured outside the motion-capture system (e.g., in an office or outdoors), no ground truth is provided. Some datasets were collected using a motorized linear slider and ground truth was collected using the slider’s position. Due to vibrations induced by the slider motor, the very noisy IMU data was not recorded.
4.1 Data Format
|events.txt||One event per line||timestamp x y polarity|
|images.txt||One image reference per line||timestamp filename|
|images/00000000.png||Images referenced from images.txt|
|imu.txt||One measurement per line||timestamp ax ay az gx gy gz|
|groundtruth.txt||One ground-truth measurement per line||timestamp px py pz qx qy qz qw|
|calib.txt||Camera parameters||fx fy cx cy k1 k2 p1 p2 k3|
The datasets are provided in standard text form that is described here. For convenience, they can also be downloaded as binary rosbag files (the details are on the website). The format of the text files is described in Table 1.
The ground-truth pose is with respect to the (arbitrary) motion-capture origin that has the -axis gravity-aligned (pointing upwards). The orientation is provided as a unit quaternion , where and
are the scalar and vector components, respectively. This convention was proposed as a standard by JPL(Breckenridge, 1979).
All values are reported in SI units. While the timestamps were originally recorded as POSIX, we subtracted the lowest timestamp as offset such that all datasets start at zero. This helps to avoid numerical difficulties when dealing with microsecond resolution timestamps of the events.
Images are provided as PNG files. The list of all images and their timestamps is provided in a separate file. The typical framerate is , but it varies with the exposure time.
The IMU axes have the same orientation as those of the optical coordinate frame (i.e., the positive -axis is aligned with the optical axis and so are the - and -axes).
4.2 List of Datasets
The provided datasets are summarized in Table 2 and Fig. 3. All the datasets contain increasing speeds, different scenes, and varying degrees of freedom777The DAVIS was moved by hand, the dominant motion is described.: for the shapes, poster, and boxes datasets, the motion first starts with excitation of each single degree of freedom separately; then combined and faster excitations are performed. This leads to increasing difficulty and a higher event rate over time.
|Name||Motion||Scene||GT||T ||TS ||RS ||NE [-]|
|shapes_rotation||Rotation, incr. speed||Fig. 2(a)||yes|
|shapes_translation||Translation, incr. speed||Fig. 2(a)||yes|
|shapes_6dof||6 DOF, incr. speed||Fig. 2(a)||yes|
|poster_rotation||Rotation, incr. speed||Fig. 2(b)||yes|
|poster_translation||Translation, incr. speed||Fig. 2(b)||yes|
|poster_6dof||6 DOF, incr. speed||Fig. 2(b)||yes|
|boxes_rotation||Rotation, incr. speed||Fig. 2(c)||yes|
|boxes_translation||Translation, incr. speed||Fig. 2(c)||yes|
|boxes_6dof||6 DOF, incr. speed||Fig. 2(c)||yes|
|hdr_poster||6 DOF, incr. speed||Fig. 2(b)||yes|
|hdr_boxes||6 DOF, incr. speed||Fig. 2(c)||yes|
|outdoors_walking||6 DOF, walking||Fig. 2(d)||no||n/a||n/a|
|outdoors_running||6 DOF, running||Fig. 2(d)||no||n/a||n/a|
|dynamic_rotation||Rotation, incr. speed||Fig. 2(e)||yes|
|dynamic_translation||Translation, incr. speed||Fig. 2(e)||yes|
|dynamic_6dof||6 DOF, incr. speed||Fig. 2(e)||yes|
|calibration||6 DOF, slow||Fig. 2(f)||yes|
|office_zigzag||6-DOF, zigzag, slow||Fig. 2(g)||no||n/a||n/a|
|office_spiral||6-DOF, spiral, slow||Fig. 2(g)||no||n/a||n/a|
|urban||Linear, slow||Fig. 2(h)||no||n/a||n/a|
|slider_close||Linear, const, speed||Fig. 2(i)||yes*|
|slider_far||Linear, const, speed||Fig. 2(i)||yes*|
|slider_hdr_close||Linear, const. speed||Fig. 2(j)||yes*|
|slider_hdr_far||Linear, const. speed||Fig. 2(j)||yes*|
|slider_depth||Linear, const. speed||Fig. 2(k)||yes*|
|simulation_3planes||Translation, circle||Fig. 2(l)||yes#|
|simulation_3walls||6 DOF||Fig. 2(m)||yes#|
In the high-dynamic-range (HDR) sequences (hdr_poster, hdr_boxes, and slider_hdr), a spotlight was used to create large intrascene contrasts. For hdr_poster, we measured and in the dark and bright areas, respectively.
The outdoors datasets were acquired in an urban environment both walking and running. While no ground truth is available, we returned precisely to the same location after a large loop.
The dynamic datasets were collected in a mock-up office environment viewed by the motion-capture system, with a moving person first sitting at a desk, then moving around.
A calibration dataset is also available, for instance in case the user wishes to use a different camera model or different methods for hand-eye calibration. The dimensions of the calibration pattern (a checkerboard) are tiles of . For the lower half of the table, different settings (lenses, focus, etc.) were used. Thus, while we provide the intrinsic calibration, no calibration datasets are available.
The slider_close, slider_far, slider_hdr_close, and slider_hdr_far datasets were recorded with a motorized linear slider parallel to a textured wall at , , , and , respectively.
For the datasets, we applied two different sets of biases (parameters) for the DAVIS, as listed in Table 3. The first set, labeled “indoors”, was used in all datasets but outdoors_walking, outdoors_running, and urban, where the set “outdoors” was applied. For the simulated datasets, we used a contrast threshold of and for the simulation_3planes and simulation_3walls, respectively.
For the simulated scenes, we also provide the 3D world model in Blender (cf. Fig. 2(l) and 2(m)). In addition to the intensity images and events, these datasets include a depth map for each image frame at , encoded as 32-bit floating-point values (in the OpenEXR data format).
First, we calibrated the DAVIS intrinsically using a checkerboard pattern. Then, we computed the hand-eye calibration that we applied to the subsequent dataset recordings so that the ground-truth poses that we provide are those of the event camera (i.e., the “eye”), not those of the motion-capture trackable (i.e., the “hand”) attached to the camera. We also included a calibration dataset in case a different camera model or improved hand-eye calibration method is required.
5.1 Intrinsic Camera Calibration
We used the standard pinhole camera model with radial-tangential distortion using the implementation of ROS and OpenCV888http://wiki.ros.org/camera_calibration/Tutorials/MonocularCalibration. We used three radial distortion coefficients (, , and ) and two for tangential distortion ( and ). The distortion coefficients are listed in calib.txt in the same order as in OpenCV. We provide a dataset for post-calibration in case that another method is preferred.
5.2 Hand-Eye Calibration
For the indoor datasets, we provide accurate and high-frequency () pose data from a motion-capture system. However, the coordinate frame used by the motion-capture system is different from the optical coordinate frame of the DAVIS. Thus, we performed a hand-eye calibration before acquiring the datasets. Fig. 4 shows the coordinate frames and transformations used to solve the hand-eye calibration problem. The frames are those of the world , the hand , the camera (Fig. 0(a)), and the checkerboard . For the transformations, Fig. 4 shows both the compact standard notation of hand-eye calibration problems and a more explicit one: the Euclidean transformation ( homogeneous matrix representation) maps points from frame to frame according to .
More specifically, we first use a linear algorithm (Tsai and Lenz, 1989) to provide an initial solution of the hand-eye calibration problem , where are correspondences of relative hand-hand () and eye-eye () poses at different times ( and ), respectively, and is the unknown eye-to-hand transformation. Then, using the second formulation of hand-eye calibration problems, of the form , where and are the hand-to-motion-capture and eye-to-checkerboard transformations for the -th pose, respectively, we refined by jointly estimating the hand-eye and robot-world (i.e., motion-capture–checkerboard) transformations that minimize the reprojection error in the image plane:
where is the squared Euclidean distance between the measured projection of the -th checkerboard corner on the -th camera and the predicted corner , which is a function of the intrinsic camera parameters and the extrinsic parameters predicted using the motion-capture data. This non-linear least-squares problem is solved iteratively using the Gauss-Newton method. The initial value of is given by , with provided by the above-mentioned linear algorithm. We included a dataset for post-calibration in case another method is preferred.
The ground-truth pose gives the position and orientation of the event camera with respect to the world (i.e., the motion-capture system). Hence, it already incorporates the computed hand-eye transformation. That is, while the motion-capture system outputs , we apply the hand-eye calibration and directly report as ground-truth pose.
5.3 Camera-IMU Calibration
The calibration dataset can be used to compute the Euclidean transformation between the camera and IMU reference frames. Running the publicly available software Kalibr (Furgale et al., 2013) on the calibration dataset provides such a transformation, from the camera (i.e., the “eye” ) to the IMU, given by
that is, the rotation matrix is approximately the identity (i.e., camera and IMU axes have the same orientation) and the translation is dominantly along the optical axis (the IMU is a couple of cm behind the camera’s optical center for the used lens). Additionally, due to the IMU’s built-in low-pass filter, the IMU measurements lag behind the images (and the events). This temporal shift is also reported by Kalibr.
6 Known Issues
6.1 Clock Drift and Offset
The clocks from motion-capture system and the DAVIS are not hardware-synchronized. We observed clock drift of about . To counteract the clock drift, we reset the clocks before each dataset recording. Since all datasets are rather short (in the order of ), the effect of drift is negligible. A small, dataset-dependent offset between the DAVIS and motion-capture timestamps is present since the timestamps were reset in software.
Bardow et al. (2016)
Bardow P, Davison AJ and Leutenegger S (2016) Simultaneous Optical Flow and
Intensity Estimation From an Event Camera.
IEEE Int. Conf. Computer Vision and Pattern Recognition (CVPR).
- Barranco et al. (2016) Barranco F, Fermuller C, Aloimonos Y and Delbruck T (2016) A Dataset for Visual Navigation with Neuromorphic Methods. Frontiers in Neuroscience 10(49).
- Benosman et al. (2014) Benosman R, Clercq C, Lagorce X, Ieng SH and Bartolozzi C (2014) Event-Based Visual Flow. IEEE Trans. Neural Netw. Learn. Syst. 25(2): 407–417.
- Brandli et al. (2014) Brandli C, Berner R, Yang M, Liu SC and Delbruck T (2014) A 240x180 130dB 3us Latency Global Shutter Spatiotemporal Vision Sensor. IEEE J. of Solid-State Circuits 49(10): 2333–2341.
- Breckenridge (1979) Breckenridge WG (1979) Quaternions proposed standard conventions. Technical report, NASA Jet Propulsion Laboratory.
- Censi and Scaramuzza (2014) Censi A and Scaramuzza D (2014) Low-Latency Event-Based Visual Odometry. In: IEEE Int. Conf. on Robotics and Automation (ICRA).
Cook et al. (2011)
Cook M, Gugelmann L, Jug F, Krautz C and Steger A (2011) Interacting Maps for
Fast Visual Interpretation.
Int. Joint Conf. on Neural Networks (IJCNN). pp. 770–776.
- Delbruck et al. (2014) Delbruck T, Villanueva V and Longinotti L (2014) Integration of Dynamic Vision Sensor with Inertial Measurement Unit for Electronically Stabilized Event-Based Vision. In: Int. Conf. on Circuits and Systems (ISCAS). pp. 2636–2639.
- Furgale et al. (2013) Furgale P, Rehder J and Siegwart R (2013) Unified temporal and spatial calibration for multi-sensor systems. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS).
- Gallego and Scaramuzza (2016) Gallego G and Scaramuzza D (2016) Accurate Angular Velocity Estimation with an Event Camera. IEEE Robotics and Automation Letters .
- Kim et al. (2014) Kim H, Handa A, Benosman R, Ieng SH and Davison AJ (2014) Simultaneous Mosaicing and Tracking with an Event Camera. In: British Machine Vision Conf. (BMVC).
- Kim et al. (2016) Kim H, Leutenegger S and Davison A (2016) Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera. In: Eur. Conf. on Computer Vision (ECCV).
- Kueng et al. (2016) Kueng B, Mueggler E, Gallego G and Scaramuzza D (2016) Low-Latency Visual Odometry using Event-based Feature Tracks. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS).
- Mueggler et al. (2015) Mueggler E, Gallego G and Scaramuzza D (2015) Continuous-Time Trajectory Estimation for Event-based Vision Sensors. In: Robotics: Science and Systems (RSS).
- Mueggler et al. (2014) Mueggler E, Huber B and Scaramuzza D (2014) Event-based, 6-DOF Pose Tracking for High-Speed Maneuvers. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS).
- Rebecq et al. (2016a) Rebecq H, Gallego G and Scaramuzza D (2016a) EMVS: Event-based Multi-View Stereo. In: British Machine Vision Conf. (BMVC).
- Rebecq et al. (2016b) Rebecq H, Horstschäfer T, Gallego G and Scaramuzza D (2016b) EVO: A Geometric Approach to Event-based 6-DOF Parallel Tracking and Mapping in Real-Time. IEEE Robotics and Automation Letters .
- Reinbacher et al. (2016) Reinbacher C, Graber G and Pock T (2016) Real-Time Intensity-Image Reconstruction for Event Cameras Using Manifold Regularisation. In: British Machine Vision Conf. (BMVC).
- Rueckauer and Delbruck (2016) Rueckauer B and Delbruck T (2016) Evaluation of Event-Based Algorithms for Optical Flow with Ground-Truth from Inertial Measurement Sensor. Frontiers in Neuroscience 10(176).
- Tsai and Lenz (1989) Tsai RY and Lenz RK (1989) A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration. IEEE Trans. Robotics 5(3): 345–358.
- Weikersdorfer et al. (2013) Weikersdorfer D, Hoffmann R and Conradt J (2013) Simultaneous Localization and Mapping for event-based Vision Systems. In: Int. Conf. on Computer Vision Systems (ICVS).
- Yuan and Ramalingam (2016) Yuan W and Ramalingam S (2016) Fast Localization and Tracking using Event Sensors. In: IEEE Int. Conf. on Robotics and Automation (ICRA).