[Paper Review] DDD17: End-To-End DAVIS Driving Dataset

Jonathan Binas, Daniel Neil+2 more|arXiv (Cornell University)|Nov 4, 2017

CCD and CMOS Imaging Sensors98 citations

TL;DR

The paper introduces DDD17, the first open DAVIS driving dataset with synchronized APS and DVS data plus vehicle diagnostics, and provides an initial end-to-end steering prediction study using CNNs on APS and DVS inputs.

ABSTRACT

Event cameras, such as dynamic vision sensors (DVS), and dynamic and active-pixel vision sensors (DAVIS) can supplement other autonomous driving sensors by providing a concurrent stream of standard active pixel sensor (APS) images and DVS temporal contrast events. The APS stream is a sequence of standard grayscale global-shutter image sensor frames. The DVS events represent brightness changes occurring at a particular moment, with a jitter of about a millisecond under most lighting conditions. They have a dynamic range of >120 dB and effective frame rates >1 kHz at data rates comparable to 30 fps (frames/second) image sensors. To overcome some of the limitations of current image acquisition technology, we investigate in this work the use of the combined DVS and APS streams in end-to-end driving applications. The dataset DDD17 accompanying this paper is the first open dataset of annotated DAVIS driving recordings. DDD17 has over 12 h of a 346x260 pixel DAVIS sensor recording highway and city driving in daytime, evening, night, dry and wet weather conditions, along with vehicle speed, GPS position, driver steering, throttle, and brake captured from the car's on-board diagnostics interface. As an example application, we performed a preliminary end-to-end learning study of using a convolutional neural network that is trained to predict the instantaneous steering angle from DVS and APS visual data.

Motivation & Objective

• Introduce DDD17, the first open DAVIS driving dataset with end-to-end labeling for driving scenarios.
• Demonstrate data synchronization and recording software for APS, DVS, and vehicle diagnostics.
• Provide a preliminary end-to-end steering angle prediction study using CNNs on APS and DVS data.

Proposed method

• Use a DAVIS346B camera to capture synchronized APS and DVS data at 346x260 resolution.
• Record vehicle control and diagnostics data via the OpenXC interface at ~10 Hz for multiple sensors.
• Develop ddd17-utils for synchronized, event-driven data storage in HDF5 and DAVIS cAER formats.
• Train a CNN with 4 conv layers (8 feature maps each, 3x3 kernels) to predict steering angle from 80x60 subsampled APS/DVS inputs.
• Overlay and compare APS vs DVS inputs to assess end-to-end learning usability.

Experimental results

Research questions

• RQ1 Can APS and DVS streams be effectively fused for end-to-end driving tasks such as steering angle prediction?
• RQ2 What is the usability of the DDD17 dataset and tooling for training CNN-based steering models?
• RQ3 How does end-to-end learning performance on DDD17 compare between APS-only and DVS-only inputs under varied driving conditions?
• RQ4 What limitations exist in using DDD17 for learning-based driving due to sensor modality and data balance?

Key findings

• DDD17 provides over 12 hours of DAVIS driving data under diverse conditions across 1000+ km.
• The dataset includes synchronized APS, DVS, and vehicle diagnostics stored in HDF5 with per-stream timestamps.
• Preliminary steering prediction experiments with a CNN show usability of the data and tools, though quantitative results are inconclusive.
• A single 1.5-hour recording was used to compare APS and DVS inputs for steering prediction.
• The paper demonstrates the feasibility of end-to-end learning on DAVIS data and releases accompanying software for data handling.