[Paper Review] Event Probability Mask (EPM) and Event Denoising Convolutional Neural Network (EDnCNN) for Neuromorphic Cameras

R. Wes Baldwin, Mohammed Almatrafi+2 more|arXiv (Cornell University)|Mar 18, 2020

Advanced Memory and Neural Computing47 references4 citations

TL;DR

This paper introduces the Event Probability Mask (EPM), a spatial-temporal label for real-world neuromorphic camera events that estimates the likelihood of an event occurring per pixel based on APS intensity gradients and IMU motion data. It proposes EDnCNN, a CNN-based denoising model trained to minimize a new benchmark metric (RPMD), achieving a 148-point average RPMD improvement over noisy data and outperforming prior methods in 12 out of 16 scenes with statistical significance (p = 0.00248).

ABSTRACT

This paper presents a novel method for labeling real-world neuromorphic camera sensor data by calculating the likelihood of generating an event at each pixel within a short time window, which we refer to as "event probability mask" or EPM. Its applications include (i) objective benchmarking of event denoising performance, (ii) training convolutional neural networks for noise removal called "event denoising convolutional neural network" (EDnCNN), and (iii) estimating internal neuromorphic camera parameters. We provide the first dataset (DVSNOISE20) of real-world labeled neuromorphic camera events for noise removal.

Motivation & Objective

• To address the lack of reliable benchmarking for event denoising in real-world neuromorphic camera data due to unknown noise distributions.
• To enable machine learning-based denoising by creating a large-scale, real-world labeled dataset (DVSNOISE20) with event probability masks (EPM).
• To develop a robust, data-driven method for denoising DVS events using deep learning, trained to align with EPM as a proxy for noise-free events.
• To calibrate internal neuromorphic camera parameters (e.g., threshold values) using maximum likelihood estimation based on EPM predictions.
• To establish a new objective benchmarking metric, Relative Plausibility Measure of Denoising (RPMD), for quantitative evaluation of denoising performance.

Proposed method

• EPM is computed by combining spatial intensity gradients from APS frames and motion data from an IMU to estimate the log-likelihood of event generation at each pixel over short time windows.
• The EPM serves as a proxy for noise-free DVS events, enabling supervised training of denoising models despite the absence of ground truth for real-world events.
• EDnCNN is a two-stream convolutional neural network that learns to classify events as noise or signal by comparing input events to the EPM label.
• The model is trained to minimize the RPMD metric, which quantifies the plausibility of denoising results relative to the EPM.
• Maximum likelihood estimation is used to calibrate internal DVS threshold parameters by fitting the EPM model to observed event data.
• The DVSNOISE20 dataset is collected using a DAVIS camera with synchronized APS, DVS, and IMU sensors, and includes EPM labels for 16 real-world scenes.

Experimental results

Research questions

• RQ1 How can we objectively benchmark event denoising performance in real-world neuromorphic camera data when ground truth is unavailable?
• RQ2 Can we create a reliable, data-driven label (EPM) that predicts the likelihood of event generation based on observable sensor data?
• RQ3 To what extent can a deep learning model (EDnCNN) trained on EPM labels improve denoising performance compared to existing hand-crafted or heuristic methods?
• RQ4 How robust is the proposed method to variations in scene content, motion, and noise levels across diverse real-world conditions?
• RQ5 Can internal neuromorphic camera parameters be accurately estimated from natural scene data using the EPM framework?

Key findings

• EDnCNN achieved an average RPMD improvement of 148 points over raw noisy data, demonstrating significant denoising performance gains.
• EDnCNN outperformed all baseline methods (FSAE, IE+TE, BAF, NN, NN2) in 12 out of 16 scenes, with a statistically significant p-value of 0.00248 via the Wilcoxon signed-rank test.
• The method showed robustness to noise levels in simulated data, maintaining high performance even as background activity noise increased.
• EDnCNN produced qualitatively superior results, preserving edge structures and removing isolated noise events more effectively than methods like IE+TE or BAF.
• The EPM-based calibration method successfully estimated internal DVS threshold values, validating the model’s consistency with physical sensor behavior.
• The DVSNOISE20 dataset, with EPM labels and RPMD benchmarking, provides a new standard for evaluating and comparing future neuromorphic denoising algorithms.