[Paper Review] Spike Camera and Its Coding Methods

Siwei Dong, Tiejun Huang+1 more|arXiv (Cornell University)|Apr 10, 2021

Advanced Optical Imaging Technologies7 references4 citations

TL;DR

This paper introduces a spike camera that captures high-speed motion by generating spike streams based on luminance intensity accumulation at each pixel, firing a spike when a threshold is exceeded. It proposes two decoding methods to reconstruct textures from the spike stream, enabling accurate playback of fast-moving scenes with reduced data rates and high temporal resolution.

ABSTRACT

This paper introduces a spike camera with a distinct video capture scheme and proposes two methods of decoding the spike stream for texture reconstruction. The spike camera captures light and accumulates the converted luminance intensity at each pixel. A spike is fired when the accumulated intensity exceeds the dispatch threshold. The spike stream generated by the camera indicates the luminance variation. Analyzing the patterns of the spike stream makes it possible to reconstruct the picture of any moment which enables the playback of high speed movement.

Motivation & Objective

• Address the challenge of capturing and reconstructing high-speed video with conventional cameras limited by frame rate and data bandwidth.
• Develop a novel video capture mechanism that uses event-based spike generation to efficiently record rapid luminance changes.
• Enable accurate reconstruction of visual content from sparse spike streams to support high-speed motion playback.
• Minimize data transmission and storage requirements while preserving temporal and spatial fidelity of dynamic scenes.

Proposed method

• The spike camera accumulates light intensity at each pixel over time, generating a spike when the intensity exceeds a predefined threshold.
• Spike generation is event-driven, producing a stream of timestamped events that encode luminance variation rather than full-frame snapshots.
• Two decoding methods are proposed: one based on temporal integration of spike counts and another using spatiotemporal interpolation to reconstruct pixel intensities.
• The reconstruction process leverages the statistical patterns in spike timing to infer the original luminance distribution across frames.
• The system models pixel intensity changes as a sequence of threshold-crossing events, enabling efficient encoding of dynamic scenes.
• The decoding algorithms are designed to be computationally lightweight, suitable for real-time processing on edge devices.

Experimental results

Research questions

• RQ1 Can an event-based camera system based on luminance thresholding effectively capture high-speed motion with minimal data?
• RQ2 How accurately can texture and visual content be reconstructed from sparse spike streams compared to conventional video?
• RQ3 What decoding strategies maximize reconstruction fidelity while minimizing computational cost?
• RQ4 To what extent does the spike-based coding method reduce data volume without sacrificing temporal resolution?
• RQ5 How does the system perform in reconstructing fast-moving scenes with high dynamic range or rapid intensity changes?

Key findings

• The spike camera achieves high temporal resolution by capturing luminance changes through event-driven spike generation, enabling reconstruction of fast motion.
• The proposed decoding methods successfully reconstruct visual textures from spike streams with minimal artifacts and high fidelity.
• The system reduces data volume significantly compared to traditional video, as only spike events are transmitted rather than full frames.
• Reconstruction quality is preserved even at low spike rates, demonstrating robustness to sparse sampling.
• The decoding process enables playback of high-speed motion sequences with accurate temporal alignment and visual coherence.
• The method demonstrates feasibility for real-time applications in surveillance, robotics, and high-speed imaging.