[Paper Review] A Compressive Sensing Video dataset using Pixel-wise coded exposure

Sathyaprakash Narayanan, Yeshwanth Bethi+1 more|arXiv (Cornell University)|May 24, 2019

Sparse and Compressive Sensing Techniques9 references4 citations

TL;DR

This paper introduces the first comprehensive video dataset compressed via pixel-wise coded exposure for compressive sensing, enabling efficient storage and processing of video data while preserving object tracking and localization. By merging K frames into a single compressed frame using sparsity and incoherence principles, the dataset supports end-to-end research in object detection, tracking, and reconstruction directly on compressed data with minimal human labeling via YOLO-based pseudo-labeling.

ABSTRACT

Manifold amount of video data gets generated every minute as we read this document, ranging from surveillance to broadcasting purposes. There are two roadblocks that restrain us from using this data as such, first being the storage which restricts us from only storing the information based on the hardware constraints. Secondly, the computation required to process this data is highly expensive which makes it infeasible to work on them. Compressive sensing(CS)[2] is a signal process technique[11], through optimization, the sparsity of a signal can be exploited to recover it from far fewer samples than required by the Shannon-Nyquist sampling theorem. There are two conditions under which recovery is possible. The first one is sparsity which requires the signal to be sparse in some domain. The second one is incoherence which is applied through the isometric property which is sufficient for sparse signals[9][10]. To sustain these characteristics, preserving all attributes in the uncompressed domain would help any kind of in this field. However, existing dataset fallback in terms of continuous tracking of all the object present in the scene, very few video datasets have comprehensive continuous tracking of objects. To address these problems collectively, in this work we propose a new comprehensive video dataset, where the data is compressed using pixel-wise coded exposure [3] that resolves various other impediments.

Motivation & Objective

• To address the growing challenge of storing and processing massive video data due to high storage and computational costs.
• To overcome the limitations of existing video datasets, which lack continuous object tracking and comprehensive labeling across all objects in scenes.
• To create a new benchmark dataset for compressive sensing in video, supporting direct processing on compressed frames without full reconstruction.
• To enable privacy-preserving video analysis by embedding motion information in a way that obscures content from unauthorized users.
• To establish a foundation for future research in low-complexity, high-efficiency video processing using compressive sensing.

Proposed method

• The dataset is created by capturing monochrome videos of single objects (persons and cars) under varying motion conditions: static object with moving background, moving object with static background, and both moving.
• Pixel-wise coded exposure is used to compress K consecutive frames into a single compressed frame, leveraging signal sparsity and incoherence for reconstruction.
• A pre-trained YOLO v3 model is used to generate initial bounding boxes for each frame, which are then merged into larger bounding boxes in the compressed domain using min/max coordinate pooling.
• Pseudo-labeling is employed to reduce human annotation effort: YOLO-generated boxes are manually corrected in VATIC to produce high-quality ground truth for the compressed frames.
• Data is stored in NPZ format for efficient array storage and in JSON format for structured label metadata, including class ID, frame number, and bounding box coordinates.
• The dataset includes 284 car clips and 91 person clips totaling ~90 minutes of footage with diverse aspect ratios and motion dynamics.

Experimental results

Research questions

• RQ1 Can a video dataset compressed via pixel-wise coded exposure support accurate object detection and tracking without full reconstruction?
• RQ2 How effective is pseudo-labeling using YOLO v3 for generating ground truth in the compressed sensing domain?
• RQ3 To what extent does pixel-wise coded exposure preserve spatial and temporal information necessary for object localization in compressed frames?
• RQ4 Can a compressive sensing-based video dataset reduce computational and storage costs while maintaining utility for downstream tasks like tracking and recognition?
• RQ5 How does the inclusion of diverse motion patterns (object and background movement) affect the performance of algorithms operating directly on compressed frames?

Key findings

• The dataset comprises 284 car clips and 91 person clips, totaling approximately 90 minutes of monochrome video with diverse motion dynamics and aspect ratios.
• Pseudo-labeling using YOLO v3 significantly reduced human annotation effort, with human validation in VATIC improving box tightness and detecting missed objects.
• The merging of bounding boxes across K frames into a single larger box in the compressed frame enables consistent localization and tracking in the CS domain.
• The use of pixel-wise coded exposure enables a compression rate K that reduces storage and processing needs by a factor of K, while preserving motion and object structure.
• The dataset is the first dedicated to compressive sensing in video, offering a foundation for research in low-complexity, privacy-preserving video analysis.
• Future releases will include higher compression rates, denser videos, and additional annotations such as segmentation and pose estimation.