[Paper Review] CoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics

Jinwoo Hwang, Minsu Kim+6 more|arXiv (Cornell University)|Jul 2, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

CoVA proposes a novel cascade architecture that accelerates video analytics by performing object tracking and frame selection in the compressed domain, reducing decoding overhead while supporting both temporal and spatial queries. It achieves 4.8× throughput improvement over state-of-the-art systems with only modest accuracy loss by leveraging compression metadata for blob detection and selective decoding of representative frames.

ABSTRACT

Modern retrospective analytics systems leverage cascade architecture to mitigate bottleneck for computing deep neural networks (DNNs). However, the existing cascades suffer two limitations: (1) decoding bottleneck is either neglected or circumvented, paying significant compute and storage cost for pre-processing; and (2) the systems are specialized for temporal queries and lack spatial query support. This paper presents CoVA, a novel cascade architecture that splits the cascade computation between compressed domain and pixel domain to address the decoding bottleneck, supporting both temporal and spatial queries. CoVA cascades analysis into three major stages where the first two stages are performed in compressed domain while the last one in pixel domain. First, CoVA detects occurrences of moving objects (called blobs) over a set of compressed frames (called tracks). Then, using the track results, CoVA prudently selects a minimal set of frames to obtain the label information and only decode them to compute the full DNNs, alleviating the decoding bottleneck. Lastly, CoVA associates tracks with labels to produce the final analysis results on which users can process both temporal and spatial queries. Our experiments demonstrate that CoVA offers 4.8x throughput improvement over modern cascade systems, while imposing modest accuracy loss.

Motivation & Objective

• Address the decoding bottleneck in video analytics systems that arises from processing full video decompression before DNN inference.
• Support both temporal and spatial queries in retrospective video analytics, which existing cascade systems often exclude.
• Enable efficient, query-agnostic analysis by offloading early processing stages to the compressed domain without pre-processing or transcoding.
• Minimize compute and storage costs by selecting only a minimal set of frames for full DNN inference.
• Maintain high accuracy while drastically reducing the number of decoded frames through intelligent label propagation.

Proposed method

• CoVA splits video analytics into three stages: compressed-domain blob tracking, minimal frame selection, and pixel-domain DNN inference with label propagation.
• It uses a refitted neural network for compressed-domain blob tracking, leveraging encoding metadata such as motion vectors and quantization parameters.
• Frame selection is guided by spatiotemporal coherence, selecting representative frames that maximize coverage with minimal decoding cost.
• Labels from the selected frames are propagated across the entire track using motion and spatial consistency from the compressed-domain analysis.
• The system operates on standard video codecs (e.g., H.264/265) without requiring pre-processing, transcoding, or custom storage formats.
• The final analysis results are stored once per video and reused for all subsequent queries, enabling fast response times.

Experimental results

Research questions

• RQ1 Can compressed-domain analysis reduce the decoding bottleneck in video analytics without pre-processing or transcoding?
• RQ2 Can a cascade architecture support both temporal and spatial queries while maintaining high throughput?
• RQ3 How effective is compressed-domain blob tracking using only encoding metadata for object detection?
• RQ4 What is the trade-off between decoding cost reduction and accuracy loss in a query-agnostic video analytics pipeline?
• RQ5 Can selective decoding of representative frames preserve accuracy while minimizing computational load?

Key findings

• CoVA reduces the decoding workload by 83.6% compared to state-of-the-art systems by processing most of the pipeline in the compressed domain.
• The system achieves 4.8× higher throughput than existing cascade-based video analytics systems, significantly accelerating query processing.
• The accuracy loss introduced by compressed-domain tracking and selective decoding is modest, with minimal impact on final classification and localization results.
• CoVA supports both temporal queries (e.g., when an object appears) and spatial queries (e.g., object in upper right region), which prior systems often exclude.
• The system's query-agnostic results are stored once per video, enabling fast response to future queries without reprocessing.
• The approach is compatible with standard video codecs and does not require changes to video ingestion or storage pipelines.