[Paper Review] A Coding Framework and Benchmark towards Low-Bitrate Video Understanding

Yuan Tian, Lu Guo+4 more|arXiv (Cornell University)|Feb 6, 2022

Advanced Image Processing Techniques4 citations

TL;DR

This paper proposes the Understanding-oriented Video Coding (UVC) framework, a dual-stream video compression system that enhances downstream video understanding tasks by transmitting a learnable analytic bitstream alongside standard compressed video. Using self-supervised contrastive learning on edge maps and adaptive, dynamic network architectures, UVC significantly improves action recognition, detection, and tracking performance across seven datasets and four compression levels, outperforming industrial codecs like H.264 and H.265 with minimal bit cost increase.

ABSTRACT

Video compression is indispensable to most video analysis systems. Despite saving transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Finally, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be available at \url{https://github.com/tianyuan168326/VCS-Pytorch}.

Motivation & Objective

• To address the degradation of video understanding performance caused by standard video compression, which discards semantics-critical information.
• To design a video coding framework that preserves information essential for downstream tasks without sacrificing compression efficiency.
• To enable a single, one-time training of the framework for deployment across multiple video understanding tasks.
• To build a comprehensive benchmark for evaluating compressed video understanding across diverse datasets and compression levels.

Proposed method

• Introduces a dual-stream architecture: a standard industrial video bitstream and a learnable analytic bitstream for downstream task enhancement.
• Employs self-supervised contrastive learning on neural edge maps as a task-agnostic surrogate optimization target to avoid joint training with downstream models.
• Designs an adaptive encoder network (Enc-Net) with dynamic convolution kernels and a difference map pathway to reduce redundancy and bit cost.
• Uses an attention mechanism to skip features already well-captured by the video stream, improving compression efficiency.
• Employs a recipient network (R-Net) with adaptive kernel indexing to reconstruct the analytic stream efficiently at the decoder.
• Supports flexible extension by integrating advanced components like Transformers or hyperprior models, and can be upgraded to H.265 without retraining.

Experimental results

Research questions

• RQ1 Can a learnable analytic bitstream be effectively used to preserve semantics-critical information for video understanding in compressed videos?
• RQ2 How can the analytic stream be optimized without access to downstream models or labeled data?
• RQ3 To what extent can adaptive and dynamic network architectures reduce the bit cost of the analytic stream while maintaining performance?
• RQ4 How does the UVC framework generalize across diverse video understanding tasks and datasets under varying compression levels?

Key findings

• UVC achieves a 14.47% absolute improvement in Top-1 accuracy and 8.72% in Top-5 accuracy over H.264 baseline when using H.265 as the video codec, demonstrating strong performance gains.
• The ablation study shows that removing the difference map pathway increases the analytic stream bit cost by ~35%, confirming its role in reducing redundancy.
• Replacing adaptive convolution with fixed or plain convolutions reduces performance significantly, proving the value of input-adaptive kernel learning in Enc-Net.
• The edge-side computational cost of Enc-Net is 20× lower than Balle et al. [95], making it suitable for deployment on resource-constrained devices.
• The 0-bit model variant achieves superior performance without adding any computational cost to the edge, proving the framework's efficiency.
• UVC consistently outperforms industrial codecs across all seven large-scale datasets and four compression levels on action recognition, action detection, and multiple object tracking tasks.