[Paper Review] Deep Contextual Video Compression

Jiahao Li, Bin Li+1 more|arXiv (Cornell University)|Sep 30, 2021

Advanced Image Processing TechniquesComputer Science37 references115 citations

TL;DR

The paper introduces DCVC, a conditional coding framework that uses high-dimensional feature-domain context learned via MEMC in the context of video compression, achieving substantial bitrate savings over prior DL-based methods and x265. It jointly optimizes encoding, decoding, and entropy modeling using learnable context.

ABSTRACT

Most of the existing neural video compression methods adopt the predictive coding framework, which first generates the predicted frame and then encodes its residue with the current frame. However, as for compression ratio, predictive coding is only a sub-optimal solution as it uses simple subtraction operation to remove the redundancy across frames. In this paper, we propose a deep contextual video compression framework to enable a paradigm shift from predictive coding to conditional coding. In particular, we try to answer the following questions: how to define, use, and learn condition under a deep video compression framework. To tap the potential of conditional coding, we propose using feature domain context as condition. This enables us to leverage the high dimension context to carry rich information to both the encoder and the decoder, which helps reconstruct the high-frequency contents for higher video quality. Our framework is also extensible, in which the condition can be flexibly designed. Experiments show that our method can significantly outperform the previous state-of-the-art (SOTA) deep video compression methods. When compared with x265 using veryslow preset, we can achieve 26.0% bitrate saving for 1080P standard test videos.

Motivation & Objective

• Define and learn a learnable context (condition) for video compression beyond residue coding.
• Design a conditioning framework that influences encoder, decoder, and entropy model in a unified way.
• Leverage feature-domain motion-compensated context to improve reconstruction, especially high-frequency contents.
• Demonstrate that conditional coding with temporal priors surpasses residue-based approaches in rate-distortion performance.

Proposed method

• Propose a conditional coding-based framework where the current frame is encoded conditioned on a learned context  bar{x}_t derived from the previous decoded frame via feature-domain MEMC.
• Define context as high-dimensional feature-domain information rather than a pixel-domain prediction; enable the context to feed encoder, decoder, and entropy model.
• Use a temporal prior in the entropy model along with a hyper-prior and autoregressive components to estimate latent code distributions and bitrate.
• Learn the context by warping a feature-extracted representation of the previous frame using motion vectors learned via MEMC, followed by a refinement network to produce  bar{x}_t.
• Train with a rate-distortion objective L = Lambda * D + R, where D is distortion (MSE or MS-SSIM) and R is cross-entropy-based bitrate.

Experimental results

Research questions

• RQ1 Can a learnable, high-dimensional context improve inter-frame coding beyond simple residue subtraction?
• RQ2 How can MEMC be integrated in feature space to guide context extraction for better compression?
• RQ3 What entropy-model architecture (hyper prior, autoregressive, temporal prior) yields the best rate-distortion performance for conditional coding?
• RQ4 Does temporal priors enable faster, more parallelizable entropy coding without sacrificing compression gains?
• RQ5 How does DCVC compare to state-of-the-art DL-based codecs and traditional codecs across resolutions and content types?

Key findings

• DCVC achieves substantial bitrate savings over prior DL-based codecs and x265 veryslow, e.g., 26.0% on 1080p standard test videos vs x265 veryslow.
• DCVC outperforms DVCPro across tested datasets and bitrates, with BD-Bitrate improvements of up to 26.0% PSNR on MCL-JCV and UVG (1080p).
• Higher gains are observed on high-resolution videos, where feature-domain context carries richer information for high-frequency content.
• A temporal prior-based entropy model with or without spatial priors offers competitive or superior performance, with the best results when combining hyper prior, temporal prior, and optional spatial prior.
• Ablation shows that concatenating context features provides larger gains than RGB-prediction conditioning, and temporal prior improves results, especially under conditional coding.