[Paper Review] Conditional Entropy Coding for Efficient Video Compression

Jerry Liu, Shenlong Wang+5 more|arXiv (Cornell University)|Aug 20, 2020

Advanced Image Processing Techniques35 references4 citations

TL;DR

This paper proposes a lightweight, entropy-focused video compression framework that models conditional entropy between independent latent codes from a deep image compressor, achieving competitive performance with H.265 and state-of-the-art deep learning baselines while being thousands of times faster in decoding. It further introduces internal learning during inference to improve rate-distortion trade-off by ~10% without increasing decoding time.

ABSTRACT

We propose a very simple and efficient video compression framework that only focuses on modeling the conditional entropy between frames. Unlike prior learning-based approaches, we reduce complexity by not performing any form of explicit transformations between frames and assume each frame is encoded with an independent state-of-the-art deep image compressor. We first show that a simple architecture modeling the entropy between the image latent codes is as competitive as other neural video compression works and video codecs while being much faster and easier to implement. We then propose a novel internal learning extension on top of this architecture that brings an additional 10% bitrate savings without trading off decoding speed. Importantly, we show that our approach outperforms H.265 and other deep learning baselines in MS-SSIM on higher bitrate UVG video, and against all video codecs on lower framerates, while being thousands of times faster in decoding than deep models utilizing an autoregressive entropy model.

Motivation & Objective

• Address the high computational cost and slow decoding speeds of existing deep learning-based video compression methods.
• Reduce complexity by avoiding explicit frame transformations like motion compensation or residual coding.
• Improve compression efficiency by modeling inter-frame correlations solely through conditional entropy on latent codes.
• Enable faster, more practical deployment of learned video compression in real-world systems like self-driving cars and video streaming platforms.

Proposed method

• Use a pre-trained deep image compressor to independently encode each video frame into a latent code, bypassing frame-level transformations.
• Train a conditional entropy model that estimates the probability of the second frame’s latent code given the first, reducing joint bitrate via entropy coding.
• Apply the conditional entropy model to adjacent frame pairs in a sequence, enabling end-to-end video compression without explicit motion compensation.
• Introduce internal learning during inference: optimize latent codes iteratively to improve rate-distortion performance while keeping decoding speed unchanged.
• Use standard backpropagation with PyTorch’s autograd for internal learning, though the implementation is unoptimized.
• Evaluate the model on UVG, CDVL, and NorthAmerica datasets across varying framerates and bitrates to assess robustness and performance.

Experimental results

Research questions

• RQ1 Can a video compression system based solely on conditional entropy modeling between independent frame latent codes outperform traditional and deep learning-based codecs?
• RQ2 Does eliminating explicit motion compensation and residual coding lead to a significant speedup in decoding without sacrificing compression efficiency?
• RQ3 Can internal learning of latent codes during inference improve rate-distortion performance without increasing decoding time?
• RQ4 How does the proposed method perform relative to H.265 and other deep learning baselines at low framerates and high bitrates?
• RQ5 To what extent is the method robust across diverse video content and framerates, especially in scenarios where traditional codecs struggle?

Key findings

• The base conditional entropy model achieves performance competitive with H.265 and state-of-the-art deep learning video codecs in MS-SSIM, especially at higher bitrates on the UVG dataset.
• At lower framerates (e.g., 12 Hz on UVG and 6 Hz on CDVL), the method outperforms H.265 by a wide margin in MS-SSIM, even when using only single-image compression as baseline.
• The conditional entropy model reduces bitrate by 20–50% compared to single-image compression across all tested framerates and datasets.
• Internal learning provides an additional ~10% bitrate saving across all bitrates and settings, with no increase in decoding time.
• Decoding is thousands of times faster than autoregressive entropy models, making the method practical for real-time applications.
• On the NorthAmerica dataset, the model outperforms H.265 in MS-SSIM at higher bitrates and maintains superiority when the ego-vehicle is in motion.