[Paper Review] Improving Deep Video Compression by Resolution-adaptive Flow Coding

Zhihao Hu, Zhenghao Chen+4 more|arXiv (Cornell University)|Sep 13, 2020

Advanced Vision and Imaging32 references5 citations

TL;DR

This paper proposes Resolution-adaptive Flow Coding (RaFC), a novel framework for deep video compression that adaptively selects optimal resolution representations for optical flow maps at both frame-level (RaFC-frame) and block-level (RaFC-block) using rate-distortion optimization. By dynamically choosing high- or low-resolution flow maps and motion features based on motion complexity, RaFC achieves state-of-the-art performance, reducing motion coding bits by up to 70% compared to baseline DVC methods while maintaining high reconstruction quality.

ABSTRACT

In the learning based video compression approaches, it is an essential issue to compress pixel-level optical flow maps by developing new motion vector (MV) encoders. In this work, we propose a new framework called Resolution-adaptive Flow Coding (RaFC) to effectively compress the flow maps globally and locally, in which we use multi-resolution representations instead of single-resolution representations for both the input flow maps and the output motion features of the MV encoder. To handle complex or simple motion patterns globally, our frame-level scheme RaFC-frame automatically decides the optimal flow map resolution for each video frame. To cope different types of motion patterns locally, our block-level scheme called RaFC-block can also select the optimal resolution for each local block of motion features. In addition, the rate-distortion criterion is applied to both RaFC-frame and RaFC-block and select the optimal motion coding mode for effective flow coding. Comprehensive experiments on four benchmark datasets HEVC, VTL, UVG and MCL-JCV clearly demonstrate the effectiveness of our overall RaFC framework after combing RaFC-frame and RaFC-block for video compression.

Motivation & Objective

• To address the limitation of single-resolution flow representation in deep video compression, which fails to adapt to varying motion complexity across frames.
• To improve motion information compression by enabling dynamic resolution selection for both global (frame-level) and local (block-level) motion features.
• To optimize the rate-distortion trade-off in motion coding by automatically selecting the most efficient resolution for each frame or block.
• To achieve superior video compression performance compared to existing deep learning-based methods like DVC, H.264, and H.265.

Proposed method

• Proposes RaFC-frame, a frame-level scheme that selects the optimal resolution for the entire flow map based on motion complexity and rate-distortion (RD) criterion.
• Introduces RaFC-block, a block-level scheme that independently selects resolution for each local block of motion features to match local motion patterns.
• Employs multi-resolution representations for both input optical flow maps and output motion features to enable resolution adaptation.
• Uses a rate-distortion optimization framework to determine the best resolution choice at both frame and block levels, minimizing bit cost while preserving quality.
• Combines RaFC-frame and RaFC-block into a unified RaFC framework that jointly optimizes resolution selection across the entire video sequence.
• Applies the RD-optimized resolution selection to both motion vector encoding and residual coding, improving overall compression efficiency.

Experimental results

Research questions

• RQ1 Can multi-resolution flow representation improve motion coding efficiency in deep video compression?
• RQ2 Does frame-level resolution adaptation based on global motion complexity lead to better rate-distortion performance?
• RQ3 Can block-level resolution selection based on local motion patterns further enhance compression efficiency?
• RQ4 How does the combination of frame-level and block-level resolution adaptation compare to single-resolution baselines?
• RQ5 To what extent can resolution adaptation reduce motion coding bit usage without sacrificing visual quality?

Key findings

• The overall RaFC framework outperforms H.264, H.265, and the baseline DVC method on four benchmark datasets: HEVC Class E, VTL, UVG, and MCL-JCV.
• RaFC reduces motion coding bits by up to 70% at the same PSNR compared to the baseline DVC method, significantly improving motion compression efficiency.
• At low bit rates (low λ), the method favors low-resolution flow maps and large blocks, consistent with traditional compression strategies for bit savings.
• At high bit rates (high λ), the method increasingly selects high-resolution flow maps and smaller blocks, indicating adaptive use of higher detail when bits are available.
• Visualization confirms that small blocks are preferentially selected in high-motion regions (e.g., object boundaries), while large blocks are used in smooth areas, aligning with motion complexity.
• The percentage of bits used for motion coding drops significantly in RaFC compared to DVC—e.g., from 61.11% to 38.89% at λ=256—demonstrating reduced motion bit cost.