[Paper Review] Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model

Haojie Liu, Ming Lu+5 more|arXiv (Cornell University)|Jul 9, 2020

Advanced Image Processing Techniques44 references4 citations

TL;DR

This paper proposes an end-to-end neural video coding framework (NVC) that leverages multiscale motion compensation via a pyramid decoder and a novel multiscale motion compensation network (MS-MCN), along with adaptive spatiotemporal context modeling and nonlocal attention for improved rate-distortion performance. NVC achieves consistent gains over H.265/HEVC and other learned video coders across diverse video sequences, with average BD-rate savings of -50.83% in PSNR and -46.52% in MS-SSIM.

ABSTRACT

Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals, respectively. Novel features of NVC include: 1) To estimate and compensate motion over a large range of magnitudes, we propose an unsupervised multiscale motion compensation network (MS-MCN) together with a pyramid decoder in the VAE for coding motion features that generates multiscale flow fields, 2) we design a novel adaptive spatiotemporal context model for efficient entropy coding for motion information, 3) we adopt nonlocal attention modules (NLAM) at the bottlenecks of the VAEs for implicit adaptive feature extraction and activation, leveraging its high transformation capacity and unequal weighting with joint global and local information, and 4) we introduce multi-module optimization and a multi-frame training strategy to minimize the temporal error propagation among P-frames. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods following the common test conditions, demonstrating consistent gains across all popular test sequences for both PSNR and MS-SSIM distortion metrics.

Motivation & Objective

• To address the limitations of traditional block-based video coding, which rely on handcrafted tools and are constrained by diminishing returns from Moore’s Law, by developing an end-to-end deep learning-based video coding framework.
• To improve inter-frame prediction accuracy and robustness across large motion vectors by introducing a multiscale motion compensation network (MS-MCN) with a pyramid decoder for generating multiscale optical flows.
• To enhance entropy coding efficiency by modeling spatiotemporal correlations in motion information using an adaptive context model that combines spatial, temporal, and hyperprior information.
• To mitigate temporal error propagation in predictive coding by employing a multi-frame training strategy that minimizes reconstruction loss across multiple consecutive P-frames.
• To achieve superior rate-distortion performance compared to existing learned video coders and standard codecs like H.265/HEVC through joint optimization of intra, motion, and residual coding components.

Proposed method

• The NVC framework uses three separate variational autoencoders (VAEs) for neuro-Intra, neuro-Motion, and neuro-Res to compress intra-pixel, motion, and residual information in an end-to-end manner.
• A novel multiscale motion compensation network (MS-MCN) is introduced to estimate and compensate motion across multiple scales, generating multiscale flow fields that improve prediction accuracy for large motion vectors.
• The pyramid decoder in the neuro-Motion VAE enables hierarchical representation of motion features, allowing efficient multiscale flow estimation and reconstruction.
• An adaptive spatiotemporal context model is designed to improve entropy coding by jointly modeling spatial context, temporal context from previous frames, and hyperpriors from the VAE bottleneck.
• Nonlocal attention modules (NLAM) are integrated at the VAE bottlenecks to implicitly extract adaptive features with global and local receptive fields, enhancing representation capacity.
• Multi-module optimization and a multi-frame training strategy are employed to minimize temporal error propagation by jointly optimizing reconstruction loss across multiple consecutive P-frames.

Experimental results

Research questions

• RQ1 Can a deep neural network-based video coding framework outperform traditional block-based codecs like H.265/HEVC in terms of rate-distortion efficiency?
• RQ2 Does the use of multiscale motion compensation with a pyramid decoder significantly improve motion estimation and prediction accuracy, especially for large motion vectors?
• RQ3 Can an adaptive spatiotemporal context model enhance entropy coding efficiency for motion information beyond spatial context and hyperpriors?
• RQ4 To what extent does multi-frame training reduce temporal error propagation and improve stability in predictive video coding?
• RQ5 How does the integration of nonlocal attention modules at VAE bottlenecks affect the representation quality and compression efficiency of video features?

Key findings

• NVC achieves an average BD-rate gain of -50.83% in PSNR and -46.52% in MS-SSIM across all test sequences compared to H.265/HEVC, demonstrating consistent superiority.
• The proposed MS-MCN with a pyramid decoder outperforms single-scale motion compensation with a cascaded denoising network, achieving 0.2–0.3 dB PSNR gains across a wide range of bit rates.
• Multi-frame training significantly improves temporal stability and performance consistency, reducing quality degradation across GOPs compared to single-frame training.
• Motion information compression benefits most from temporal priors in high-motion content, with up to 58.63% bit savings in the HEVC Class E dataset.
• The adaptive spatiotemporal context model reduces motion bit rate by efficiently modeling correlations across space, time, and hyperpriors, especially in high-motion sequences.
• The use of nonlocal attention modules enhances feature representation by capturing long-range dependencies, contributing to improved reconstruction quality and coding efficiency.