[Paper Review] MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement

Wenbo Bao, Wei‐Sheng Lai+3 more|arXiv (Cornell University)|Oct 20, 2018

Advanced Vision and Imaging59 references32 citations

TL;DR

This paper proposes MEMC-Net, a deep neural network that jointly learns motion estimation and motion compensation for video frame interpolation and enhancement. By introducing a fully differentiable adaptive warping layer that combines optical flow and interpolation kernels, the model achieves state-of-the-art results in interpolation, super-resolution, denoising, and deblocking with improved computational efficiency and visual quality.

ABSTRACT

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.

Motivation & Objective

• Address the limitations of existing learning-based video frame interpolation methods that either estimate only optical flow or only compensation kernels, leading to blurry results or sensitivity to large motion.
• Integrate motion estimation and motion compensation within an end-to-end trainable deep learning framework to combine the strengths of classical MEMC methods and data-driven approaches.
• Develop a novel adaptive warping layer that fuses optical flow and learned interpolation kernels to synthesize high-quality intermediate frames.
• Extend the proposed architecture to multiple video enhancement tasks, including super-resolution, denoising, and deblocking, demonstrating its generalization capability.
• Improve visual quality and computational efficiency by jointly optimizing flow and kernel estimation networks through back-propagation.

Proposed method

• Propose a fully differentiable adaptive warping layer that combines optical flow and learned interpolation kernels to synthesize target frame pixels.
• Train a flow estimation network and a kernel estimation network end-to-end using back-propagation, enabling joint optimization of motion estimation and compensation.
• Estimate occlusion masks to adaptively blend warped frames and reduce artifacts in regions with motion discontinuities or missing data.
• Apply a post-processing CNN to refine pixels in holes and unreliable regions caused by occlusions or motion blur.
• Use residual blocks and context aggregation to enhance feature representation and preserve fine details at motion boundaries.
• Adapt the same architecture to video super-resolution, denoising, and deblocking by modifying the input and loss functions while keeping the core network structure unchanged.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework jointly optimize motion estimation and motion compensation to improve video frame interpolation quality?
• RQ2 How does the integration of optical flow and learned interpolation kernels via an adaptive warping layer affect visual fidelity and computational efficiency?
• RQ3 To what extent can a MEMC-Net-based architecture generalize across multiple video enhancement tasks beyond interpolation?
• RQ4 Does the proposed method outperform state-of-the-art methods in terms of PSNR, SSIM, and visual quality on benchmark datasets?
• RQ5 How effective is the occlusion-aware blending and post-processing module in reducing artifacts in complex motion regions?

Key findings

• MEMC-Net achieves state-of-the-art performance on video frame interpolation, outperforming methods like ToFlow, MIND, and EpicFlow in both quantitative metrics and visual quality on the Vimeo90k and DAVIS datasets.
• On the BayesSR super-resolution dataset, MEMC-Net_SR achieves higher PSNR than EDSR (SISR) and other video super-resolution models, despite using fewer residual blocks and fewer filters.
• For video denoising, MEMC-Net_DN achieves 1.24 dB and 1.95 dB PSNR gains over the second-best method on the Vimeo90k and V-BM4D datasets, respectively.
• In video deblocking, MEMC-Net_DB outperforms EDSR_DB, ToFlow, and V-BM4D, effectively reducing blocky artifacts while preserving fine textures.
• The improved variant MEMC-Net* with enhanced context modeling produces sharper results with better detail recovery, especially around motion boundaries.
• Qualitative results show that MEMC-Net produces clearer edges, fewer artifacts, and better preservation of fine textures compared to existing methods.