[Paper Review] Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation

Guozhen Zhang, Yuhan Zhu+4 more|arXiv (Cornell University)|Mar 1, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes EMA-VFI, a novel video frame interpolation method that leverages inter-frame attention to explicitly extract both motion and appearance features in a unified manner, enhancing appearance via attention maps while reusing them for motion vector estimation. The approach achieves state-of-the-art performance on multiple datasets with lower computational overhead than comparable models.

ABSTRACT

Effectively extracting inter-frame motion and appearance information is important for video frame interpolation (VFI). Previous works either extract both types of information in a mixed way or elaborate separate modules for each type of information, which lead to representation ambiguity and low efficiency. In this paper, we propose a novel module to explicitly extract motion and appearance information via a unifying operation. Specifically, we rethink the information process in inter-frame attention and reuse its attention map for both appearance feature enhancement and motion information extraction. Furthermore, for efficient VFI, our proposed module could be seamlessly integrated into a hybrid CNN and Transformer architecture. This hybrid pipeline can alleviate the computational complexity of inter-frame attention as well as preserve detailed low-level structure information. Experimental results demonstrate that, for both fixed- and arbitrary-timestep interpolation, our method achieves state-of-the-art performance on various datasets. Meanwhile, our approach enjoys a lighter computation overhead over models with close performance. The source code and models are available at https://github.com/MCG-NJU/EMA-VFI.

Motivation & Objective

• To address the limitations of existing VFI methods that either mix motion and appearance features or use separate modules, leading to representation ambiguity and high computational cost.
• To enable efficient, scalable, and accurate video frame interpolation for both fixed- and arbitrary-timestep generation.
• To preserve fine-grained low-level structure by integrating CNNs with Transformers in a hybrid architecture that reduces attention overhead on high-resolution inputs.
• To unify motion and appearance feature extraction using a single inter-frame attention mechanism, avoiding redundant modules.

Proposed method

• The method reuses the inter-frame attention map not only to enhance appearance features through feature aggregation but also to estimate motion vectors via weighted displacement of neighboring features.
• Appearance features are refined by attending to temporal neighbors, improving contextual representation without mixing with motion signals.
• Motion features are derived directly from the attention map, enabling scalable motion cues for arbitrary-timestep interpolation.
• A hybrid CNN-Transformer architecture is employed: CNNs extract high-resolution low-level features, while Transformer blocks with inter-frame attention process low-resolution features to reduce computational cost.
• The model uses lightweight heads for motion estimation and appearance refinement, ensuring efficiency without sacrificing performance.
• The number and complexity of Transformer blocks can be adjusted to balance performance and inference speed, enabling scalable deployment.

Experimental results

Research questions

• RQ1 Can inter-frame attention be reused to simultaneously extract appearance enhancement and motion representation, avoiding separate modules?
• RQ2 How can inter-frame attention be applied efficiently at high resolution to avoid excessive computational cost?
• RQ3 Can motion features extracted from attention maps be effectively scaled for arbitrary-timestep frame interpolation?
• RQ4 Does a hybrid CNN-Transformer design preserve fine-grained details while reducing the computational burden of self-attention?
• RQ5 How does the proposed method compare in performance and efficiency to state-of-the-art VFI models?

Key findings

• EMA-VFI achieves state-of-the-art performance on Vimeo90K, Xiph-2K, and Xiph-4K datasets for both fixed- and arbitrary-timestep interpolation.
• On the Xiph-2K dataset, the method achieves a PSNR of 36.74 and SSIM of 0.944, outperforming previous SOTA methods.
• Using motion features as cues for arbitrary-timestep interpolation yields better results than directly injecting time t, with only a 3ms increase in runtime.
• Doubling the number of Transformer blocks or their channels improves performance significantly, with the latter showing a more pronounced gain due to higher capacity.
• Replacing the Transformer with CNNs at higher stages degrades performance, while using Transformers only at the lowest scale results in a substantial drop in accuracy, confirming the necessity of multi-scale attention.
• The model maintains a low inference time of 30ms on HD(720p) resolution, outperforming models with similar or better accuracy in terms of computational efficiency.