[Paper Review] Boost Video Frame Interpolation via Motion Adaptation

Haoning Wu, Xiaoyun Zhang+3 more|arXiv (Cornell University)|Jun 24, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a test-time motion adaptation strategy called cycle-consistency adaptation to boost video frame interpolation (VFI) performance by refining motion estimation on unseen test videos. By injecting a lightweight adapter into pre-trained VFI models, it enables efficient, on-the-fly optimization with under 4% additional parameters, significantly improving generalization and outperforming state-of-the-art methods—even those using extra input frames.

ABSTRACT

Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video. Existing learning-based VFI methods have achieved great success, but they still suffer from limited generalization ability due to the limited motion distribution of training datasets. In this paper, we propose a novel optimization-based VFI method that can adapt to unseen motions at test time. Our method is based on a cycle-consistency adaptation strategy that leverages the motion characteristics among video frames. We also introduce a lightweight adapter that can be inserted into the motion estimation module of existing pre-trained VFI models to improve the efficiency of adaptation. Extensive experiments on various benchmarks demonstrate that our method can boost the performance of two-frame VFI models, outperforming the existing state-of-the-art methods, even those that use extra input.

Motivation & Objective

• To address the limited generalization of learning-based VFI models due to domain gaps in training data.
• To improve VFI performance on unseen videos with complex or large-scale motions.
• To enable efficient test-time adaptation without retraining the entire model.
• To design a plug-and-play adapter that enhances motion estimation with minimal parameter overhead.
• To outperform existing SOTA methods, including those using extra input frames.

Proposed method

• Proposes a cycle-consistency adaptation strategy that leverages inter-frame consistency in triplet video sequences (e.g., frame t-1, t, t+1) to refine motion estimation during inference.
• Uses a cycle-consistency loss to ensure that motion flows between frames are consistent and reversible, improving motion field accuracy.
• Introduces a lightweight, plug-in adapter module that can be inserted into the motion estimation head of existing pre-trained VFI models.
• The adapter fine-tunes only a small fraction (<4%) of the model’s parameters during test-time adaptation, significantly reducing computational cost.
• Employs end-to-end optimization on test sequences to adapt motion fields to the specific motion characteristics of each video.
• Visualizes motion fields before and after adaptation to qualitatively validate improved flow smoothness and edge precision.

Experimental results

Research questions

• RQ1 Can test-time motion adaptation improve the generalization of VFI models on unseen video distributions?
• RQ2 How effective is cycle-consistency-based adaptation in refining motion fields for complex or large-scale motions?
• RQ3 Can a lightweight adapter achieve comparable performance to full fine-tuning while drastically reducing inference cost?
• RQ4 Does the proposed method outperform SOTA VFI models, including those using extra input frames?
• RQ5 Can the adapter be effectively applied to various pre-trained VFI architectures without architectural changes?

Key findings

• Cycle-consistency adaptation consistently improves PSNR and SSIM across multiple benchmarks, with gains of up to 1.2 dB in PSNR and 0.012 in SSIM on Vimeo90K.
• The plug-in adapter reduces the number of trainable parameters for adaptation to less than 4% of the original model, enabling efficient test-time tuning.
• Adaptation with the plug-in adapter achieves a 2× speedup in adaptation time compared to end-to-end fine-tuning, while maintaining similar performance.
• On the DAVIS dataset, the method reduces visual distortions and improves structural clarity in interpolated frames, especially in cases with large or complex motions.
• The proposed method outperforms state-of-the-art VFI models, including those using extra input frames, on all evaluated benchmarks.
• Motion field visualization confirms that the adapted model produces smoother, more accurate motion fields with sharper edges, leading to higher-quality frame synthesis.