[Paper Review] Video Frame Interpolation without Temporal Priors

Youjian Zhang, Chaoyue Wang+1 more|arXiv (Cornell University)|Dec 2, 2021

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes a video frame interpolation method that operates without requiring temporal priors such as known frame rate or exposure time. By deriving a general quadratic motion trajectory from four consecutive frames and using a second-order residual network to restore sharp key states, the method enables accurate interpolation across varying camera settings, achieving state-of-the-art results on real-world blurry videos without retraining.

ABSTRACT

Video frame interpolation, which aims to synthesize non-exist intermediate frames in a video sequence, is an important research topic in computer vision. Existing video frame interpolation methods have achieved remarkable results under specific assumptions, such as instant or known exposure time. However, in complicated real-world situations, the temporal priors of videos, i.e. frames per second (FPS) and frame exposure time, may vary from different camera sensors. When test videos are taken under different exposure settings from training ones, the interpolated frames will suffer significant misalignment problems. In this work, we solve the video frame interpolation problem in a general situation, where input frames can be acquired under uncertain exposure (and interval) time. Unlike previous methods that can only be applied to a specific temporal prior, we derive a general curvilinear motion trajectory formula from four consecutive sharp frames or two consecutive blurry frames without temporal priors. Moreover, utilizing constraints within adjacent motion trajectories, we devise a novel optical flow refinement strategy for better interpolation results. Finally, experiments demonstrate that one well-trained model is enough for synthesizing high-quality slow-motion videos under complicated real-world situations. Codes are available on https://github.com/yjzhang96/UTI-VFI.

Motivation & Objective

• To address the challenge of video frame interpolation in real-world scenarios where temporal priors like frame rate and exposure time are unknown or variable.
• To overcome the limitations of existing methods that rely on fixed temporal assumptions and fail under varying camera settings.
• To develop a general-purpose interpolation framework that can handle both inter-frame and intra-frame motion blur without requiring prior knowledge of shutter speed or frame interval.
• To enable a single trained model to generalize across diverse exposure conditions, eliminating the need for retraining per camera configuration.

Proposed method

• The method derives a general curvilinear motion trajectory using four consecutive frames, without assuming known exposure time or frame rate.
• It introduces a second-order residual key-state restoration network to recover sharp start and end states from blurry frames, enabling accurate motion estimation.
• The approach models motion using a quadratic function under the assumption of constant acceleration, allowing interpolation across uneven time intervals.
• A novel optical flow refinement strategy is proposed, leveraging constraints between adjacent motion trajectories to improve flow accuracy.
• The framework performs both inter-frame and intra-frame interpolation by distinguishing between motion blur and temporal gaps.
• The model is trained end-to-end on low-frame-rate, blurry videos and generalizes to various exposure settings without retraining.

Experimental results

Research questions

• RQ1 Can video frame interpolation be effectively performed when the exposure time and frame interval are unknown or variable across different cameras?
• RQ2 How can motion trajectories be modeled accurately in the absence of temporal priors such as frame rate or shutter speed?
• RQ3 To what extent can a single model generalize across diverse exposure settings without retraining?
• RQ4 Can a unified framework handle both inter-frame and intra-frame interpolation in blurry video sequences?
• RQ5 How does the proposed optical flow refinement strategy improve interpolation quality compared to standard flow estimation?

Key findings

• The proposed method achieves a PSNR of 34.37 and SSIM of 0.9801 on the GoPro-5-5 benchmark, outperforming the baseline by 1.95 dB in PSNR.
• The ablation study shows that the second-order residual network improves PSNR by approximately 1.5 dB over first-order residual networks.
• The flow refinement module increases PSNR by 0.6 dB compared to the model without refinement, demonstrating its effectiveness.
• The model generalizes well across different exposure settings, with consistent performance on GoPro-7-3 and GoPro-9-1 benchmarks.
• The key-state restoration network effectively recovers sharp content from blurry frames, enabling accurate motion estimation even under variable exposure.
• The method achieves state-of-the-art performance on real-world blurry video datasets without requiring temporal priors or retraining for different camera settings.