[论文解读] Quadratic video interpolation

Xiangyu Xu, Siyao Li+3 more|arXiv (Cornell University)|Nov 2, 2019

Advanced Vision and Imaging被引用 47

一句话总结

本文提出一种二次视频插值方法，利用邻近帧的加速度信息来合成准确的中间帧，并由流反向层和自适应流过滤帮助实现高质量帧合成。

ABSTRACT

Video interpolation is an important problem in computer vision, which helps overcome the temporal limitation of camera sensors. Existing video interpolation methods usually assume uniform motion between consecutive frames and use linear models for interpolation, which cannot well approximate the complex motion in the real world. To address these issues, we propose a quadratic video interpolation method which exploits the acceleration information in videos. This method allows prediction with curvilinear trajectory and variable velocity, and generates more accurate interpolation results. For high-quality frame synthesis, we develop a flow reversal layer to estimate flow fields starting from the unknown target frame to the source frame. In addition, we present techniques for flow refinement. Extensive experiments demonstrate that our approach performs favorably against the existing linear models on a wide variety of video datasets.

研究动机与目标

• Motivate improving video frame interpolation beyond linear motion by exploiting acceleration information.
• Develop a quadratic flow model that predicts forward displacement from frame 0 to intermediate time t.
• Introduce a flow reversal layer to estimate backward flows for synthesis from known frames.
• Propose an adaptive flow filtering network to refine flow estimates and reduce artifacts.
• Demonstrate state-of-the-art performance across multiple datasets (GOPRO, Adobe240, UCF101, DAVIS).

提出的方法

• Predict forward flow f0→t with a quadratic model using f0→1 and f0→−1.
• Estimate backward flow f t→0 via a differentiable flow reversal layer that aggregates projected flows.
• Refine the reversed flow with an adaptive filtering network that learns a sampling offset and residuals.
• Warp and fuse warped frames I0 and I1 using learned masks to synthesize the intermediate frame.
• Train end-to-end with a loss combining L1 and perceptual terms.
• Optionally extend framework to higher-order (cubic) models.

实验结果

研究问题

• RQ1 Does incorporating acceleration information via a quadratic model improve interpolation quality over linear methods?
• RQ2 Can a differentiable flow reversal layer enable effective backward flow estimation for high-quality frame synthesis?
• RQ3 Does adaptive flow filtering reduce artifacts in reversed flow maps and improve visual fidelity?
• RQ4 How does the proposed quadratic interpolation perform across diverse datasets compared to state-of-the-art methods?

主要发现

Method	GOPRO Whole PSNR	GOPRO Whole SSIM	GOPRO IE	Adobe240 Whole PSNR	Adobe240 Whole SSIM	Adobe240 IE
Phase	23.95	0.700	17.89	22.05	0.620	22.08
DVF	21.94	0.776	21.30	20.55	0.720	25.14
SepConv	29.52	0.922	9.26	27.69	0.895	11.38
SuperSloMo	29.00	0.918	9.51	27.33	0.892	11.50
Ours w/o qua.	29.57	0.923	9.02	27.86	0.898	10.93
Ours	31.27	0.948	7.23	29.62	0.929	8.73

• The quadratic method outperforms linear methods on GOPRO and Adobe240 in PSNR and SSIM while reducing interpolation error (IE).
• Using acceleration information (quadratic flow) yields substantial PSNR/SSIM gains over w/o qua. by noticeable margins on several datasets.
• Flow reversal layer produces more accurate backward flow than simple linear combinations, improving synthesis quality.
• Adaptive flow filtering mitigates artifacts in reversed flows, yielding cleaner interpolations.
• Across UCF101 and DAVIS, the quadratic model achieves competitive PSNR/SSIM + lower IE, indicating good generalization.