[Paper Review] Asymmetric Bilateral Motion Estimation for Video Frame Interpolation

Junheum Park, Chul Lee+1 more|arXiv (Cornell University)|Aug 15, 2021

Advanced Vision and Imaging42 references4 citations

TL;DR

This paper proposes Asymmetric Bilateral Motion Estimation (ABME), a novel video frame interpolation method that improves motion accuracy by refining symmetric bilateral motion fields using asymmetric motion vectors from an intermediate anchor frame. By combining dynamic filtering and residual refinement, ABME achieves state-of-the-art performance, outperforming prior methods by up to 2 dB on Vimeo90K and 1 dB on challenging subsets.

ABSTRACT

We propose a novel video frame interpolation algorithm based on asymmetric bilateral motion estimation (ABME), which synthesizes an intermediate frame between two input frames. First, we predict symmetric bilateral motion fields to interpolate an anchor frame. Second, we estimate asymmetric bilateral motions fields from the anchor frame to the input frames. Third, we use the asymmetric fields to warp the input frames backward and reconstruct the intermediate frame. Last, to refine the intermediate frame, we develop a new synthesis network that generates a set of dynamic filters and a residual frame using local and global information. Experimental results show that the proposed algorithm achieves excellent performance on various datasets. The source codes and pretrained models are available at https://github.com/JunHeum/ABME.

Motivation & Objective

• Address motion estimation inaccuracies in video frame interpolation caused by the rigid linear motion constraint in symmetric bilateral motion models.
• Overcome limitations of forward and backward warping, particularly hole and overlap artifacts, by leveraging backward warping with refined motion fields.
• Improve interpolation quality in occluded and motion-boundary regions where symmetric motion models fail due to non-linear trajectories.
• Develop a synthesis network that effectively fuses warped frames using dynamic filters and global contextual features to reduce artifacts and noise.
• Achieve state-of-the-art performance across diverse datasets, including challenging sequences with fast motion and occlusions.

Proposed method

• Predict symmetric bilateral motion fields between input frames to generate an intermediate anchor frame via backward warping.
• Estimate asymmetric bilateral motion fields from the anchor frame to both input frames, relaxing the linear motion constraint to better model non-linear trajectories.
• Warp the input frames backward using the asymmetric motion fields to generate candidate warped frames for reconstruction.
• Design a synthesis network with two subnetworks: FilterNet generates dynamic filters using local features to weight candidate warped frames, and RefineNet reconstructs a residual frame using global context.
• Integrate a learnable reliability mask and offset map in the asymmetric motion refinement module to improve motion accuracy in challenging regions.
• Fuse multiple warped frame candidates (symmetric and asymmetric) using learned dynamic filters and refine the output with contextual feature maps to enhance fidelity.

Experimental results

Research questions

• RQ1 Can asymmetric bilateral motion estimation improve motion accuracy in video frame interpolation compared to symmetric models?
• RQ2 How does relaxing the linear motion constraint affect interpolation quality in regions with occlusions and motion boundaries?
• RQ3 To what extent do dynamic filters and residual refinement enhance frame synthesis when combining multiple warped inputs?
• RQ4 What is the contribution of the reliability mask and offset map in refining asymmetric motion fields?
• RQ5 Does combining symmetric and asymmetric warped frame candidates lead to better performance than using either alone?

Key findings

• The proposed ABME method achieves state-of-the-art performance on the Vimeo90K dataset, outperforming prior methods by up to 2 dB in PSNR on the Easy and Medium subsets.
• On the Hard and Extreme subsets, ABME improves performance by at least 1 dB over the best existing method, demonstrating robustness to challenging motion patterns.
• The combination of symmetric and asymmetric warped frame candidates yields the best result (36.18 PSNR on Vimeo90K), outperforming both symmetric-only (35.91) and asymmetric-only (36.05) settings.
• The reliability mask and offset map in ABMR-Net each contribute significantly to motion refinement, improving PSNR by 0.49 dB and 0.52 dB respectively on Vimeo90K when both are used.
• The inclusion of filtered feature maps ($\widetilde{C}_t$) in RefineNet improves reconstruction by 0.1 dB in PSNR, confirming its role in noise reduction and context-aware restoration.
• Visual analysis confirms that ABME reduces artifacts and errors around moving objects and occlusions—especially in fast-motion sequences—compared to symmetric or approximated motion models.