[Paper Review] AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation

Zhen Li, Zuo-Liang Zhu+4 more|arXiv (Cornell University)|Apr 19, 2023

Advanced Vision and Imaging4 citations

TL;DR

AMT proposes a novel video frame interpolation network using all-pairs multi-field transforms to improve flow estimation fidelity and diversity. By leveraging bidirectional correlation volumes and multiple refined flow fields from coarse flows, AMT achieves state-of-the-art performance with higher efficiency than prior methods, outperforming IFRNet-L by +0.15 dB PSNR on Vimeo90K while using 75% fewer FLOPs and 65% fewer parameters.

ABSTRACT

We present All-Pairs Multi-Field Transforms (AMT), a new network architecture for video frame interpolation. It is based on two essential designs. First, we build bidirectional correlation volumes for all pairs of pixels, and use the predicted bilateral flows to retrieve correlations for updating both flows and the interpolated content feature. Second, we derive multiple groups of fine-grained flow fields from one pair of updated coarse flows for performing backward warping on the input frames separately. Combining these two designs enables us to generate promising task-oriented flows and reduce the difficulties in modeling large motions and handling occluded areas during frame interpolation. These qualities promote our model to achieve state-of-the-art performance on various benchmarks with high efficiency. Moreover, our convolution-based model competes favorably compared to Transformer-based models in terms of accuracy and efficiency. Our code is available at https://github.com/MCG-NKU/AMT.

Motivation & Objective

• To address the limitations of existing flow-based video frame interpolation methods in handling large motions and occlusions.
• To improve the fidelity and diversity of predicted task-oriented flows by introducing a new correlation and refinement mechanism.
• To achieve state-of-the-art performance with high efficiency, reducing computational cost without sacrificing accuracy.
• To demonstrate that convolution-based models can outperform Transformer-based models in accuracy and efficiency for frame interpolation.

Proposed method

• AMT constructs bidirectional correlation volumes between all pixel pairs to enhance flow estimation fidelity, especially for large motions.
• It employs a scaled lookup strategy to resolve coordinate mismatches caused by invisible frames during correlation computation.
• The model uses cross-scale, joint updates of bilateral flows and content features via all-pairs correlation to improve motion consistency across scales.
• Multiple groups of fine-grained flow fields are derived from a single pair of coarse flows, enabling diverse warping candidates for each pixel.
• Each input frame is separately backward-warping using these multiple flow fields, improving reconstruction in occluded regions.
• An adaptive fusion mechanism combines warped features with residual refinement, enabling better detail recovery and texture consistency.

Experimental results

Research questions

• RQ1 Why do previous VFI methods struggle with accurate flow estimation under large motions and occlusions?
• RQ2 How can flow estimation be improved to be both consistent with ground truth and diverse in local details?
• RQ3 Can a single coarse flow pair be enhanced through multiple refined flow fields to better handle ambiguous regions like occlusions?
• RQ4 To what extent does all-pairs correlation improve flow and content feature refinement across scales?
• RQ5 Can a convolution-based architecture outperform Transformer-based models in both accuracy and efficiency for frame interpolation?

Key findings

• AMT achieves state-of-the-art performance on Vimeo90K, surpassing IFRNet-L by +0.15 dB PSNR with 75% fewer FLOPs and 65% fewer parameters.
• The small AMT-S model outperforms IFRNet-B by +0.17 dB PSNR while using only 60% of its FLOPs and parameters.
• Using three pairs of flow fields provides a significant performance gain, with saturation observed at seven pairs, indicating optimal trade-off at five pairs for larger models.
• Ablation studies confirm that all-pairs correlation is critical for joint flow and content feature refinement, as removing it causes a dramatic PSNR drop.
• Multi-field refinement significantly improves occlusion handling, enabling consistent texture recovery in background regions obscured by moving objects.
• The adaptive fusion of multiple warped features with residual refinement outperforms average-based or non-residual variants, proving the importance of per-frame detail compensation.