[Paper Review] Understanding Deformable Alignment in Video Super-Resolution

Kelvin C. K. Chan, Xintao Wang+3 more|arXiv (Cornell University)|Sep 15, 2020

Advanced Vision and Imaging17 citations

TL;DR

This paper reveals that deformable convolution in video super-resolution performs implicit feature alignment by combining spatial warping and convolution, sharing a core formulation with flow-based alignment but enabling greater offset diversity. The authors propose an offset-fidelity loss that constrains learned offsets to align with optical flow, stabilizing training and improving alignment accuracy, which leads to significant PSNR gains in video super-resolution by reducing warping errors and occlusion effects.

ABSTRACT

Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.

Motivation & Objective

• To understand the underlying mechanism of deformable convolution in video super-resolution alignment.
• To formally establish the relationship between deformable alignment and flow-based alignment.
• To investigate the role of offset diversity in enhancing alignment accuracy and super-resolution quality.
• To address the instability of deformable alignment training, particularly offset overflow.
• To propose a novel offset-fidelity loss that leverages optical flow to stabilize and improve deformable alignment.

Proposed method

• Deformable alignment is decomposed into a combination of feature-level spatial warping and standard convolution, revealing its formal equivalence to flow-based alignment.
• The method demonstrates that deformable convolution introduces multiple offsets per feature location (proportional to kernel size), increasing offset diversity compared to single-offset flow-based methods.
• An offset-fidelity loss is proposed to constrain learned offsets within a threshold of optical flow predictions, preventing overflow and stabilizing training.
• The formulation allows for flexible introduction of offset diversity into flow-based alignment by decoupling the number of offsets from kernel size.
• The approach is evaluated on EDVR, a state-of-the-art video super-resolution model, using PSNR and qualitative comparisons on the REDS4 benchmark.
• Modulation masks are used to weight contributions of diverse offsets, with lower mask values indicating less importance for offsets far from optical flow.

Experimental results

Research questions

• RQ1 How is deformable alignment formally related to flow-based alignment in video super-resolution?
• RQ2 What is the role of offset diversity in improving alignment accuracy and super-resolution performance?
• RQ3 Why does deformable alignment outperform flow-based alignment despite their shared underlying formulation?
• RQ4 How can the instability of deformable alignment training—particularly offset overflow—be mitigated?
• RQ5 Can optical flow be used to guide and stabilize the learning of deformable convolution offsets?

Key findings

• Deformable alignment is mathematically equivalent to a combination of spatial warping and convolution, sharing the same core formulation as flow-based alignment.
• The key difference lies in offset diversity: deformable alignment uses multiple offsets per location (proportional to kernel size), while flow-based alignment uses only one, leading to improved robustness against occlusion and motion errors.
• The proposed offset-fidelity loss successfully prevents offset overflow during training, stabilizing the network and avoiding degradation into a single-image SR model.
• With the offset-fidelity loss, EDVR achieves significantly better performance, with visible improvements in detail recovery, especially in high-motion regions like moving car wheels.
• When the number of offsets increases, a growing proportion of modulation masks take small values (≈0), indicating that many offsets become redundant, and PSNR gains saturate beyond a certain point.
• The study demonstrates that offset diversity is the primary factor behind the superior performance of deformable alignment, not just a different optimization strategy.