[Paper Review] TDAN: Temporally Deformable Alignment Network for Video Super-Resolution

Yapeng Tian, Yulun Zhang+2 more|arXiv (Cornell University)|Dec 7, 2018

Advanced Image Processing TechniquesComputer Science26 references72 citations

TL;DR

TDAN introduces a one-stage, feature-level temporal alignment using deformable convolutions to align supporting frames with a reference frame for video super-resolution, achieving state-of-the-art results on Vid4 BI degradation and competitive performance elsewhere.

ABSTRACT

Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames). Due to varying motion of cameras or objects, the reference frame and each support frame are not aligned. Therefore, temporal alignment is a challenging yet important problem for VSR. Previous VSR methods usually utilize optical flow between the reference frame and each supporting frame to wrap the supporting frame for temporal alignment. Therefore, the performance of these image-level wrapping-based models will highly depend on the prediction accuracy of optical flow, and inaccurate optical flow will lead to artifacts in the wrapped supporting frames, which also will be propagated into the reconstructed HR video frame. To overcome the limitation, in this paper, we propose a temporal deformable alignment network (TDAN) to adaptively align the reference frame and each supporting frame at the feature level without computing optical flow. The TDAN uses features from both the reference frame and each supporting frame to dynamically predict offsets of sampling convolution kernels. By using the corresponding kernels, TDAN transforms supporting frames to align with the reference frame. To predict the HR video frame, a reconstruction network taking aligned frames and the reference frame is utilized. Experimental results demonstrate the effectiveness of the proposed TDAN-based VSR model.

Motivation & Objective

• Motivate robust video super-resolution (VSR) without explicit optical flow estimation.
• Develop a light-weight, end-to-end trainable framework that aligns supporting frames to a reference frame at the feature level.
• Propose a deformable alignment mechanism that predicts sampling offsets for adaptive temporal alignment.
• Demonstrate improved reconstruction quality over flow-based methods on standard VSR benchmarks.

Proposed method

• Propose TDAN, a temporally deformable alignment network that predicts offsets for deformable convolution kernels using features from reference and supporting frames.
• Use a three-part TDAN design: feature extraction, deformable alignment with predicted sampling offsets, and aligned frame reconstruction to produce I_i^{LR'}.
• Integrate a reconstruction network that fuses 2N+1 frames (including the reference) to predict the high-resolution frame I_t^{HR}.
• Train end-to-end with a dual loss: L_align to push aligned supporting frames toward the reference and L_sr to enforce accurate HR reconstruction.
• Ground TDAN training in self-supervision by using the reference frame as the alignment target (no ground-truth aligned frames needed).

Experimental results

Research questions

• RQ1 Can one-stage, feature-level temporal alignment via deformable convolutions outperform traditional optical-flow-based alignment in VSR?
• RQ2 How does the TDAN-driven alignment affect PSNR/SSIM on standard benchmarks compared to flow-based and single-image SR methods?
• RQ3 What is the impact of varying the number of deformable layers on TDAN performance and convergence?
• RQ4 Is TDAN robust to different degradation configurations and real-world unknown degradations?

Key findings

• TDAN achieves state-of-the-art performance among flow-based VSR methods and competitive results versus SISR baselines on BI degradation (Vid4).
• TDAN outperforms TOFlow and other flow-based methods in PSNR and SSIM on Vid4 BI setup (City, Walk, Calendar, Foliage) with average gains.
• On BD degradation, TDAN generally surpasses SPMC and FRVSR in PSNR but may be outperformed in SSIM by DUF, though still competitive.
• Model size of TDAN is comparable to FRVSR/DUF and significantly smaller than leading SISR models like RCAN/RDN/TOFlow, while delivering strong VSR results.
• Ablation shows more deformable layers improve performance, with D4 (comparable in size to FRVSR/DUF) achieving state-of-the-art results in many settings.
• TDAN demonstrates robustness on real-world sequences with unknown degradation, producing sharper edges and more details than competing methods.