[Paper Review] Video Super-Resolution Transformer

Jiezhang Cao, Yawei Li+2 more|arXiv (Cornell University)|Jun 12, 2021

Advanced Image Processing TechniquesComputer Science4 references132 citations

TL;DR

This paper introduces VSR-Transformer, a Transformer variant for video super-resolution that uses a spatial-temporal convolutional self-attention (STCSA) layer and a bidirectional optical flow-based feed-forward (BOFF) layer to capture locality and enable feature propagation/alignment across frames.

ABSTRACT

Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. Thus, it seems to be straightforward to apply the vision Transformer to solve VSR. However, the typical block design of Transformer with a fully connected self-attention layer and a token-wise feed-forward layer does not fit well for VSR due to the following two reasons. First, the fully connected self-attention layer neglects to exploit the data locality because this layer relies on linear layers to compute attention maps. Second, the token-wise feed-forward layer lacks the feature alignment which is important for VSR since this layer independently processes each of the input token embeddings without any interaction among them. In this paper, we make the first attempt to adapt Transformer for VSR. Specifically, to tackle the first issue, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information. For the second issue, we design a bidirectional optical flow-based feed-forward layer to discover the correlations across different video frames and also align features. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our proposed method. The code will be available at https://github.com/caojiezhang/VSR-Transformer.

Motivation & Objective

• Motivate improving VSR by leveraging Transformers while addressing locality and cross-frame alignment limitations.
• Propose STCSA to exploit spatial-temporal locality in video frames.
• Introduce BOFF to enable feature propagation and alignment across frames via optical flow.
• Demonstrate effectiveness on benchmark VSR datasets and compare with state-of-the-art methods.

Proposed method

• Propose spatial-temporal convolutional self-attention (STCSA) that unfolds input frames into local 3D patches and computes attention over patches to capture locality.
• Provide a theoretical analysis showing STCSA advantages over fully connected self-attention for learning k-pattern locality (Theorem 2).
• Introduce a bidirectional optical flow-based feed-forward layer (BOFF) that uses forward/backward flows to warp features and perform bidirectional propagation and fusion across frames.
• Incorporate 3D fixed spatial-temporal positional encodings to preserve position information in a permutation-invariant architecture.
• Build an encoder-reconstruction pipeline with a feature extractor, VSR-Transformer encoder, and reconstruction network.
• Train and evaluate on REDS4, Vimeo-90K-T, and Vid4 with standard PSNR/SSIM metrics.

Experimental results

Research questions

• RQ1 Can STCSA effectively exploit locality in video data compared to traditional fully connected self-attention in Vision Transformers?
• RQ2 Does the bidirectional optical flow-based feed-forward layer improve feature propagation and alignment across frames for VSR?
• RQ3 What is the impact of incorporating spatial-temporal positional encodings on VSR performance?
• RQ4 How does the proposed VSR-Transformer compare to state-of-the-art VSR methods across standard benchmarks?
• RQ5 Is the model scalable in terms of parameters while delivering competitive VSR results?

Key findings

• VSR-Transformer achieves highest PSNR and competitive SSIM on REDS4 for 4x VSR compared to several baselines.
• On Vimeo-90K-T, the method yields strong PSNR/SSIM, outperforming several 7-frame baselines.
• Across Vid4 (Y-channel), the approach achieves leading average performance among reported methods.
• The STCSA layer demonstrates theoretical and empirical advantages in capturing locality over FCSA.
• BOFF enables effective feature propagation and cross-frame alignment, contributing to improved VSR performance.
• The model with 64 channels can outperform EDVR-L with 128 channels, particularly for scenarios with limited frame availability.