[Paper Review] Temporal Modulation Network for Controllable Space-Time Video Super-Resolution

Gang Xu, Jun Xu+4 more|arXiv (Cornell University)|Apr 21, 2021

Advanced Image Processing Techniques47 references4 citations

TL;DR

This paper proposes a Temporal Modulation Network (TMNet) for controllable space-time video super-resolution, enabling interpolation of arbitrary intermediate frames through a Temporal Modulation Block (TMB) that dynamically adjusts deformable convolution kernels using a temporal parameter. The method integrates short-term motion cues via a Locally-temporal Feature Comparison (LFC) module and long-term context via Bi-directional Deformable ConvLSTM, achieving state-of-the-art performance on three benchmarks with PSNR up to 37.04dB on Vimeo-90K Fast.

ABSTRACT

Space-time video super-resolution (STVSR) aims to increase the spatial and temporal resolutions of low-resolution and low-frame-rate videos. Recently, deformable convolution based methods have achieved promising STVSR performance, but they could only infer the intermediate frame pre-defined in the training stage. Besides, these methods undervalued the short-term motion cues among adjacent frames. In this paper, we propose a Temporal Modulation Network (TMNet) to interpolate arbitrary intermediate frame(s) with accurate high-resolution reconstruction. Specifically, we propose a Temporal Modulation Block (TMB) to modulate deformable convolution kernels for controllable feature interpolation. To well exploit the temporal information, we propose a Locally-temporal Feature Comparison (LFC) module, along with the Bi-directional Deformable ConvLSTM, to extract short-term and long-term motion cues in videos. Experiments on three benchmark datasets demonstrate that our TMNet outperforms previous STVSR methods. The code is available at https://github.com/CS-GangXu/TMNet.

Motivation & Objective

• To address the limitation of existing STVSR methods that can only generate pre-defined intermediate frames, restricting their use in flexible broadcasting scenarios.
• To improve feature interpolation accuracy by incorporating controllable temporal modulation into deformable convolution for arbitrary frame-rate generation.
• To enhance motion modeling by fusing both short-term and long-term temporal cues for better spatial alignment and reconstruction.
• To achieve high-quality, temporally consistent video super-resolution with reduced artifacts and improved perceptual quality.

Proposed method

• Introduce a Temporal Modulation Block (TMB) that transforms a temporal parameter t into a modulation vector vt to control deformable convolution kernel sampling for arbitrary intermediate frame interpolation.
• Design a Locally-temporal Feature Comparison (LFC) module to extract and compare multi-frame features for accurate short-term motion modeling and spatial alignment.
• Employ a Bi-directional Deformable ConvLSTM to capture long-term motion variations across the entire video sequence.
• Implement a two-stage temporal feature fusion strategy: first local fusion via LFC, then global fusion via GFF, to effectively combine short- and long-term cues.
• Combine high-resolution features 𝒮𝐻 with initial low-resolution features 𝒮𝐿 before Pixel-Shuffle layers to preserve motion and structural details.
• Train the network end-to-end on Vimeo-90K, Adobe240fps, and Vid4 datasets using L1 loss for reconstruction.

Experimental results

Research questions

• RQ1 Can a deep learning model interpolate arbitrary intermediate frames in space-time video super-resolution, rather than being limited to fixed-frame-rate outputs?
• RQ2 How can motion cues from adjacent frames be effectively modeled to improve feature interpolation and reconstruction quality?
• RQ3 What is the impact of combining local and global temporal feature fusion on STVSR performance?
• RQ4 Does incorporating both high-resolution and initial low-resolution features improve the final reconstruction quality?

Key findings

• TMNet achieves state-of-the-art performance on three benchmark datasets, with a PSNR of 37.04dB on the Vimeo-90K Fast set, outperforming previous methods.
• The ablation study confirms that the LFC module is essential, as removing it reduces performance by 0.07dB on Vid4 and up to 0.17dB on Vimeo- Fast.
• The TMB with nonlinear layers slightly outperforms the linear variant, achieving 26.95dB vs. 26.93dB on Adobe240fps, indicating the benefit of nonlinearity in temporal modulation.
• Fusing high-quality features 𝒮𝐻 with initial features 𝒮𝐿 improves PSNR by 0.07–0.12dB across datasets, validating the feature concatenation strategy.
• The two-stage fusion (LFC → GFF) is critical: reversing the order (GFF → LFC) causes training collapse due to noisy long-term signals.
• TMNet enables flexible interpolation at any desired frame rate, such as generating frames at 0.3, 0.5, and 0.7 of the interval, unlike prior methods restricted to fixed intermediate frames.