[Paper Review] Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

Yi Xiao, Qiangqiang Yuan+5 more|arXiv (Cornell University)|Apr 10, 2023

Advanced Vision and Imaging68 references4 citations

TL;DR

This paper proposes LGTD, a novel satellite video super-resolution method that leverages local and global temporal differences for effective motion compensation. By modeling short-term RGB differences between adjacent frames and long-term bidirectional differences across sequences, LGTD improves feature alignment and reconstruction fidelity, achieving state-of-the-art performance on five satellite video datasets with a PSNR gain of up to 0.23dB over single-directional baselines.

ABSTRACT

Optical-flow-based and kernel-based approaches have been extensively explored for temporal compensation in satellite Video Super-Resolution (VSR). However, these techniques are less generalized in large-scale or complex scenarios, especially in satellite videos. In this paper, we propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies since we observed that these discrepancies offer distinct and mutually complementary properties. Specifically, we devise a Short-term Temporal Difference Module (S-TDM) to extract local motion representations from RGB difference maps between adjacent frames, which yields more clues for accurate texture representation. To explore the global dependency in the entire frame sequence, a Long-term Temporal Difference Module (L-TDM) is proposed, where the differences between forward and backward segments are incorporated and activated to guide the modulation of the temporal feature, leading to a holistic global compensation. Moreover, we further propose a Difference Compensation Unit (DCU) to enrich the interaction between the spatial distribution of the target frame and temporal compensated results, which helps maintain spatial consistency while refining the features to avoid misalignment. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches. Code will be available at https://github.com/XY-boy/LGTD

Motivation & Objective

• To address the challenge of inaccurate temporal compensation in satellite video super-resolution due to static backgrounds and complex motion.
• To overcome the limitations of optical flow and kernel-based methods in handling large-scale, redundant satellite video sequences.
• To exploit temporal differences—both local and global—as a more robust alternative to optical flow for motion representation.
• To improve spatial consistency and feature fidelity during super-resolution reconstruction by introducing a Difference Compensation Unit (DCU).
• To achieve state-of-the-art performance in satellite video super-resolution using a hybrid attention mechanism for reconstruction.

Proposed method

• Proposes a Short-term Temporal Difference Module (S-TDM) that computes RGB differences between adjacent frames to extract local motion representations.
• Introduces a Long-term Temporal Difference Module (L-TDM) that models bidirectional differences between forward and backward segments to capture global temporal dependencies.
• Uses smoothed forward and backward features to reduce redundancy before computing holistic temporal differences for long-range compensation.
• Employs a Difference Compensation Unit (DCU) to guide temporal feature modulation by aligning spatial and temporal representations, reducing misalignment.
• Incorporates a hybrid attention mechanism combining self-attention and channel attention to reconstruct high-quality HR frames from combined local and global features.
• Designs a two-stage feature fusion strategy to integrate S-TDM and L-TDM outputs with the target frame for improved spatial-temporal consistency.

Figure 1: Comparison between RGB temporal differences and optical flow maps. The optical flow maps were generated by PyFlow [ 34 ] . Temporal differences activate more accurate and sharp cues than optical flows. Besides, the local temporal difference and global temporal difference are not equally in

Experimental results

Research questions

• RQ1 Can explicit temporal difference modeling outperform optical flow-based methods in satellite video super-resolution?
• RQ2 How can short-term and long-term temporal differences be effectively combined to improve motion representation in satellite videos?
• RQ3 What is the impact of bidirectional temporal difference modeling on capturing global motion patterns in satellite video sequences?
• RQ4 How does the Difference Compensation Unit (DCU) improve alignment and spatial consistency in temporal feature learning?
• RQ5 Can a hybrid attention mechanism effectively reconstruct high-fidelity HR frames from combined local and global temporal features?

Key findings

• The proposed LGTD model achieves state-of-the-art performance on five mainstream satellite video datasets, demonstrating superior PSNR and visual quality.
• The bidirectional L-TDM configuration improves PSNR by 0.23dB over single-directional models with only a marginal increase in FLOPs.
• The Difference Compensation Unit (DCU) enhances spatial fidelity, improving PSNR by 0.07dB compared to a baseline without DCU.
• The hybrid attention mechanism in reconstruction outperforms both standalone self-attention and channel attention, achieving the highest PSNR of 35.38 dB on the Jilin-T dataset.
• Ablation studies confirm that both S-TDM and L-TDM contribute significantly to performance, with S-TDM enhancing local texture and L-TDM improving global motion consistency.
• The method effectively reduces redundancy in satellite video by leveraging temporal differences, which provide a more accurate and sparse representation of motion than optical flow.

Figure 2: The overall structure of our proposed Local-Global Temporal Difference learning network (LGTD). It consists of four modules: (1) Short-term Temporal Difference Module (S-TDM), which is used for local temporal compensation; (2) Long-term Temporal Difference Module (L-TDM), which is proposed