[论文解读] Local-Global Temporal Difference Learning for Satellite Video Super-Resolution

Yi Xiao, Qiangqiang Yuan+5 more|arXiv (Cornell University)|Apr 10, 2023

Advanced Vision and Imaging参考文献 68被引用 4

一句话总结

该论文提出了一种名为LGTD的新方法，用于卫星视频超分辨率，通过利用局部和全局时间差异实现有效的运动补偿。通过建模相邻帧之间的短期RGB差异以及序列中长期的双向差异，LGTD提升了特征对齐与重建保真度，在五个卫星视频数据集上实现了最先进性能，相较于单向基线方法，PSNR最高提升0.23dB。

ABSTRACT

Optical-flow-based and kernel-based approaches have been extensively explored for temporal compensation in satellite Video Super-Resolution (VSR). However, these techniques are less generalized in large-scale or complex scenarios, especially in satellite videos. In this paper, we propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies since we observed that these discrepancies offer distinct and mutually complementary properties. Specifically, we devise a Short-term Temporal Difference Module (S-TDM) to extract local motion representations from RGB difference maps between adjacent frames, which yields more clues for accurate texture representation. To explore the global dependency in the entire frame sequence, a Long-term Temporal Difference Module (L-TDM) is proposed, where the differences between forward and backward segments are incorporated and activated to guide the modulation of the temporal feature, leading to a holistic global compensation. Moreover, we further propose a Difference Compensation Unit (DCU) to enrich the interaction between the spatial distribution of the target frame and temporal compensated results, which helps maintain spatial consistency while refining the features to avoid misalignment. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches. Code will be available at https://github.com/XY-boy/LGTD

研究动机与目标

• 为解决由于静态背景和复杂运动导致的卫星视频超分辨率中时间补偿不准确的问题。
• 克服光流法与基于核的方法在处理大规模、冗余的卫星视频序列时的局限性。
• 将时间差异（局部与全局）作为比光流更鲁棒的运动表征方式，以替代传统方法。
• 通过引入差异补偿单元（DCU），提升超分辨率重建过程中的空间一致性和特征保真度。
• 通过混合注意力机制实现卫星视频超分辨率的最先进性能。

提出的方法

• 提出一种短期时间差异模块（S-TDM），通过计算相邻帧之间的RGB差异，提取局部运动表征。
• 引入一种长期时间差异模块（L-TDM），通过建模前向与后向片段之间的双向差异，捕捉全局时间依赖性。
• 在计算整体时间差异以实现长距离补偿前，对平滑的前向与后向特征进行处理，以减少冗余。
• 采用差异补偿单元（DCU）通过对齐空间与时间表征，引导时间特征调制，降低错位现象。
• 结合自注意力与通道注意力的混合注意力机制，从融合的局部与全局特征中重建高质量的HR帧。
• 设计一种两阶段特征融合策略，将S-TDM与L-TDM的输出与目标帧进行融合，以提升时空一致性。

Figure 1: Comparison between RGB temporal differences and optical flow maps. The optical flow maps were generated by PyFlow [ 34 ] . Temporal differences activate more accurate and sharp cues than optical flows. Besides, the local temporal difference and global temporal difference are not equally in

实验结果

研究问题

• RQ1 显式的时间差异建模是否能在卫星视频超分辨率中超越基于光流的方法？
• RQ2 如何有效结合短期与长期时间差异，以提升卫星视频中的运动表征能力？
• RQ3 双向时间差异建模对捕捉卫星视频序列中全局运动模式有何影响？
• RQ4 差异补偿单元（DCU）如何提升时间特征学习中的对齐性与空间一致性？
• RQ5 混合注意力机制是否能有效从融合的局部与全局时间特征中重建出高保真度的HR帧？

主要发现

• 所提出的LGTD模型在五个主流卫星视频数据集上实现了最先进性能，展现出更优的PSNR与视觉质量。
• 双向L-TDM配置相较于单向模型仅带来微小的FLOPs增加，但PSNR提升达0.23dB。
• 差异补偿单元（DCU）增强了空间保真度，相较于无DCU的基线，PSNR提升0.07dB。
• 重建阶段采用的混合注意力机制优于单独使用自注意力或通道注意力，使吉隆-T数据集的PSNR达到最高值35.38 dB。
• 消融实验表明，S-TDM与L-TDM均对性能有显著贡献：S-TDM提升局部纹理质量，L-TDM增强全局运动一致性。
• 该方法通过利用时间差异有效减少了卫星视频中的冗余，时间差异提供了比光流更精确、更稀疏的运动表征。

Figure 2: The overall structure of our proposed Local-Global Temporal Difference learning network (LGTD). It consists of four modules: (1) Short-term Temporal Difference Module (S-TDM), which is used for local temporal compensation; (2) Long-term Temporal Difference Module (L-TDM), which is proposed