[论文解读] Can SAM Boost Video Super-Resolution?

Zhihe Lu, Zeyu Xiao+3 more|arXiv (Cornell University)|May 11, 2023

Advanced Image Processing Techniques被引用 14

一句话总结

本论文提出 SEEM，一种轻量级的 SAM 引导的 refined 模块，用于将 Segment Anything Model 的语义先验注入到现有的 VSR 方法（EDVR 和 BasicVSR），通过高效微调提升对齐、融合与重建。

ABSTRACT

The primary challenge in video super-resolution (VSR) is to handle large motions in the input frames, which makes it difficult to accurately aggregate information from multiple frames. Existing works either adopt deformable convolutions or estimate optical flow as a prior to establish correspondences between frames for the effective alignment and fusion. However, they fail to take into account the valuable semantic information that can greatly enhance it; and flow-based methods heavily rely on the accuracy of a flow estimate model, which may not provide precise flows given two low-resolution frames. In this paper, we investigate a more robust and semantic-aware prior for enhanced VSR by utilizing the Segment Anything Model (SAM), a powerful foundational model that is less susceptible to image degradation. To use the SAM-based prior, we propose a simple yet effective module -- SAM-guidEd refinEment Module (SEEM), which can enhance both alignment and fusion procedures by the utilization of semantic information. This light-weight plug-in module is specifically designed to not only leverage the attention mechanism for the generation of semantic-aware feature but also be easily and seamlessly integrated into existing methods. Concretely, we apply our SEEM to two representative methods, EDVR and BasicVSR, resulting in consistently improved performance with minimal implementation effort, on three widely used VSR datasets: Vimeo-90K, REDS and Vid4. More importantly, we found that the proposed SEEM can advance the existing methods in an efficient tuning manner, providing increased flexibility in adjusting the balance between performance and the number of training parameters. Code will be open-source soon.

研究动机与目标

• 探究来自 SAM 的语义先验在大幅运动与降级下是否能提升 VSR 的性能。
• 提出 SEEM，一种将 SAM 得到的掩码与帧特征融合以提升对齐与融合的插件模块。
• 展示 SEEM 与滑动窗口和双向递归 VSR 架构的兼容性。
• 证明 SEEM 在参数高效微调下能带来性能提升。
• 提供有关 SEEM 性能与可训练参数之间权衡的见解。

提出的方法

• 通过对降质的低分辨率帧应用 SAM 并为图像中的对象生成掩码，获得基于 SAM 的表示。
• 设计 SEEM，通过卷积映射和通道注意力块将基于 SAM 的表示与帧特征结合，产生带残差连接的语义感知特征。
• 将 SEEM 集成到 EDVR 中，以改进对齐、融合和重建（用 SEEM 增强的操作替换标准 EDVR 流程的部分）。
• 通过 SEEM 细化扭曲特征和重建表示，将 SEEM 集成到 BasicVSR 中（适用于前向和后向分支）。
• 通过仅训练 SEEM 参数并固定基础 VSR 模型，实现高效微调。

实验结果

研究问题

• RQ1 当处理降级、低分辨率帧时，来自 SAM 的语义掩码能否为 VSR 提供稳健的先验？
• RQ2 SEEM 是否在滑动窗口和双向递归 VSR 框架中改善对齐、融合和重建？
• RQ3 SEEM 是否兼容参数高效微调？性能提升与可训练参数之间的权衡是什么？
• RQ4 SEEM 的改进是否在多个 VSR 数据集（REDS、Vimeo-90K、Vid4）以及跨领域迁移（从 Vimeo-90K 到 Vid4）上具有普适性？

主要发现

• SEEM 在 REDS4、Vimeo-90K 和 Vid4 数据集上持续提升 EDVR 与 BasicVSR。
• 在 REDS4 上，EDVR+SEEM 的 PSNR/SSIM 平均提升最多达到 0.0254/0.00094，BasicVSR+SEEM 提升最多达到 0.0877/0.00131。
• 在 Vimeo-90K 上，EDVR+SEEM 的 PSNR/SSIM 平均提升为 0.0421/0.00036，BasicVSR+SEEM 为 0.1184/0.00102。
• SEEM 使得只更新 SEEM 参数即可实现高效微调，在较少的可训练参数下实现显著收益。
• SEEM 在从 Vimeo-90K 训练迁移到 Vid4 评估时提高泛化能力（Table 4 结果显示持续收益）。
• 消融研究显示将 SEEM 添加到 BasicVSR 的任一 Forward 或 Backward 分支均有益，且同时添加到两者时获得最佳结果。