[论文解读] Zooming Out on Zooming In: Advancing Super-Resolution for Remote Sensing

Piper Wolters, Favyen Bastani+1 more|arXiv (Cornell University)|Nov 29, 2023

Cryospheric studies and observations被引用 8

一句话总结

论文提出 CLIPScore 作为遥感超分辨率的感知度量，构建大规模 S2-NAIP 数据集，比较 GAN、扩散与 CNN 基方法，并显示 GANs（ESRGAN）在 CLIPScore 下优于其他方法，而超分输出对机器端使用效果尚不理想；同时训练了一个全球模型并发布数据与代码。

ABSTRACT

Super-Resolution for remote sensing has the potential for huge impact on planet monitoring by producing accurate and realistic high resolution imagery on a frequent basis and a global scale. Despite a lot of attention, several inconsistencies and challenges have prevented it from being deployed in practice. These include the lack of effective metrics, fragmented and relatively small-scale datasets for training, insufficient comparisons across a suite of methods, and unclear evidence for the use of super-resolution outputs for machine consumption. This work presents a new metric for super-resolution, CLIPScore, that corresponds far better with human judgments than previous metrics on an extensive study. We use CLIPScore to evaluate four standard methods on a new large-scale dataset, S2-NAIP, and three existing benchmark datasets, and find that generative adversarial networks easily outperform more traditional L2 loss-based models and are more semantically accurate than modern diffusion models. We also find that using CLIPScore as an auxiliary loss can speed up the training of GANs by 18x and lead to improved outputs, resulting in an effective model in diverse geographies across the world which we will release publicly. The dataset, pre-trained model weights, and code are available at https://github.com/allenai/satlas-super-resolution/.

研究动机与目标

• 建立一个与人类判断一致的遥感超分辨率输出度量
• 创建规模化、公开域的数据集（S2-NAIP），以促进可扩展的训练与评估
• 系统性比较多种遥感超分方法（CNN、GAN、扩散）在多个数据集上的表现
• 评估超分输出对下游机器任务与人为可视化的有用性
• 开发并部署一个全球可用的超分模型，结合上述发现

提出的方法

• 提出 CLIPScore，一种基于 CLIP 的图像相似度度量，与人类判断的 SR 质量相关
• 构建并发布 S2-NAIP 数据集，包含来自 Sentinel-2 与 NAIP 图像的 120 万对 LR-HR 对
• 在四个数据集上基准测试 SR 方法（SRCNN、HighResNet、ESRGAN、SR3），通过 CLIPScore、LPIPS、cPSNR 评估感知质量
• 研究 SR 输出或特征是否有助于下游任务，并利用 SR 表征分析迁移学习
• 通过引入基于 CLIPScore 的辅助损失来提升 ESRGAN 的训练速度和输出质量

Figure 2 : Example of a target image (GT), an ESRGAN output at full resolution as well as downsampled 16x, and a HighResNet output, with corresponding metrics. Note that the four images are ordered from best to worst based on human preference, and PSNR and SSIM increase in an opposite trend. Our pro

实验结果

研究问题

• RQ1 CLIPScore 与遥感 SR 输出的人类判断相关性有多强
• RQ2 数据集规模（S2-NAIP）对不同方法的 SR 性能有何影响
• RQ3 哪一类 SR 方法（CNN、GAN、扩散）在遥感数据上具有最佳感知质量
• RQ4 SR 输出对下游机器任务是否有帮助，还是主要用于人类可视化
• RQ5 是否可通过 CLIP 基辅助损失与领域知识改进 SR 训练以获得更好结果

主要发现

• PSNR 与 SSIM 与遥感领域的人类判断相关性较差
• CLIPScore（特别是 CLIPA-v2）在 SR 质量上与人类偏好的一致性最高可达 84.6%
• GANs（ESRGAN）在四个数据集上的 CLIPScore 表现优于扩散和基于 L2 的模型
• 用 CLIPScore 损失训练 ESRGAN 可将训练速度提升约 18 倍，CLIPScore 提升约 9 点
• SR 输出尚未优于直接使用原始 LR 图像在下游机器任务中的表现，然而 SR 表征可提升迁移学习
• 大规模的 S2-NAIP 数据集与全球 ESRGAN 模型实现 SR 输出的可扩展公开部署

Figure 3 : The level of accuracy between human preferences and those generated by the various metrics. The x-axis is ordered from worst to best average accuracy between the two datasets. The y-axis is adjusted to a range of 40% to 90% to better show the difference in accuracy across metrics.