[论文解读] Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks

Danfeng Hong, Bing Zhang+8 more|arXiv (Cornell University)|Sep 26, 2023

Remote-Sensing Image Classification被引用 8

一句话总结

该论文为跨城市语义分割提出了 C2Seg 多模态遥感基准及 HighDAN 网络，后者基于高分辨率 HRNet 的多模态编码器，结合对抗域自适应和 Dice 损失，以提升跨城市泛化能力。

ABSTRACT

Artificial intelligence (AI) approaches nowadays have gained remarkable success in single-modality-dominated remote sensing (RS) applications, especially with an emphasis on individual urban environments (e.g., single cities or regions). Yet these AI models tend to meet the performance bottleneck in the case studies across cities or regions, due to the lack of diverse RS information and cutting-edge solutions with high generalization ability. To this end, we build a new set of multimodal remote sensing benchmark datasets (including hyperspectral, multispectral, SAR) for the study purpose of the cross-city semantic segmentation task (called C2Seg dataset), which consists of two cross-city scenes, i.e., Berlin-Augsburg (in Germany) and Beijing-Wuhan (in China). Beyond the single city, we propose a high-resolution domain adaptation network, HighDAN for short, to promote the AI model's generalization ability from the multi-city environments. HighDAN is capable of retaining the spatially topological structure of the studied urban scene well in a parallel high-to-low resolution fusion fashion but also closing the gap derived from enormous differences of RS image representations between different cities by means of adversarial learning. In addition, the Dice loss is considered in HighDAN to alleviate the class imbalance issue caused by factors across cities. Extensive experiments conducted on the C2Seg dataset show the superiority of our HighDAN in terms of segmentation performance and generalization ability, compared to state-of-the-art competitors. The C2Seg dataset and the semantic segmentation toolbox (involving the proposed HighDAN) will be available publicly at https://github.com/danfenghong.

研究动机与目标

• 通过多模态遥感数据推动跨城市语义分割，突破跨城市与区域的泛化瓶颈。
• 提供一个大型、公开可用的多模态遥感基准 (C2Seg)，包含两个跨城市场景与13类土地覆盖。
• 开发 HighDAN，一种高分辨率的多模态域自适应网络，用于跨城市知识转移。
• 证明域自适应和 Dice 损失在提升跨城市分割性能方面优于最新基线方法。

提出的方法

• 将 C2Seg 定义为两个跨城市的数据集：C2Seg-AB（德国柏林-奥格斯堡）和 C2Seg-BW（中国北京-武汉），数据为 10 m GSD 的高光谱、多光谱和 SAR 数据。
• 设计 HighDAN，一种基于 HR-Net 的高分辨率网络，将并行的高到低分辨率流融合以获得多模态表征。
• 实现一个多模态编码器，配备特征提取头和多模态 HR 子网络，以学习跨模态的高分辨率表征。
• 在特征层和类别层均嵌入对抗域自适应模块，以对齐源域与目标域的表征。
• 嵌入 Dice 损失以缓解跨城市类别不平衡问题，并提升跨域分割的鲁棒性。
• 采用三流架构处理高光谱、多光谱和 SAR 数据，共享 HR 模块参数以保持稳定性。

实验结果

研究问题

• RQ1 多模态遥感基准数据集如何支持跨城市语义分割研究？
• RQ2 通过保持高分辨率表征并利用域自适应，HighDAN 是否能提升跨城市泛化？
• RQ3 在特征层和类别层的对抗域自适应是否能降低多模态遥感数据中城市之间的域差距？
• RQ4 Dice 损失是否能缓解跨城市土地覆盖分割中的类别不平衡问题？

主要发现

• HighDAN 在 C2Seg 数据集的跨城市语义分割上优于最新基线方法。
• 多模态 HR 融合策略在保持空间拓扑的同时，通过对抗域自适应实现跨城市迁移。
• Dice 损失有助于缓解跨城市分割任务固有的类别不平衡问题。
• C2Seg 提供公开可用的跨城市多模态遥感分割基准，覆盖两对城市和三种模态。
• 数据集与 HighDAN 工具箱计划公开发布，便于广泛研究使用。