[论文解读] A Billion-scale Foundation Model for Remote Sensing Images

Keumgang Cha, Junghoon Seo+1 more|arXiv (Cornell University)|Apr 11, 2023

Advanced Neural Network Applications被引用 16

一句话总结

该论文研究在 MillionAID 上使用 MAE 进行预训练的 vision transformer 增加参数数量，如何影响下游遥感任务，结果显示千亿级模型在旋转对象检测和语义分割上提升，并在若干基准上达到最先进水平。

ABSTRACT

As the potential of foundation models in visual tasks has garnered significant attention, pretraining these models before downstream tasks has become a crucial step. The three key factors in pretraining foundation models are the pretraining method, the size of the pretraining dataset, and the number of model parameters. Recently, research in the remote sensing field has focused primarily on the pretraining method and the size of the dataset, with limited emphasis on the number of model parameters. This paper addresses this gap by examining the effect of increasing the number of model parameters on the performance of foundation models in downstream tasks such as rotated object detection and semantic segmentation. We pretrained foundation models with varying numbers of parameters, including 86M, 605.26M, 1.3B, and 2.4B, to determine whether performance in downstream tasks improved with an increase in parameters. To the best of our knowledge, this is the first billion-scale foundation model in the remote sensing field. Furthermore, we propose an effective method for scaling up and fine-tuning a vision transformer in the remote sensing field. To evaluate general performance in downstream tasks, we employed the DOTA v2.0 and DIOR-R benchmark datasets for rotated object detection, and the Potsdam and LoveDA datasets for semantic segmentation. Experimental results demonstrated that, across all benchmark datasets and downstream tasks, the performance of the foundation models and data efficiency improved as the number of parameters increased. Moreover, our models achieve the state-of-the-art performance on several datasets including DIOR-R, Postdam, and LoveDA.

研究动机与目标

• 研究模型规模（参数数量）对遥感基础模型的影响。
• 在 MillionAID 上使用 MAE 进行预训练以检验扩展效应。
• 评估下游在旋转对象检测和语义分割基准上的性能。
• 展示高分辨率遥感任务中对 Vision Transformer 的有效扩展与微调。

提出的方法

• 在 MillionAID 上对多种参数规模（86M、605.26M、1.3B、2.4B）的 Vision Transformer 主干进行 MAE 预训练。
• 通过调整隐藏层大小、MLP 大小、头数和并行度来扩展 ViT，同时保持 12 层以研究参数对性能的影响。
• 将简单的 ViT 预训练主干改造为 ViTDET，并引入局部/全局注意力，以用于下游任务。
• 使用缩放块（转置卷积、归一化、GELU、池化）来对高分辨率任务进行特征上采样/下采样。
• 在旋转对象检测（DOTA v2.0、DIOR-R）和语义分割（Potsdam、LoveDA）上进行微调。
• 预训练后再进行微调设置包括 MAE 重建 75% 的被遮挡 Patch、400 个预训练时代、AdamW，以及 fp16 + activation checkpointing。

Figure 2: A brief introduction of self supervised learning, such as contrastive learning, self-distillation and masked image modeling in computer vision. In contrastive learning, the positive pairs of data are brought closer together while the negative pairs are pushed further apart. Self-distillati

实验结果

研究问题

• RQ1 增加模型参数数量是否会提升遥感基础模型的下游性能？
• RQ2 在遥感数据集上以 MAE 进行预训练的亿参数Vision Transformer 是否能在旋转对象检测和语义分割上优于较小的模型？
• RQ3 哪些架构改造（ViTDET、尺度块、并行注意力）对遥感定位任务有效？
• RQ4 在标准 RS 基准上，是否存在数据效率提升随参数量增大而显现的证据？
• RQ5 亿级遥感基础模型是否在 DIOR-R、Potsdam 和 LoveDA 上达到最先进结果？

主要发现

• 随着参数数量的增加，所有基准和下游任务的性能均有所提升。
• 亿级规模（2.4B 参数）模型在若干 RS 数据集（包括 DIOR-R、Potsdam、LoveDA）上达到最先进的性能。
• 通过并行性和调整后的隐藏层/MLP 大小扩展 ViT 能有效支持对象定位任务，如旋转对象检测和语义分割。
• 在 MillionAID 上进行的 MAE 预训练为下游 RS 任务提供了强烈的领域内表示，使微调数据效率更高。
• 基于 ViTDET 的下游头部通过局部/全局注意力在高分辨率 RS 输入中平衡了计算和内存使用。

Figure 3: This figure explains how to effectively increase the number of parameters of the vision transformer, and the two models have substantially the same amount of computation and number of parameters. In the field of natural language processing, multi head self attention and feed forward blocks