[Paper Review] Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model

Di Wang, Qiming Zhang+5 more|arXiv (Cornell University)|Aug 8, 2022

Remote-Sensing Image Classification38 citations

TL;DR

The paper pretrains plain Vision Transformers (ViTs) with MAE on a large RS dataset (~100M parameters) and introduces rotated varied-size window attention (RVSA) to adapt them for RS tasks, achieving state-of-the-art object detection on DOTA-V1.0 while offering competitive classification and segmentation results and improved data efficiency.

ABSTRACT

Large-scale vision foundation models have made significant progress in visual tasks on natural images, with vision transformers being the primary choice due to their good scalability and representation ability. However, large-scale models in remote sensing (RS) have not yet been sufficiently explored. In this paper, we resort to plain vision transformers with about 100 million parameters and make the first attempt to propose large vision models tailored to RS tasks and investigate how such large models perform. To handle the large sizes and objects of arbitrary orientations in RS images, we propose a new rotated varied-size window attention to replace the original full attention in transformers, which can significantly reduce the computational cost and memory footprint while learning better object representation by extracting rich context from the generated diverse windows. Experiments on detection tasks show the superiority of our model over all state-of-the-art models, achieving 81.24% mAP on the DOTA-V1.0 dataset. The results of our models on downstream classification and segmentation tasks also show competitive performance compared to existing advanced methods. Further experiments show the advantages of our models in terms of computational complexity and data efficiency in transferring.

Motivation & Objective

• Demonstrate the feasibility of pretraining plain ViTs (~100M parameters) on remote sensing data for RS tasks.
• Investigate whether a non-hierarchical (plain) ViT can achieve competitive RS task performance with suitable pretraining.
• Develop RVSA to handle arbitrary orientations and scales in RS images while reducing computation.
• Evaluate the transferability, efficiency, and robustness of the pretrained plain ViTs on RS detection, classification, and segmentation tasks.

Proposed method

• Pretrain plain ViT and ViTAE backbones (~100M params) with MAE on MillionAID in an unlabeled setting.
• Replace full self-attention with Rotated Varied-Size Attention (RVSA) during finetuning to handle RS data with arbitrary orientations.
• Introduce a rotation angle in the learned window configurations, enabling oriented, varied-size attention windows.
• Replace MHSA with RVSA (and variants) in selected layers to form RS backbones suitable for downstream tasks.
• Train and evaluate on RS tasks including scene classification (UCM, AID, NWPU), object detection (DOTA-V1.0, DIOR-R), and segmentation, using standard RS frameworks.

Experimental results

Research questions

• RQ1 Can a plain ViT backbone pretrained with MAE on RS data achieve competitive results on RS tasks without hierarchical structuring?
• RQ2 Does RVSA improve the ability of ViTs to model RS images with objects at arbitrary orientations and scales compared to fixed-window attention?
• RQ3 What is the impact of pretraining scale and mask ratio on RS downstream performance for plain ViTs?
• RQ4 How do plain ViTs with RVSA compare to state-of-the-art RS models in terms of accuracy, efficiency, and transferability?

Key findings

• Pretraining plain ViTs (ViT-B and ViTAE-B) with MAE on MillionAID yields competitive RS task performance after finetuning.
• RVSA significantly improves RS object detection by enabling rotated, varied-size windows for attention, achieving 81.24% mAP on DOTA-V1.0.
• RVSA-based variants show strong performance on RS scene classification and segmentation tasks, with competitive results against existing advanced methods.
• The approach demonstrates advantages in computational complexity and data efficiency during transfer to RS tasks.
• A window size of 7 for RVSA achieves peak mAP on DOTA-V1.0 and DIOR-R in ablations, showing the importance of appropriate window configuration.
• The method scales to large RS images by using window-based attention to reduce FLOPs and memory while maintaining rich contextual modeling.
• The work positions plain ViTs as viable RS foundation-model backbones with effective pretraining and a specialized attention mechanism for RS characteristics.