[Paper Review] PointSAM: Pointly-Supervised Segment Anything Model for Remote Sensing Images

Nanqing Liu, Xun Xu+3 more|arXiv (Cornell University)|Sep 20, 2024

Big Data and Business Intelligence4 citations

TL;DR

This paper proposes PointSAM, a pointly-supervised adaptation of the Segment Anything Model for remote sensing images, using a self-training framework with prototype-based regularization and negative prompt calibration to mitigate error accumulation and dense-object interference. It achieves state-of-the-art performance on NWPU VHR-10, HRSID, and WHU datasets using only point annotations, outperforming SAM, SAM2, and prior weakly supervised methods.

ABSTRACT

Segment Anything Model (SAM) is an advanced foundational model for image segmentation, which is gradually being applied to remote sensing images (RSIs). Due to the domain gap between RSIs and natural images, traditional methods typically use SAM as a source pre-trained model and fine-tune it with fully supervised masks. Unlike these methods, our work focuses on fine-tuning SAM using more convenient and challenging point annotations. Leveraging SAM's zero-shot capabilities, we adopt a self-training framework that iteratively generates pseudo-labels for training. However, if the pseudo-labels contain noisy labels, there is a risk of error accumulation. To address this issue, we extract target prototypes from the target dataset and use the Hungarian algorithm to match them with prediction prototypes, preventing the model from learning in the wrong direction. Additionally, due to the complex backgrounds and dense distribution of objects in RSI, using point prompts may result in multiple objects being recognized as one. To solve this problem, we propose a negative prompt calibration method based on the non-overlapping nature of instance masks. In brief, we use the prompts of overlapping masks as corresponding negative signals, resulting in refined masks. Combining the above methods, we propose a novel Pointly-supervised Segment Anything Model named PointSAM. We conduct experiments on RSI datasets, including WHU, HRSID, and NWPU VHR-10, and the results show that our method significantly outperforms direct testing with SAM, SAM2, and other comparison methods. Furthermore, we introduce PointSAM as a point-to-box converter and achieve encouraging results, suggesting that this method can be extended to other point-supervised tasks. The code is available at https://github.com/Lans1ng/PointSAM.

Motivation & Objective

• To address the domain gap between natural images and remote sensing images (RSIs), where SAM's zero-shot performance degrades due to out-of-distribution objects and complex backgrounds.
• To enable effective fine-tuning of SAM using only point annotations—cheaper and more scalable than full mask or box annotations—while avoiding error accumulation in self-training.
• To overcome challenges in dense, cluttered RSIs where point prompts lead to multiple objects being merged into a single mask.
• To develop a method that generalizes well to weakly-supervised segmentation and can be extended to other tasks like point-to-box detection.

Proposed method

• Adopts a self-training framework where the model generates pseudo-labels from weakly-augmented inputs, enabling iterative refinement without full supervision.
• Introduces prototype-based regularization: extracts target-specific prototypes from the dataset and matches them with prediction prototypes using the Hungarian algorithm to stabilize training and reduce error accumulation.
• Proposes negative prompt calibration: uses overlapping mask predictions as negative signals to refine segmentation, leveraging the non-overlapping nature of instance masks in RSIs.
• Utilizes a dual-branch network with weak and strong augmentations to generate consistent predictions and improve robustness during self-training.
• Employs a point-to-box conversion pipeline: generates masks from point prompts, computes minimum enclosing rectangles to produce horizontal bounding boxes, and feeds them into a rotated detector.
• Uses ResNet-50 as the backbone and trains for 12 epochs on HRSID with input size 800×800 to evaluate detection performance.

Experimental results

Research questions

• RQ1 Can a self-training framework effectively fine-tune SAM for remote sensing images using only point annotations, despite noisy pseudo-labels?
• RQ2 How can prototype-based regularization improve training stability and reduce error accumulation in self-training for domain-shifted segmentation?
• RQ3 Can negative prompt calibration effectively separate densely distributed objects in RSIs that are otherwise merged under point prompts?
• RQ4 To what extent can PointSAM serve as a point-to-box generator for oriented object detection, and how does it compare to fully supervised baselines?

Key findings

• PointSAM achieves state-of-the-art performance on three benchmark RSI datasets—NWPU VHR-10, HRSID, and WHU—surpassing vanilla SAM, SAM2, and prior weakly-supervised methods like WeSAM and DePT.
• On the HRSID dataset, PointSAM + H2RBox-v2 achieves 68.9% recall and 59.5% AP50, a 15% improvement over vanilla SAM and outperforms Point2RBox in point-supervised detection.
• The prototype-based regularization significantly reduces error accumulation, as evidenced by more accurate and stable mask predictions across diverse RSI scenes.
• Negative prompt calibration effectively separates adjacent and overlapping objects, such as ships in inshore scenes and storage tanks with shadows, which other methods fail to distinguish.
• The method demonstrates strong generalization: even with only point supervision, it produces masks close to ground truth, especially in dense and complex scenes.
• The point-to-box conversion pipeline enables effective point-supervised oriented object detection, with performance approaching HBB-supervised methods, though a ~20% gap remains compared to OBB-supervised baselines.