[Paper Review] Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2

Osher Rafaeli, Tal Svoray+2 more|arXiv (Cornell University)|Aug 13, 2024

Color perception and designPsychology3 citations

TL;DR

This study evaluates prompt-based segmentation using Segment Anything Model 2 (SAM 2) and conventional CNNs (Eff-UNet) for solar panel detection in aerial imagery across varying resolutions (0.15 m and 0.25 m) and lighting conditions. SAM 2 outperforms SAM, especially under sub-optimal lighting, and YOLOv9-generated bounding boxes as prompts significantly improve accuracy over human-clicked points, while Eff-UNet slightly surpasses SAM 2 with YOLO prompts in high-resolution data due to reduced false positives from prompt bias.

ABSTRACT

This paper provides insights on the effectiveness of the zero shot, prompt-based Segment Anything Model (SAM) and its updated versions, SAM 2 and SAM 2.1, along with the non-promptable conventional neural network (CNN), for segmenting solar panels in RGB aerial remote sensing imagery. The study evaluates these models across diverse lighting conditions, spatial resolutions, and prompt strategies. SAM 2 showed slight improvements over SAM, while SAM 2.1 demonstrated notable improvements, particularly in sub-optimal lighting and low resolution conditions. SAM models, when prompted by user-defined boxes, outperformed CNN in all scenarios; in particular, user-box prompts were found crucial for achieving reasonable performance in low resolution data. Additionally, under high resolution, YOLOv9 automatic prompting outperformed user-points prompting by providing reliable prompts to SAM. Under low resolution, SAM 2.1 prompted by user points showed similar performance to SAM 2.1 prompted by YOLOv9, highlighting its zero shot improvements with a single click. In high resolution with optimal lighting imagery, Eff-UNet outperformed SAMs prompted by YOLOv9, while under sub-optimal lighting conditions, Eff-UNet, and SAM 2.1 prompted by YOLOv9, had similar performance. However, SAM is more resource-intensive, and despite improved inference time of SAM 2.1, Eff-UNet is more suitable for automatic segmentation in high resolution data. This research details strengths and limitations of each model and outlines the robustness of user-prompted image segmentation models.

Motivation & Objective

• To evaluate the performance of prompt-based SAM 2 and conventional CNNs (Eff-UNet) in segmenting solar panels across diverse remote sensing image conditions.
• To assess the impact of different prompt strategies—user-created points, user-created bounding boxes, and YOLOv9-generated boxes—on segmentation accuracy in SAM and SAM 2.
• To compare fully automatic (Eff-UNet) versus semi-automatic (SAM/SAM 2 with human prompts) approaches under low- and high-resolution, optimal- and sub-optimal-lighting conditions.
• To identify the strengths and limitations of each model in handling inconsistent remote sensing data, particularly in terms of robustness to resolution and lighting variation.

Proposed method

• The study uses three airborne datasets from Be’er Sheva, Israel (2015, 2017, 2022), with 0.25 m (low) and 0.15 m (high) spatial resolutions and varying lighting conditions.
• Segmentation models evaluated include: (1) SAM 2 and SAM (prompt-based), (2) Eff-UNet (fully automatic CNN), and (3) SAM/SAM 2 prompted by YOLOv9 object detection boxes.
• Prompt strategies tested: user-clicked points, user-drawn bounding boxes, and YOLOv9-generated bounding boxes for SAM and SAM 2.
• Performance is evaluated using IoU, F1-score, precision, recall, and accuracy across all datasets, with 3000 labeled solar panels per dataset.
• The YOLOv9 model is used to generate object proposals as prompts for SAM and SAM 2, enabling end-to-end automated prompting.
• Eff-UNet uses a compound-scaled EfficientNet encoder with a U-Net decoder, trained end-to-end on ImageNet-pretrained features for segmentation.

Experimental results

Research questions

• RQ1 How does SAM 2 compare to SAM in segmenting solar panels under sub-optimal lighting and low-resolution conditions?
• RQ2 Does YOLOv9-generated bounding box prompts outperform human-clicked point prompts in SAM and SAM 2 for solar panel segmentation?
• RQ3 How does the performance of fully automatic Eff-UNet compare to prompt-based SAM and SAM 2 across different image resolutions and lighting conditions?
• RQ4 What is the impact of prompt quality and model architecture on false positive rates in remote sensing image segmentation?

Key findings

• SAM 2 achieved an IoU of 0.75 and F1-score of 0.85 when prompted by user boxes, significantly outperforming SAM (IoU: 0.74, F1: 0.84) under sub-optimal lighting.
• SAM 2 outperformed SAM when prompted by points, achieving an IoU of 0.51 and F1-score of 0.62, compared to SAM’s 0.49 IoU and 0.61 F1-score.
• YOLOv9-generated boxes as prompts led to higher accuracy than human-clicked points, with SAM 2 achieving an IoU of 0.54 and F1-score of 0.63, compared to 0.49 and 0.61 for point prompts.
• In high-resolution imagery, Eff-UNet achieved an IoU of 0.55 and F1-score of 0.64, slightly outperforming SAM 2 with YOLOv9 prompts (IoU: 0.54, F1: 0.63), indicating better robustness to prompt bias.
• Under low-resolution conditions, user-box prompts were essential for achieving reasonable performance, as SAM models without such prompts struggled with segmentation accuracy.
• SAM models were more sensitive to prompt quality, with false positive rates increasing significantly when fed imperfect YOLOv9 boxes, while Eff-UNet showed more consistent performance across all conditions.