[Paper Review] IRSAM: Advancing Segment Anything Model for Infrared Small Target Detection

Mingjin Zhang, Yuchun Wang+4 more|arXiv (Cornell University)|Jul 10, 2024

Infrared Target Detection Methodologies4 citations

TL;DR

This paper proposes IRSAM, a novel adaptation of the Segment Anything Model (SAM) for infrared small target detection (IRSTD), enhancing feature representation through a Wavelet-based Perona-Malik Diffusion (WPMD) block in the encoder and a Granularity-Aware Decoder (GAD) for multi-granularity feature fusion. The method achieves state-of-the-art performance on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K benchmarks, significantly outperforming vanilla SAM and prior SOTA methods.

ABSTRACT

The recent Segment Anything Model (SAM) is a significant advancement in natural image segmentation, exhibiting potent zero-shot performance suitable for various downstream image segmentation tasks. However, directly utilizing the pretrained SAM for Infrared Small Target Detection (IRSTD) task falls short in achieving satisfying performance due to a notable domain gap between natural and infrared images. Unlike a visible light camera, a thermal imager reveals an object's temperature distribution by capturing infrared radiation. Small targets often show a subtle temperature transition at the object's boundaries. To address this issue, we propose the IRSAM model for IRSTD, which improves SAM's encoder-decoder architecture to learn better feature representation of infrared small objects. Specifically, we design a Perona-Malik diffusion (PMD)-based block and incorporate it into multiple levels of SAM's encoder to help it capture essential structural features while suppressing noise. Additionally, we devise a Granularity-Aware Decoder (GAD) to fuse the multi-granularity feature from the encoder to capture structural information that may be lost in long-distance modeling. Extensive experiments on the public datasets, including NUAA-SIRST, NUDT-SIRST, and IRSTD-1K, validate the design choice of IRSAM and its significant superiority over representative state-of-the-art methods. The source code are available at: github.com/IPIC-Lab/IRSAM.

Motivation & Objective

• To address the domain gap between natural images and infrared images that limits the performance of pre-trained SAM in infrared small target detection (IRSTD).
• To improve feature representation in infrared images, particularly for small targets with low signal-to-noise ratio and blurred edges.
• To enhance edge preservation and noise suppression in the encoder while improving multi-scale and multi-granularity feature fusion in the decoder.
• To achieve superior zero-shot and fine-tuned performance on IRSTD benchmarks using transfer learning with a lightweight, efficient architecture.

Proposed method

• The WPMD module replaces the gradient term in the Perona-Malik diffusion equation with a wavelet-based transform to better preserve edges and suppress noise in infrared images.
• The WPMD block is integrated into multiple stages of SAM’s vision transformer encoder to refine feature extraction at different levels of abstraction.
• A Granularity-Aware Decoder (GAD) fuses features from both shallow and deep encoder layers via a two-way transformer to enhance mask representation across varying object scales and shapes.
• The model is built upon Mobile-SAM to reduce computational cost and model size while maintaining high performance.
• The encoder uses a modified ViT-Tiny backbone with WPMD blocks inserted at multiple layers to improve structural feature learning.
• The decoder employs a two-way attention mechanism to integrate multi-granularity features, improving segmentation accuracy for small and complex targets.

Experimental results

Research questions

• RQ1 Can the pre-trained Segment Anything Model (SAM) be effectively adapted for infrared small target detection despite the domain gap between natural and infrared images?
• RQ2 How can edge preservation and noise suppression be improved in infrared images with subtle temperature transitions and low signal-to-noise ratios?
• RQ3 What is the optimal way to fuse multi-granularity features from the encoder to enhance mask representation for small and structurally complex targets?
• RQ4 How does the integration of a wavelet-based diffusion module and a multi-granularity decoder affect performance compared to vanilla SAM and existing SOTA methods?

Key findings

• IRSAM achieves an IoU score of 80.78 on the NUAA-SIRST dataset, significantly outperforming the Mobile-SAM baseline (75.84) and other SOTA methods.
• The ablation study confirms that removing the WPMD module reduces IoU by 2.05 points and increases false alarm rate (Fa) from 3.95 to 11.18, demonstrating its critical role in edge preservation.
• The GAD module, which fuses both shallow and deep features, achieves the highest performance (IoU: 80.78, nIoU: 78.39), outperforming configurations using only shallow or deep features.
• Using four WPMD blocks in the encoder yields the best performance, with feature maps showing stronger target activation and reduced noise compared to fewer blocks.
• Visual comparisons confirm that IRSAM preserves fine structural details and suppresses background clutter better than Laplacian and Sobel-based edge detection methods.
• On the IRSTD-1K and NUDT-SIRST datasets, IRSAM achieves state-of-the-art results in both objective metrics (IoU, nIoU) and subjective evaluation, confirming its robustness across diverse infrared imaging conditions.