[Paper Review] SFFNet: A Wavelet-Based Spatial and Frequency Domain Fusion Network for Remote Sensing Segmentation

Yunsong Yang, Genji Yuan+1 more|arXiv (Cornell University)|May 3, 2024

Remote-Sensing Image Classification4 citations

TL;DR

SFFNet is a two-stage wavelet-based fusion network that enhances remote sensing image segmentation by integrating spatial and frequency domain features. It uses a Wavelet Transform Feature Decomposer (WTFD) for Haar wavelet-based frequency decomposition and a Multiscale Dual-Representation Alignment Filter (MDAF) to align and fuse spatial and frequency features, achieving state-of-the-art performance with 87.73% mIoU on the Potsdam dataset.

ABSTRACT

In order to fully utilize spatial information for segmentation and address the challenge of handling areas with significant grayscale variations in remote sensing segmentation, we propose the SFFNet (Spatial and Frequency Domain Fusion Network) framework. This framework employs a two-stage network design: the first stage extracts features using spatial methods to obtain features with sufficient spatial details and semantic information; the second stage maps these features in both spatial and frequency domains. In the frequency domain mapping, we introduce the Wavelet Transform Feature Decomposer (WTFD) structure, which decomposes features into low-frequency and high-frequency components using the Haar wavelet transform and integrates them with spatial features. To bridge the semantic gap between frequency and spatial features, and facilitate significant feature selection to promote the combination of features from different representation domains, we design the Multiscale Dual-Representation Alignment Filter (MDAF). This structure utilizes multiscale convolutions and dual-cross attentions. Comprehensive experimental results demonstrate that, compared to existing methods, SFFNet achieves superior performance in terms of mIoU, reaching 84.80% and 87.73% respectively.The code is located at https://github.com/yysdck/SFFNet.

Motivation & Objective

• To address the challenge of segmenting remote sensing images in areas with large grayscale variations, such as shadows, edges, and textured regions.
• To preserve spatial detail and semantic information while incorporating frequency domain features for improved robustness.
• To bridge the semantic gap between spatial and frequency domain representations through effective feature alignment and selection.
• To enhance global modeling in remote sensing segmentation by integrating Swin Transformer with multiscale and dual-attention mechanisms.
• To demonstrate the superiority of wavelet-based frequency domain fusion over pure spatial or frequency-only methods.

Proposed method

• The network employs a two-stage design: first, spatial features are extracted using a backbone network enhanced with Swin Transformer for global modeling.
• In the second stage, spatial features are mapped into both spatial and frequency domains using the Wavelet Transform Feature Decomposer (WTFD), which applies Haar wavelet transform to decompose features into low- and high-frequency components.
• The Multiscale Dual-Representation Alignment Filter (MDAF) is introduced to align and fuse features across domains: it uses multiscale vertical convolutions for expansion and dual-cross attention to bridge semantic gaps between frequency and spatial features.
• The MDAF enables selective fusion by emphasizing relevant features from both domains, improving representation learning and feature discrimination.
• The network is trained end-to-end with cross-entropy and dice loss, optimizing for segmentation accuracy on remote sensing benchmarks.
• The architecture is evaluated on Vaihingen and Potsdam datasets, with ablation studies validating component contributions.

Experimental results

Research questions

• RQ1 Can integrating frequency domain features via wavelet transforms improve segmentation accuracy in remote sensing images with large grayscale variations?
• RQ2 How can spatial and frequency domain features be effectively aligned to reduce semantic gap and enable joint representation learning?
• RQ3 Does the proposed MDAF mechanism outperform conventional attention or fusion modules in fusing multiscale spatial and frequency features?
• RQ4 To what extent does combining Swin Transformer with wavelet-based frequency decomposition enhance global context modeling in remote sensing segmentation?
• RQ5 How does SFFNet compare to state-of-the-art methods in terms of mIoU, F1 score, and OA on standard remote sensing benchmarks?

Key findings

• SFFNet achieves 84.80% mIoU on the Vaihingen dataset, outperforming existing methods.
• On the Potsdam dataset, SFFNet achieves 87.73% mIoU, 93.36% F1 score, and 91.88% OA, surpassing ST-Unet by 2.27% in mIoU and XNet by 2.22% in mIoU.
• The ablation study confirms that both the WTFD and MDAF components significantly contribute to performance gains, with MDAF being critical for cross-domain feature alignment.
• Visual comparisons show SFFNet effectively segments challenging regions like shadows, edges, and high-texture areas, while avoiding spatial information loss seen in frequency-only methods like XNet.
• SFFNet demonstrates superior robustness and accuracy in segmenting complex remote sensing scenes, particularly in areas with significant grayscale variations.
• The integration of Haar wavelet transform with deep learning improves generalization and feature discrimination without sacrificing spatial detail.