[Paper Review] Transformer Meets Convolution: A Bilateral Awareness Network for Semantic Segmentation of Very Fine Resolution Urban Scene Images

Libo Wang, Rui Li+4 more|arXiv (Cornell University)|Jun 23, 2021

Automated Road and Building Extraction55 references9 citations

TL;DR

This paper proposes Bilateral Awareness Network (BANet), a novel architecture that fuses Transformer-based long-range dependency modeling with convolutional feature extraction to improve semantic segmentation in very fine resolution (VFR) urban scene images. By integrating a ResT-based dependency path and a stacked convolutional texture path with a linear attention fusion module, BANet achieves 64.6% mIoU on the UAVid dataset, demonstrating state-of-the-art performance on large-scale urban segmentation benchmarks.

ABSTRACT

Semantic segmentation from very fine resolution (VFR) urban scene images plays a significant role in several application scenarios including autonomous driving, land cover classification, and urban planning, etc. However, the tremendous details contained in the VFR image, especially the considerable variations in scale and appearance of objects, severely limit the potential of the existing deep learning approaches. Addressing such issues represents a promising research field in the remote sensing community, which paves the way for scene-level landscape pattern analysis and decision making. In this paper, we propose a Bilateral Awareness Network which contains a dependency path and a texture path to fully capture the long-range relationships and fine-grained details in VFR images. Specifically, the dependency path is conducted based on the ResT, a novel Transformer backbone with memory-efficient multi-head self-attention, while the texture path is built on the stacked convolution operation. Besides, using the linear attention mechanism, a feature aggregation module is designed to effectively fuse the dependency features and texture features. Extensive experiments conducted on the three large-scale urban scene image segmentation datasets, i.e., ISPRS Vaihingen dataset, ISPRS Potsdam dataset, and UAVid dataset, demonstrate the effectiveness of our BANet. Specifically, a 64.6% mIoU is achieved on the UAVid dataset. Code is available at https://github.com/WangLibo1995/GeoSeg.

Motivation & Objective

• To address the challenge of semantic segmentation in very fine resolution (VFR) urban scene images, where extreme scale and appearance variations hinder existing deep learning models.
• To overcome the limitations of current models in capturing both long-range contextual dependencies and fine-grained local details in high-resolution remote sensing imagery.
• To design a unified network architecture that effectively integrates the strengths of Transformers and convolutional networks for improved scene-level understanding in urban environments.
• To validate the proposed method on large-scale, real-world urban segmentation datasets, including ISPRS Vaihingen, Potsdam, and UAVid.

Proposed method

• The network employs a dual-path design: a dependency path based on ResT, a memory-efficient Transformer backbone with multi-head self-attention, to model long-range spatial dependencies.
• A separate texture path is constructed using stacked convolutional layers to preserve and extract fine-grained local features from VFR images.
• A linear attention-based feature aggregation module is introduced to effectively fuse features from the dependency and texture paths, reducing computational cost while maintaining performance.
• The fusion mechanism dynamically attends to complementary representations from both paths, enhancing feature representation for downstream segmentation.
• The model is trained end-to-end on three large-scale urban segmentation datasets using standard cross-entropy and Dice loss objectives.
• The architecture is optimized for high-resolution inputs, with spatial reduction and feature refinement modules to maintain resolution and detail.

Experimental results

Research questions

• RQ1 Can a dual-path network combining Transformers and convolutions improve semantic segmentation accuracy on very fine resolution urban scene images?
• RQ2 How effectively can a memory-efficient Transformer backbone like ResT capture long-range dependencies in high-resolution remote sensing data?
• RQ3 To what extent does fusing long-range dependency features with local texture features enhance segmentation performance on complex urban scenes?
• RQ4 Does the proposed linear attention-based fusion mechanism outperform standard concatenation or element-wise operations in feature integration?

Key findings

• BANet achieves a mean Intersection over Union (mIoU) of 64.6% on the UAVid dataset, setting a new state-of-the-art performance for semantic segmentation of very fine resolution urban scenes.
• The model demonstrates superior performance on the ISPRS Vaihingen and Potsdam datasets, confirming its generalization across diverse urban environments and image resolutions.
• Ablation studies show that both the dependency path (ResT) and texture path (convolutions) contribute significantly to the final performance, validating the dual-path design.
• The linear attention fusion module reduces computational complexity while maintaining high feature representation quality, making the model efficient for high-resolution inputs.
• The integration of Transformers and convolutions leads to better handling of scale variations and object appearance diversity in urban scenes.
• The model generalizes well across different urban datasets, indicating robustness to domain shifts in remote sensing imagery.