[Paper Review] Efficient Hybrid Transformer: Learning Global-local Context for Urban Sence Segmentation.

Libo Wang, Shenghui Fang+3 more|arXiv (Cornell University)|Sep 18, 2021

Advanced Neural Network Applications29 references4 citations

TL;DR

This paper proposes an efficient hybrid Transformer (EHT) for real-time urban scene segmentation, combining a CNN-based encoder for local feature extraction with a transformer-based decoder for global context modeling. The EHT achieves 66.9% mIoU on the UAVid test set with faster inference than state-of-the-art lightweight models, demonstrating superior efficiency-accuracy trade-offs.

ABSTRACT

Semantic segmentation of fine-resolution urban scene images plays a vital role in extensive practical applications, such as land cover mapping, urban change detection, environmental protection and economic assessment. Driven by rapid developments in deep learning technologies, the convolutional neural network (CNN) has dominated the semantic segmentation task for many years. Convolutional neural networks adopt hierarchical feature representation, demonstrating strong local information extraction. However, the local property of the convolution layer limits the network from capturing global context that is crucial for precise segmentation. Recently, Transformer comprise a hot topic in the computer vision domain. Transformer demonstrates the great capability of global information modelling, boosting many vision tasks, such as image classification, object detection and especially semantic segmentation. In this paper, we propose an efficient hybrid Transformer (EHT) for real-time urban scene segmentation. The EHT adopts a hybrid structure with and CNN-based encoder and a transformer-based decoder, learning global-local context with lower computation. Extensive experiments demonstrate that our EHT has faster inference speed with competitive accuracy compared with state-of-the-art lightweight models. Specifically, the proposed EHT achieves a 66.9% mIoU on the UAVid test set and outperforms other benchmark networks significantly. The code will be available soon.

Motivation & Objective

• To address the limitation of CNNs in capturing global context for fine-resolution urban scene segmentation.
• To overcome the high computational cost of pure Transformers in real-time segmentation tasks.
• To design a lightweight yet accurate model that effectively fuses local and global context for urban scene understanding.
• To achieve competitive segmentation accuracy with significantly faster inference speed compared to existing lightweight models.

Proposed method

• The EHT employs a hybrid architecture with a CNN-based encoder to extract hierarchical local features.
• A transformer-based decoder is used to model long-range dependencies and global context across the feature map.
• Global-local feature fusion is achieved by integrating features from the CNN encoder and transformer decoder.
• The model is trained end-to-end using standard segmentation loss functions on urban scene datasets.
• The design prioritizes inference efficiency by minimizing redundant computation in the transformer decoder.
• The architecture is optimized for real-time deployment on resource-constrained platforms.

Experimental results

Research questions

• RQ1 Can a hybrid CNN-Transformer architecture achieve better global context modeling than pure CNNs in urban scene segmentation?
• RQ2 How does the proposed EHT balance computational efficiency and segmentation accuracy compared to state-of-the-art lightweight models?
• RQ3 To what extent can a transformer-based decoder improve mIoU while maintaining fast inference speed in real-time urban segmentation?
• RQ4 Does the fusion of local features from a CNN encoder with global context from a transformer decoder lead to more robust segmentation in complex urban scenes?

Key findings

• The EHT achieves a 66.9% mean Intersection over Union (mIoU) on the UAVid test set, outperforming other benchmark networks.
• The model demonstrates significantly faster inference speed than state-of-the-art lightweight segmentation models.
• The hybrid architecture effectively captures both local details and long-range dependencies, improving segmentation accuracy.
• The CNN-Transformer combination enables competitive performance with reduced computational overhead.
• The model shows strong generalization capability on fine-resolution urban scene images.
• The proposed method establishes a new SOTA in inference speed while maintaining high accuracy on the UAVid benchmark.