[Paper Review] Spatial Attention Pyramid Network for Unsupervised Domain Adaptation

Congcong Li, Dawei Du+5 more|arXiv (Cornell University)|Mar 29, 2020

Domain Adaptation and Few-Shot Learning44 references7 citations

TL;DR

This paper proposes a Spatial Attention Pyramid Network (SAPNet) for unsupervised domain adaptation in object detection, instance segmentation, and semantic segmentation. By integrating multi-scale feature maps through a task-guided spatial attention mechanism, SAPNet enhances discriminative feature learning across domains, achieving state-of-the-art performance with a 3% mAP gain on Cityscapes→FoggyCityscapes and strong results on GTA5→Cityscapes.

ABSTRACT

Unsupervised domain adaptation is critical in various computer vision tasks, such as object detection, instance segmentation, and semantic segmentation, which aims to alleviate performance degradation caused by domain-shift. Most of previous methods rely on a single-mode distribution of source and target domains to align them with adversarial learning, leading to inferior results in various scenarios. To that end, in this paper, we design a new spatial attention pyramid network for unsupervised domain adaptation. Specifically, we first build the spatial pyramid representation to capture context information of objects at different scales. Guided by the task-specific information, we combine the dense global structure representation and local texture patterns at each spatial location effectively using the spatial attention mechanism. In this way, the network is enforced to focus on the discriminative regions with context information for domain adaption. We conduct extensive experiments on various challenging datasets for unsupervised domain adaptation on object detection, instance segmentation, and semantic segmentation, which demonstrates that our method performs favorably against the state-of-the-art methods by a large margin. Our source code is available at https://isrc.iscas.ac.cn/gitlab/research/domain-adaption.

Motivation & Objective

• To address domain shift in computer vision tasks like object detection and segmentation, where source and target domains exhibit significant distribution differences.
• To overcome limitations of prior methods that rely on global distribution alignment or fail to exploit multi-scale context and discriminative regions.
• To design a unified framework that effectively fuses global context and local texture patterns using attention mechanisms for better domain generalization.
• To improve performance on challenging domain adaptation benchmarks such as Cityscapes→FoggyCityscapes, PASCAL VOC→Clipart, and GTA5→Cityscapes.

Proposed method

• Construct a spatial pyramid representation using multi-scale feature maps to capture both global image-level and local region-level semantic information.
• Introduce a task-specific guided spatial attention mechanism that adaptively fuses features across scales based on discriminative region relevance.
• Use average pooling in the spatial pyramid to preserve local texture patterns and enhance feature discrimination for adversarial domain alignment.
• Apply channel-wise attention to dynamically weight multi-level features, improving representation learning in dissimilar domains.
• Integrate the spatial attention pyramid into a deep network with adversarial training to align source and target domain features.
• Leverage gradient reversal and multi-loss optimization to refine domain-invariant representations while preserving task-specific semantics.

Experimental results

Research questions

• RQ1 Can a spatial pyramid with multi-scale features improve domain adaptation performance in object detection and segmentation?
• RQ2 How does task-specific guidance enhance attention focusing on discriminative regions during unsupervised domain adaptation?
• RQ3 What is the impact of spatial pyramid depth and pooling strategy (average vs. max) on feature representation quality?
• RQ4 Does channel-wise attention improve performance when adapting between highly dissimilar domains?
• RQ5 Can the proposed method generalize across diverse domain shift scenarios, including style transfer and weather variation?

Key findings

• SAPNet achieves a 3% mAP improvement on the Cityscapes→FoggyCityscapes object detection benchmark, significantly outperforming prior state-of-the-art methods.
• On the GTA5→Cityscapes semantic segmentation task, SAPNet achieves competitive accuracy, demonstrating strong generalization across domain shifts.
• Ablation studies confirm that removing the spatial attention pyramid reduces mAP by over 6% on the most challenging PASCAL VOC→Clipart benchmark.
• Task-specific guidance improves mAP by 5.1% on PASCAL VOC→Clipart, highlighting its role in focusing on discriminative regions.
• Average pooling in the spatial pyramid yields better performance than max pooling, with a 4.7% mAP gain on PASCAL VOC→Clipart.
• Channel-wise attention provides a 1.6% mAP gain on dissimilar domains (e.g., PASCAL VOC→Clipart), but minimal improvement on similar domains, indicating its value in complex adaptation scenarios.