[Paper Review] Deeply Aligned Adaptation for Cross-domain Object Detection

Minghao Fu, Zhenshan Xie+2 more|arXiv (Cornell University)|Apr 5, 2020

Advanced Neural Network Applications35 references4 citations

TL;DR

This paper proposes Deeply Aligned Adaptation (DAA), an end-to-end domain adaptation method for cross-domain object detection using Faster R-CNN, where domain alignment is applied at multiple levels: local patch features (low-level), global features (high-level), and instance-level foreground-background features. The method introduces a transition module to bridge local and global alignment, a domain mask integration (DMI) module to enhance adversarial training, and a foreground-background aware alignment module to prevent misalignment between object and background regions, achieving state-of-the-art mAP of 39.2% on the Cityscapes-to-Foggy Cityscapes benchmark.

ABSTRACT

Cross-domain object detection has recently attracted more and more attention for real-world applications, since it helps build robust detectors adapting well to new environments. In this work, we propose an end-to-end solution based on Faster R-CNN, where ground-truth annotations are available for source images (e.g., cartoon) but not for target ones (e.g., watercolor) during training. Motivated by the observation that the transferabilities of different neural network layers differ from each other, we propose to apply a number of domain alignment strategies to different layers of Faster R-CNN, where the alignment strength is gradually reduced from low to higher layers. Moreover, after obtaining region proposals in our network, we develop a foreground-background aware alignment module to further reduce the domain mismatch by separately aligning features of the foreground and background regions from the source and target domains. Extensive experiments on benchmark datasets demonstrate the effectiveness of our proposed approach.

Motivation & Objective

• To address the challenge of domain shift in cross-domain object detection where labeled data is only available in the source domain.
• To improve feature transferability across domains by accounting for the varying transferability of features across different network layers in Faster R-CNN.
• To reduce misalignment between foreground and background regions during domain adaptation by introducing instance-level, foreground-background aware alignment.
• To enhance adversarial training for better domain-invariant feature learning through a domain mask integration (DMI) module.
• To bridge the gap between local patch-level and global image-level domain alignment using a novel transition module.

Proposed method

• A multi-level domain alignment strategy is applied: local patch-level adversarial training in early layers, global image-level adversarial training in later layers, and a transition module to smoothly connect them.
• The domain mask integration (DMI) module dynamically assigns higher weights to more informative samples during adversarial training to improve discriminator and generator optimization.
• A foreground-background aware alignment module is introduced to separately align features from region proposals, preventing foreground objects in one domain from being misaligned with background regions in the other.
• The method is integrated into a Faster R-CNN framework, enabling end-to-end training with domain adaptation modules inserted at multiple stages.
• Adversarial training is applied at different levels: patch-level for low-level features and global for high-level features, with the transition module mediating the shift in alignment strategy.
• The model is trained in an end-to-end fashion with joint optimization of detection and domain adaptation objectives.

Experimental results

Research questions

• RQ1 How can domain alignment be effectively applied across different levels of a CNN to account for varying feature transferability?
• RQ2 What is the impact of separating foreground and background region alignment on reducing domain misalignment in object detection?
• RQ3 Can a domain mask integration module improve adversarial training by focusing on more informative samples?
• RQ4 How does a transition module between local and global alignment improve feature distribution alignment?
• RQ5 Does instance-level, foreground-background aware alignment lead to better mAP in cross-domain object detection?

Key findings

• The proposed DAA model achieves an mAP of 39.2% on the Cityscapes-to-Foggy Cityscapes benchmark, outperforming existing SOTA methods.
• The ablation study shows that adding the transition module improves mAP from 27.1% to 37.0%, confirming its importance in bridging local and global alignment.
• The domain mask integration (DMI) module improves mAP from 33.9% to 37.6% when added to the model, indicating better adversarial convergence and feature generalization.
• The foreground-background aware alignment module boosts mAP from 37.6% to 39.2%, proving its effectiveness in reducing misalignment between object and background regions.
• Without distinguishing foreground and background, mAP drops to 36.9%, validating the necessity of instance-level alignment.
• The frequency histogram of domain probabilities shows that DMI leads to more balanced scores around 0.5, indicating better domain confusion and invariant feature learning.