[Paper Review] Channel-wise Alignment for Adaptive Object Detection

Hang Yang, Shan Jiang+5 more|arXiv (Cornell University)|Sep 7, 2020

Advanced Neural Network Applications39 references4 citations

TL;DR

This paper proposes a novel channel-wise domain adaptation framework for object detection that improves generalization across domain shifts by aligning feature distributions at the channel level. By introducing self and cross-channel-wise alignment modules and an RPN domain classifier, the method achieves a 5% mAP gain over prior methods on multiple domain-shift benchmarks, with strong transferability to instance segmentation.

ABSTRACT

Generic object detection has been immensely promoted by the development of deep convolutional neural networks in the past decade. However, in the domain shift circumstance, the changes in weather, illumination, etc., often cause domain gap, and thus performance drops substantially when detecting objects from one domain to another. Existing methods on this task usually draw attention on the high-level alignment based on the whole image or object of interest, which naturally, cannot fully utilize the fine-grained channel information. In this paper, we realize adaptation from a thoroughly different perspective, i.e., channel-wise alignment. Motivated by the finding that each channel focuses on a specific pattern (e.g., on special semantic regions, such as car), we aim to align the distribution of source and target domain on the channel level, which is finer for integration between discrepant domains. Our method mainly consists of self channel-wise and cross channel-wise alignment. These two parts explore the inner-relation and cross-relation of attention regions implicitly from the view of channels. Further more, we also propose a RPN domain classifier module to obtain a domain-invariant RPN network. Extensive experiments show that the proposed method performs notably better than existing methods with about 5% improvement under various domain-shift settings. Experiments on different task (e.g. instance segmentation) also demonstrate its good scalability.

Motivation & Objective

• Address domain shift in object detection caused by variations in weather, illumination, and image quality.
• Overcome limitations of existing high-level feature alignment methods that fail to exploit fine-grained channel-level patterns.
• Explore the potential of channel-wise feature representations for domain adaptation in object detection.
• Develop a scalable framework that generalizes beyond detection to instance segmentation tasks.
• Demonstrate that channel-level distribution alignment leads to significant performance gains in cross-domain detection.

Proposed method

• Propose a self channel-wise alignment (SCA) module that treats each channel as a distinct distribution and applies adversarial training to learn domain-invariant features.
• Introduce a cross channel-wise alignment (CCA) module using a decayed Gram-matrix loss to model mutual information between channels and enhance feature diversity.
• Integrate an RPN domain classifier (RDC) to align region proposals and improve robustness in region-of-interest features.
• Combine SCA, CCA, and RDC in a unified framework to enable joint optimization across channel-level and proposal-level domain adaptation.
• Use VGG-16 as the backbone and apply SCA at multiple stages (conv1-5) to assess the impact of feature depth on adaptation performance.
• Leverage a decayed Gram-matrix loss that down-weights high-activation channel pairs to emphasize informative, low-overlap channel interactions.

Experimental results

Research questions

• RQ1 Can channel-wise feature alignment lead to better domain adaptation in object detection compared to image- or instance-level alignment?
• RQ2 How do self-channel-wise and cross-channel-wise alignment components contribute individually and jointly to performance gains?
• RQ3 Does the proposed decayed Gram-matrix loss effectively capture inter-channel dependencies for improved domain generalization?
• RQ4 Can the method be extended to instance segmentation with comparable performance gains?
• RQ5 How does the placement of SCA across different backbone stages affect the final detection accuracy?

Key findings

• The proposed method achieves a 5% mAP improvement over baseline methods across three domain-shift settings: Cityscapes to Foggy-Cityscapes, SIM-10k to Cityscapes, and KITTI to Cityscapes.
• The self channel-wise alignment (SCA) module alone improves mAP by 13.5% over the source-only baseline, indicating its dominant contribution.
• The cross channel-wise alignment (CCA) module contributes 0.6% improvement over source-only, and the RPN domain classifier (RDC) adds 2.7% improvement.
• Combining SCA and CCA yields 39.3% mAP, outperforming combinations with RDC, confirming their complementary nature.
• The full model with SCA, CCA, and RDC achieves the highest mAP, demonstrating synergistic gains from multi-level alignment.
• On instance segmentation, the method improves mAP by 3.8% on bounding boxes and 12.1% on masks compared to source-only, outperforming the SOTA method SCDA.