[Paper Review] Unsupervised Domain Adaptation for Multispectral Pedestrian Detection

Dayan Guan, Xing Luo+5 more|arXiv (Cornell University)|Apr 7, 2019

Video Surveillance and Tracking Methods36 references4 citations

TL;DR

This paper proposes an unsupervised domain adaptation framework for multispectral pedestrian detection that iteratively generates pseudo annotations and updates a two-stream detector on the target domain without manual annotations. By leveraging similarity and complementarity between aligned visible and thermal images, the method achieves detection performance competitive with supervised models, improving AP by 31.37% over non-adapted baselines and 6.67% over state-of-the-art unsupervised methods.

ABSTRACT

Multimodal information (e.g., visible and thermal) can generate robust pedestrian detections to facilitate around-the-clock computer vision applications, such as autonomous driving and video surveillance. However, it still remains a crucial challenge to train a reliable detector working well in different multispectral pedestrian datasets without manual annotations. In this paper, we propose a novel unsupervised domain adaptation framework for multispectral pedestrian detection, by iteratively generating pseudo annotations and updating the parameters of our designed multispectral pedestrian detector on target domain. Pseudo annotations are generated using the detector trained on source domain, and then updated by fixing the parameters of detector and minimizing the cross entropy loss without back-propagation. Training labels are generated using the pseudo annotations by considering the characteristics of similarity and complementarity between well-aligned visible and infrared image pairs. The parameters of detector are updated using the generated labels by minimizing our defined multi-detection loss function with back-propagation. The optimal parameters of detector can be obtained after iteratively updating the pseudo annotations and parameters. Experimental results show that our proposed unsupervised multimodal domain adaptation method achieves significantly higher detection performance than the approach without domain adaptation, and is competitive with the supervised multispectral pedestrian detectors.

Motivation & Objective

• To address the challenge of domain shift between multispectral pedestrian detection datasets, such as KAIST and CVC-14, where detectors trained on one domain perform poorly on another.
• To eliminate the need for costly manual annotation on target domains by enabling unsupervised adaptation using pseudo labels.
• To improve detection robustness across varying conditions (viewpoint, weather, camera) by exploiting complementary features in visible and thermal modalities.
• To develop a training framework that maintains minimal computational overhead compared to supervised training, despite iterative refinement.

Proposed method

• Pseudo annotations are generated using a detector pre-trained on the source domain (e.g., KAIST) and then refined by fixing the detector and minimizing cross-entropy loss without back-propagation.
• Training labels are generated by fusing pseudo annotations from both visible and thermal streams, exploiting their spatial alignment, similarity, and complementary cues.
• A multi-detection loss function is defined to jointly optimize the two-stream detector on the refined pseudo labels, enabling feature learning via back-propagation.
• The framework iteratively alternates between refining pseudo annotations and updating the detector parameters, improving generalization on the target domain.
• The method leverages well-aligned visible-infrared image pairs to ensure spatial consistency in pseudo label generation, enhancing detection reliability.
• The approach is designed to be scalable and efficient, with minimal additional training time since pseudo-label optimization does not require gradient computation.

Experimental results

Research questions

• RQ1 Can unsupervised domain adaptation significantly improve multispectral pedestrian detection performance on a target domain without manual annotations?
• RQ2 How does the proposed method compare to supervised baselines in terms of detection accuracy across different lighting and environmental conditions?
• RQ3 To what extent can pseudo annotations generated from a source-domain detector generalize to a target domain with domain shift?
• RQ4 Does the integration of complementary visible and thermal features through iterative refinement improve detection robustness?
• RQ5 Can the proposed method maintain training efficiency while achieving performance close to fully supervised approaches?

Key findings

• The proposed unsupervised multimodal domain adaptation (UMDA) method achieves a pixel-level average precision (AP) of 0.8023 on the CVC-14 dataset, which is 8.84% higher than the ACF+T+THOG baseline and 6.67% lower than the supervised SMDA model.
• UMDA improves detection performance by 31.37% in AP compared to a non-adapted baseline (WMDA), demonstrating the effectiveness of domain adaptation.
• The method achieves competitive results on both daytime and nighttime conditions, with 0.7688 and 0.8503 AP, respectively, showing robustness across lighting variations.
• Qualitative results show that UMDA performs comparably to SMDA when human features are distinct in either modality, but struggles in cluttered backgrounds or low-contrast scenarios.
• The training time of the proposed method is nearly identical to supervised training, as the additional cost of pseudo-label optimization is minimal due to no back-propagation.
• The two-stream detection supervision module in SMDA contributes to improved performance, confirming the value of modality-specific feature learning in multispectral detection.