[Paper Review] MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformer

Jinze Yu, Jiaming Liu+8 more|arXiv (Cornell University)|May 3, 2022

Advanced Neural Network Applications4 citations

TL;DR

MTTrans proposes an end-to-end cross-domain object detection framework based on the mean teacher training paradigm and multi-level feature alignment in Deformable DETR. By generating high-quality pseudo labels from unlabeled target data and aligning features across domains at global, instance, and local levels, MTTrans achieves state-of-the-art performance, notably improving mAP from 52.6 to 57.9 on the Sim10k to Cityscapes benchmark.

ABSTRACT

Recently, DEtection TRansformer (DETR), an end-to-end object detection pipeline, has achieved promising performance. However, it requires large-scale labeled data and suffers from domain shift, especially when no labeled data is available in the target domain. To solve this problem, we propose an end-to-end cross-domain detection Transformer based on the mean teacher framework, MTTrans, which can fully exploit unlabeled target domain data in object detection training and transfer knowledge between domains via pseudo labels. We further propose the comprehensive multi-level feature alignment to improve the pseudo labels generated by the mean teacher framework taking advantage of the cross-scale self-attention mechanism in Deformable DETR. Image and object features are aligned at the local, global, and instance levels with domain query-based feature alignment (DQFA), bi-level graph-based prototype alignment (BGPA), and token-wise image feature alignment (TIFA). On the other hand, the unlabeled target domain data pseudo-labeled and available for the object detection training by the mean teacher framework can lead to better feature extraction and alignment. Thus, the mean teacher framework and the comprehensive multi-level feature alignment can be optimized iteratively and mutually based on the architecture of Transformers. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in three domain adaptation scenarios, especially the result of Sim10k to Cityscapes scenario is remarkably improved from 52.6 mAP to 57.9 mAP. Code will be released.

Motivation & Objective

• Address domain shift in object detection when transferring from a source domain with labels to a target domain without labels.
• Overcome the limitation of prior unsupervised domain adaptation methods that underutilize unlabeled target data due to lack of annotations.
• Improve the quality of pseudo labels generated by the mean teacher framework in the presence of domain shift.
• Develop a comprehensive multi-level feature alignment strategy to enhance domain-invariant representation learning in Transformer-based detectors.
• Enable iterative and mutual optimization between pseudo label generation and feature alignment in an end-to-end manner.

Proposed method

• Adapt the mean teacher framework to unsupervised domain adaptation (UDA) in object detection, using a student-teacher model setup with exponential moving average (EMA) for the teacher network.
• Introduce domain query-based feature alignment (DQFA) to align global image and object proposal features across domains using cross-scale self-attention.
• Propose bi-level graph-based prototype alignment (BGPA) to model and align instance-level features via prototypes built from object features.
• Implement token-wise image feature alignment (TIFA) to align local-level features at the token level in the image feature map.
• Integrate the mean teacher framework with multi-level feature alignment to jointly optimize pseudo label quality and feature alignment in an end-to-end manner.
• Use the pseudo-labeled target domain data during training to improve feature extraction and model generalization, enabling full utilization of unlabeled data.

Experimental results

Research questions

• RQ1 Can the mean teacher framework be effectively adapted to unsupervised domain adaptation in object detection, particularly when source and target domains exhibit significant distribution shifts?
• RQ2 How can the quality of pseudo labels generated by the mean teacher be improved under domain shift, especially when the teacher model is not domain-invariant?
• RQ3 What multi-level feature alignment strategies are most effective for enhancing domain-invariant representation learning in Transformer-based object detectors?
• RQ4 To what extent does joint optimization between pseudo label generation and feature alignment improve detection performance in cross-domain settings?
• RQ5 How do individual components of the proposed framework (e.g., DQFA, BGPA, TIFA) contribute to the overall performance gain in domain adaptation?

Key findings

• MTTrans achieves a new state-of-the-art mAP of 57.9 on the Sim10k to Cityscapes benchmark, representing a significant improvement from the previous SOTA of 52.6 mAP.
• The ablation study shows that removing the decoder-specific DQFA component leads to the largest performance drop (-0.451 mAP), highlighting its critical role in global feature alignment.
• Replacing BGPA with TIFA in the decoder or swapping TIFA with BGPA in the encoder results in performance drops of -0.460 mAP and -0.345 mAP, respectively, indicating the importance of component-specific design.
• Removing shared object queries between teacher and student models causes the largest performance drop (-0.616 mAP), confirming the necessity of consistent query learning.
• Pseudo label visualization confirms that MTTrans generates higher-quality pseudo labels than MT-DefDETR, and the teacher model’s predictions are more accurate than the student’s, enabling effective knowledge distillation.
• Qualitative results on Foggy Cityscapes show that MTTrans better detects distant and fog-obscured objects compared to Deformable DETR and SFA, with higher confidence scores on true positives.