[Paper Review] Domain Adaptation with Joint Learning for Generic, Optical Car Part Recognition and Detection Systems (Go-CaRD)

Lukas Stappen, Xinchen Du+3 more|arXiv (Cornell University)|Jun 15, 2020

Advanced Neural Network Applications43 references4 citations

TL;DR

This paper proposes Go-CaRD, a generic optical car part recognition and detection system using joint learning and domain adaptation to improve performance across diverse real-world automotive environments. By training on a novel, publicly available MuSe-CAR-Part dataset with human-vehicle interactions, the method achieves an F1 score of 93.67% for recognition and mAP of 63.01% for detection, outperforming standard fine-tuning and unsupervised domain adaptation by over 5% absolute gain in mAP on test sets.

ABSTRACT

Systems for the automatic recognition and detection of automotive parts are crucial in several emerging research areas in the development of intelligent vehicles. They enable, for example, the detection and modelling of interactions between human and the vehicle. In this paper, we quantitatively and qualitatively explore the efficacy of deep learning architectures for the classification and localisation of 29 interior and exterior vehicle regions on three novel datasets. Furthermore, we experiment with joint and transfer learning approaches across datasets and point out potential applications of our systems. Our best network architecture achieves an F1 score of 93.67 % for recognition, while our best localisation approach utilising state-of-the-art backbone networks achieve a mAP of 63.01 % for detection. The MuSe-CAR-Part dataset, which is based on a large variety of human-car interactions in videos, the weights of the best models, and the code is publicly available to academic parties for benchmarking and future research.

Motivation & Objective

• To develop a generic, robust system for optical recognition and detection of 29 interior and exterior car parts under real-world, variable conditions.
• To address the lack of datasets and models for fine-grained car part detection in dynamic, human-interactive environments.
• To improve model generalization across domains by introducing a joint learning and domain adaptation framework that combines source and target data during training.
• To release a new, large-scale, publicly available dataset (MuSe-CAR-Part) and trained models to support future research and benchmarking in automotive computer vision.

Proposed method

• The authors created three new datasets: Close-CAR (close-up, suboptimal conditions), Mix-CAR (multi-model, multi-configuration), and MuSe-CAR-Part (human-vehicle interaction videos), with the latter made publicly available.
• They employed state-of-the-art CNN backbones (e.g., Darknet) and adapted them for joint learning, where features from a source domain (no humans) and target domain (with human occlusion) are trained simultaneously.
• The domain adaptation strategy involves joint training with data from both domains, enabling the model to learn shared representations while adapting to distribution shifts caused by human interactions and varying lighting.
• The method uses transfer learning by initializing networks with ImageNet weights and fine-tuning on automotive-specific data, while incorporating domain-specific data via joint optimization.
• For detection, they applied two-stage object detectors (e.g., Faster R-CNN) with feature fusion mechanisms to improve localization accuracy under occlusion and viewpoint variation.
• The system was evaluated using standard metrics: F1 score for recognition and mean average precision (mAP) for detection, with ablation studies comparing joint learning, fine-tuning, and unsupervised domain adaptation.

Experimental results

Research questions

• RQ1 Can joint learning across source and target domains improve generalization in car part recognition and detection under real-world, variable conditions?
• RQ2 How does joint domain adaptation compare to standard fine-tuning and unsupervised domain adaptation in terms of mAP and F1 score on unseen automotive datasets?
• RQ3 To what extent can a generic model trained on diverse car models and configurations generalize to human-vehicle interaction scenarios with occlusion and dynamic lighting?
• RQ4 What performance gains can be achieved by incorporating human-occluded data during training, especially in real-world video sequences?

Key findings

• The best recognition model achieved an F1 score of 93.67% on the test set of the classification dataset, demonstrating high accuracy in identifying 29 car parts.
• The detection system using joint learning achieved a mAP of 63.01% on the test set of the Mix-CAR dataset, significantly outperforming standard fine-tuning and unsupervised domain adaptation.
• On the human-vehicle interaction dataset (MuSe-CAR-Part), joint learning improved mAP from 29.62% (fine-tuning baseline) to 41.07%, a 10-percentage-point absolute gain.
• The joint learning approach improved mAP by over 5% absolute on both the Mix-CAR development and test sets compared to standard transfer learning and unsupervised domain adaptation.
• The MuSe-CAR-Part dataset, containing real-world videos of human-car interactions, was successfully used to train and evaluate models, demonstrating robustness to occlusion and variable viewpoints.
• The release of the MuSe-CAR-Part dataset, trained models, and code enables future benchmarking and accelerates research in intelligent vehicle perception and human-vehicle interaction.