[Paper Review] simCrossTrans: A Simple Cross-Modality Transfer Learning for Object Detection with ConvNets or Vision Transformers

Xiaoke Shen, Ioannis Stamos|arXiv (Cornell University)|Mar 20, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes simCrossTrans, a simple cross-modality transfer learning method that enables 3D depth-only object detection by fine-tuning pre-trained 2D models (ConvNets or Vision Transformers) on pseudo-images derived from point clouds. It achieves a 16.1% mAP50 gain over baseline methods using ViTs, surpassing previous SOTA by 15.4% and closing to RGB-only SOTA with only a 1% gap.

ABSTRACT

Transfer learning is widely used in computer vision (CV), natural language processing (NLP) and achieves great success. Most transfer learning systems are based on the same modality (e.g. RGB image in CV and text in NLP). However, the cross-modality transfer learning (CMTL) systems are scarce. In this work, we study CMTL from 2D to 3D sensor to explore the upper bound performance of 3D sensor only systems, which play critical roles in robotic navigation and perform well in low light scenarios. While most CMTL pipelines from 2D to 3D vision are complicated and based on Convolutional Neural Networks (ConvNets), ours is easy to implement, expand and based on both ConvNets and Vision transformers(ViTs): 1) By converting point clouds to pseudo-images, we can use an almost identical network from pre-trained models based on 2D images. This makes our system easy to implement and expand. 2) Recently ViTs have been showing good performance and robustness to occlusions, one of the key reasons for poor performance of 3D vision systems. We explored both ViT and ConvNet with similar model sizes to investigate the performance difference. We name our approach simCrossTrans: simple cross-modality transfer learning with ConvNets or ViTs. Experiments on SUN RGB-D dataset show: with simCrossTrans we achieve $13.2\%$ and $16.1\%$ absolute performance gain based on ConvNets and ViTs separately. We also observed the ViTs based performs $9.7\%$ better than the ConvNets one, showing the power of simCrossTrans with ViT. simCrossTrans with ViTs surpasses the previous state-of-the-art (SOTA) by a large margin of $+15.4\%$ mAP50. Compared with the previous 2D detection SOTA based RGB images, our depth image only system only has a $1\%$ gap. The code, training/inference logs and models are publicly available at https://github.com/liketheflower/simCrossTrans

Motivation & Objective

• To explore the upper bound performance of 3D sensor-only object detection systems in low-light or privacy-constrained environments.
• To bridge the modality gap between 2D RGB images and 3D point clouds using transfer learning for improved detection accuracy.
• To evaluate whether Vision Transformers (ViTs) outperform ConvNets in cross-modality transfer learning for depth-based object detection.
• To develop a simple, extensible, and modular framework that reuses existing 2D pre-trained models without architectural changes.

Proposed method

• Convert 3D point clouds into pseudo-2D images using a projection-based transformation to enable compatibility with 2D pre-trained models.
• Fine-tune pre-trained 2D models (ResNet-50 and Swin-T) on the pseudo-2D depth images using the same training pipeline as in 2D vision.
• Use identical model architectures and hyperparameters for both ConvNet and ViT backbones to ensure fair comparison.
• Apply standard object detection head on top of the fine-tuned backbone for inference on depth-only data.
• Leverage the contextual and spatial priors learned from 2D ImageNet-style pre-training to improve feature representation in 3D-only detection.
• Utilize feature visualization and histogram analysis to compare the internal representations of ConvNets and ViTs after cross-modality fine-tuning.

Experimental results

Research questions

• RQ1 Can cross-modality transfer learning from 2D RGB images to 3D depth images significantly improve object detection performance in depth-only systems?
• RQ2 How does the performance of Vision Transformers compare to ConvNets when applied to cross-modality transfer learning in 3D object detection?
• RQ3 To what extent can pre-trained 2D models generalize to depth-only data through pseudo-image conversion and fine-tuning?
• RQ4 Does the use of self-supervised pre-training in the source modality further improve transfer performance in the target modality?

Key findings

• simCrossTrans achieves a 13.2% absolute mAP50 improvement over baseline when using ResNet-50 as the backbone.
• With Swin-T as the backbone, simCrossTrans achieves a 16.1% absolute mAP50 gain, significantly outperforming the previous SOTA by +15.4%.
• Vision Transformers outperform ConvNets by 9.7% in mAP50 under the same training setup, demonstrating their superiority in cross-modality transfer.
• The method reduces the performance gap between depth-only and RGB-only detection systems to just 1% mAP50, indicating strong generalization.
• Hard categories like laptop, keyboard, and stool benefit most from simCrossTrans, showing that contextual priors from 2D pre-training help in low-signal scenarios.
• Feature visualization reveals denser and more structured feature maps in ViTs compared to the sparser outputs in ConvNets, suggesting better feature learning with attention mechanisms.