[Paper Review] CAT-Det: Contrastively Augmented Transformer for Multi-modal 3D Object Detection

Yanan Zhang, Jiaxin Chen+1 more|arXiv (Cornell University)|Apr 1, 2022

Advanced Neural Network Applications4 citations

TL;DR

CAT-Det proposes a contrastively augmented Transformer framework for multi-modal 3D object detection, leveraging a two-stream architecture with Pointformer and Imageformer branches, enhanced by a Cross-Modal Transformer (CMT) for long-range context encoding. It introduces One-way Multi-modal Data Augmentation (OMDA) via hierarchical contrastive learning on point-clouds only, achieving state-of-the-art performance on KITTI with a 75.42% mAP, significantly improving detection accuracy for hard cases like pedestrians and cyclists.

ABSTRACT

In autonomous driving, LiDAR point-clouds and RGB images are two major data modalities with complementary cues for 3D object detection. However, it is quite difficult to sufficiently use them, due to large inter-modal discrepancies. To address this issue, we propose a novel framework, namely Contrastively Augmented Transformer for multi-modal 3D object Detection (CAT-Det). Specifically, CAT-Det adopts a two-stream structure consisting of a Pointformer (PT) branch, an Imageformer (IT) branch along with a Cross-Modal Transformer (CMT) module. PT, IT and CMT jointly encode intra-modal and inter-modal long-range contexts for representing an object, thus fully exploring multi-modal information for detection. Furthermore, we propose an effective One-way Multi-modal Data Augmentation (OMDA) approach via hierarchical contrastive learning at both the point and object levels, significantly improving the accuracy only by augmenting point-clouds, which is free from complex generation of paired samples of the two modalities. Extensive experiments on the KITTI benchmark show that CAT-Det achieves a new state-of-the-art, highlighting its effectiveness.

Motivation & Objective

• To address the challenge of inter-modal discrepancy between LiDAR point-clouds and RGB images in 3D object detection.
• To improve feature representation by capturing long-range intra-modal and inter-modal dependencies in both modalities.
• To develop an effective data augmentation strategy that avoids complex paired data generation and enhances model generalization.
• To achieve state-of-the-art performance on the KITTI benchmark for multi-modal 3D object detection.

Proposed method

• CAT-Det employs a two-stream architecture with a Pointformer (PT) branch for point-cloud processing and an Imageformer (IT) branch for image feature extraction.
• A Cross-Modal Transformer (CMT) module is introduced to perform cross-attention and feature fusion, enabling holistic modeling of inter-modal relationships with fine-grained attention weights.
• The method uses hierarchical contrastive learning via One-way Multi-modal Data Augmentation (OMDA), applying augmentations only to point-clouds at both point-level and object-level to improve robustness.
• Negative samples for contrastive learning are selected using a memory bank at the object-level, enhancing feature discrimination without requiring paired image augmentations.
• The model replaces traditional PointNet++ and 2D CNNs with Transformers to achieve larger receptive fields and better context modeling.
• Feature fusion in CMT preserves both single-modal and cross-modal features, outperforming simple concatenation or attention-based fusion.

Experimental results

Research questions

• RQ1 How can long-range intra-modal and inter-modal dependencies be effectively captured in multi-modal 3D object detection?
• RQ2 Can data augmentation be effectively applied to only one modality (LiDAR) to improve multi-modal detection without complex paired data generation?
• RQ3 What is the impact of hierarchical contrastive learning at both point and object levels on model generalization and accuracy?
• RQ4 How does the proposed Cross-Modal Transformer (CMT) compare to conventional fusion methods like concatenation or attention-based fusion?
• RQ5 To what extent does the proposed framework outperform existing state-of-the-art methods on the KITTI benchmark?

Key findings

• CAT-Det achieves a new state-of-the-art mAP of 75.42% on the KITTI 3D object detection benchmark, outperforming the previous best method by 3.53% in mAP.
• The ablation study confirms that each component—TPI, CMT, and OMDA—contributes significantly, with OMDA alone improving mAP by 3.53% over the baseline.
• The CMT module outperforms both concatenation and attention-based fusion, achieving 71.89% mAP when combined with TPI, due to its ability to preserve and refine both single-modal and cross-modal features.
• Object-level contrastive learning with a memory bank (CA-O-MB) yields the highest performance gain, increasing mAP by 2.65% compared to point-level only augmentation.
• The model shows superior performance on hard categories such as pedestrians and cyclists, with mAP improvements of 7.18% and 5.59% respectively, indicating strong generalization on small or partial instances.
• Visualizations confirm that CAT-Det effectively enhances features of sparse or distant objects by leveraging complementary cues from both modalities through the CMT module.