[Paper Review] Radar Camera Fusion via Representation Learning in Autonomous Driving

Dong Xu, Binnan Zhuang+2 more|arXiv (Cornell University)|Mar 14, 2021

Advanced Neural Network Applications30 references7 citations

TL;DR

This paper proposes AssociationNet, a deep representation learning framework for radar-camera fusion in autonomous driving that outperforms rule-based methods by 11.6% in F1 score. It uses pseudo-image construction from radar pins and 2D bounding boxes, combined with a loss sampling mechanism and ordinal loss, to enable robust, end-to-end data association without LiDAR-based ground truth.

ABSTRACT

Radars and cameras are mature, cost-effective, and robust sensors and have been widely used in the perception stack of mass-produced autonomous driving systems. Due to their complementary properties, outputs from radar detection (radar pins) and camera perception (2D bounding boxes) are usually fused to generate the best perception results. The key to successful radar-camera fusion is the accurate data association. The challenges in the radar-camera association can be attributed to the complexity of driving scenes, the noisy and sparse nature of radar measurements, and the depth ambiguity from 2D bounding boxes. Traditional rule-based association methods are susceptible to performance degradation in challenging scenarios and failure in corner cases. In this study, we propose to address radar-camera association via deep representation learning, to explore feature-level interaction and global reasoning. Additionally, we design a loss sampling mechanism and an innovative ordinal loss to overcome the difficulty of imperfect labeling and to enforce critical human-like reasoning. Despite being trained with noisy labels generated by a rule-based algorithm, our proposed method achieves a performance of 92.2% F1 score, which is 11.6% higher than the rule-based teacher. Moreover, this data-driven method also lends itself to continuous improvement via corner case mining.

Motivation & Objective

• Address the challenge of radar-camera data association in autonomous driving, where traditional rule-based methods fail in complex or rare scenarios.
• Develop a scalable, low-cost perception solution that does not rely on expensive LiDAR-based ground-truth annotations.
• Overcome the limitations of noisy, imperfect labels from rule-based association algorithms through robust deep learning training strategies.
• Enable global reasoning and feature-level interaction between radar and camera modalities via a learned representation framework.
• Design a training pipeline that supports continuous improvement through corner case mining, even with imperfect supervision.

Proposed method

• Construct a pseudo-image by projecting radar pins and 2D bounding boxes onto the camera image plane and concatenating their features with the RGB image.
• Feed the resulting pseudo-image into a convolutional neural network (CNN) to learn high-level semantic representations for object-level association.
• Use a loss sampling mechanism that balances positive and negative pairs during training to mitigate the impact of noisy labels.
• Introduce a novel ordinal loss to enforce human-like reasoning by modeling relative confidence between matched and unmatched pairs.
• Train the model end-to-end using labels generated by a rule-based algorithm, enabling deployment in real-world fleets without LiDAR.
• Leverage global context through feature maps from the final CNN layer to enable scene-dependent reasoning beyond local distance metrics.

Figure 1 : An illustration of the associations between radar detections (radar pins) and camera detections (2D bounding boxes). The context of the scene is illustrated in the top picture, with the image captured by the camera along with the detected bounding boxes and the projected radar pins (shown

Experimental results

Research questions

• RQ1 Can a deep representation learning approach outperform rule-based radar-camera association in complex driving scenarios with noisy and sparse radar data?
• RQ2 How effective is a loss sampling mechanism in reducing the negative impact of imperfect labels from rule-based association on model training?
• RQ3 To what extent can an ordinal loss improve model generalization by encoding critical reasoning logic into the learning process?
• RQ4 Can a learning-based fusion framework achieve high performance without relying on LiDAR-based ground-truth, enabling scalability for mass-produced autonomous systems?
• RQ5 How does the proposed method handle corner cases and rare scenarios compared to traditional heuristic methods?

Key findings

• AssociationNet achieves a 92.2% F1 score on the test set, representing an 11.6% improvement over the rule-based teacher method (80.6% F1).
• The loss sampling mechanism with a 1:1 positive-to-negative pair ratio boosts performance by 1.1% in F1 score compared to no sampling.
• The ordinal loss with an optimal weight of 2.0 increases the F1 score by 1.8% compared to the baseline without ordinal supervision.
• The model demonstrates robustness in challenging scenarios, correctly associating objects even in dense traffic with multiple large vehicles.
• Despite being trained on noisy labels from a rule-based system, AssociationNet generalizes better than the teacher model, especially in recall, improving from 73.6% to 93.9%.
• Visualization shows that the model correctly associates distant or occluded vehicles in complex scenes, though small object size and occlusion still lead to some errors.

Figure 2 : An overview of AssociationNet. Process a illustrates how the radar pins and 2D bounding boxes are first projected into the camera image plane and then produce a pseudo-image. Process b illustrates how the final pseudo-image is composed by concatenating all the features of radar pins, boun