[Paper Review] CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection

Su Pang, Daniel Morris+1 more|arXiv (Cornell University)|Sep 2, 2020

Advanced Neural Network ApplicationsComputer Science30 references41 citations

TL;DR

CLOCs introduces a fast, late-fusion network that merges 2D image detections with 3D LiDAR detections at the candidate level before NMS, improving 3D/BEV detection on KITTI, especially at long range.

ABSTRACT

There have been significant advances in neural networks for both 3D object detection using LiDAR and 2D object detection using video. However, it has been surprisingly difficult to train networks to effectively use both modalities in a way that demonstrates gain over single-modality networks. In this paper, we propose a novel Camera-LiDAR Object Candidates (CLOCs) fusion network. CLOCs fusion provides a low-complexity multi-modal fusion framework that significantly improves the performance of single-modality detectors. CLOCs operates on the combined output candidates before Non-Maximum Suppression (NMS) of any 2D and any 3D detector, and is trained to leverage their geometric and semantic consistencies to produce more accurate final 3D and 2D detection results. Our experimental evaluation on the challenging KITTI object detection benchmark, including 3D and bird's eye view metrics, shows significant improvements, especially at long distance, over the state-of-the-art fusion based methods. At time of submission, CLOCs ranks the highest among all the fusion-based methods in the official KITTI leaderboard. We will release our code upon acceptance.

Motivation & Objective

• Motivate multi-modal fusion to improve 3D object detection beyond single-modality detectors.
• Propose a late-fusion architecture that leverages detection candidates rather than raw data.
• Create a probabilistic, data-driven fusion module that learns dependencies between 2D and 3D detections.
• Ensure the method is fast with low memory footprint and compatible with off-the-shelf detectors.
• Demonstrate improvements on KITTI across 3D and BEV metrics, especially at long distance.

Proposed method

• Fuse 2D and 3D detection candidates before NMS to avoid suppressing true positives from single modalities.
• Construct a sparse tensor that encodes geometric (IoU across image plane) and semantic (class, score) relations between 2D and 3D detections.
• Process non-empty tensor elements with a small set of 1x1 convolutions to generate a fused probability map.
• Train the fusion network with cross-entropy loss enhanced by focal loss to handle class imbalance.
• Use a log-likelihood based score for single-modality detectors to improve fusion performance.
• Operate on pre-existing detectors (no re-training required) and deliver low latency (≈<3 ms per frame) on standard GPUs.

Experimental results

Research questions

• RQ1 Can late fusion of 2D and 3D detection candidates outperform single-modality detectors on KITTI?
• RQ2 Does incorporating geometric IoU and semantic consistency improve multi-modal detection accuracy?
• RQ3 How does CLOCs perform when fused with different 2D/3D detectors on KITTI 3D/BEV benchmarks?
• RQ4 What is the impact of using a sparse tensor representation and 1x1 convolutions on speed and memory?
• RQ5 What score representation (log-likelihood vs sigmoid) yields better fusion outcomes?

Key findings

• CLOCs improves multiple fusion baselines across 3D and BEV metrics on KITTI validation, with notable gains at long range (40–50 m).
• Using IoU, 2D score, 3D score, and distance channels yields the best performance; removal of IoU or 3D score degrades results.
• Focal loss and log-likelihood scores outperform alternatives like sigmoid for fusion inputs.
• Fusion with Cascade R-CNN and PV-RCNN etc. achieves leading or near-leading results among fusion methods on KITTI test/validation.
• CLOCs maintains fast inference, adding less than 3 ms per frame on desktop GPUs.