[Paper Review] FFPA-Net: Efficient Feature Fusion with Projection Awareness for 3D Object Detection

Chaokang Jiang, Guangming Wang+3 more|arXiv (Cornell University)|Sep 15, 2022

Advanced Neural Network Applications4 citations

TL;DR

FFPA-Net proposes an efficient 3D object detection network that uses projection-aware convolution to accelerate LiDAR point cloud feature extraction and introduces two plug-and-play fusion modules—LiCamFuse and BiLiCamFuse—for accurate, real-time cross-modal feature fusion between LiDAR and RGB data. The method achieves state-of-the-art 3D detection performance on KITTI with only 59.40 ms inference time on an RTX 2080Ti.

ABSTRACT

Promising complementarity exists between the texture features of color images and the geometric information of LiDAR point clouds. However, there still present many challenges for efficient and robust feature fusion in the field of 3D object detection. In this paper, first, unstructured 3D point clouds are filled in the 2D plane and 3D point cloud features are extracted faster using projection-aware convolution layers. Further, the corresponding indexes between different sensor signals are established in advance in the data preprocessing, which enables faster cross-modal feature fusion. To address LiDAR points and image pixels misalignment problems, two new plug-and-play fusion modules, LiCamFuse and BiLiCamFuse, are proposed. In LiCamFuse, soft query weights with perceiving the Euclidean distance of bimodal features are proposed. In BiLiCamFuse, the fusion module with dual attention is proposed to deeply correlate the geometric and textural features of the scene. The quantitative results on the KITTI dataset demonstrate that the proposed method achieves better feature-level fusion. In addition, the proposed network shows a shorter running time compared to existing methods.

Motivation & Objective

• Address the challenge of inefficient and inaccurate feature fusion between sparse LiDAR point clouds and dense RGB images in 3D object detection.
• Reduce computational overhead from KNN and farthest point sampling in point cloud processing by projecting 3D coordinates into a 2D plane with preserved geometric information.
• Mitigate cross-modal misalignment between LiDAR points and image pixels through learned, distance-aware fusion weights and bidirectional feature interaction.
• Design plug-and-play modules that enable efficient, high-accuracy fusion without architectural overhaul.
• Achieve real-time inference speed while maintaining or improving detection accuracy on standard benchmarks like KITTI.

Proposed method

• Project raw 3D LiDAR coordinates into a 2D plane to enable faster, structured feature extraction using projection-aware convolution layers.
• Preprocess data to precompute and store cross-modal correspondence indexes between LiDAR points and image pixels, enabling fast lookup during fusion.
• Introduce LiCamFuse, a fusion module that computes soft query weights based on Euclidean distance between bimodal features to improve alignment.
• Propose BiLiCamFuse, a dual-attention fusion module that enables bidirectional feature interaction by allowing LiDAR points to query neighboring pixel features and vice versa.
• Use local KNN operations on the projected 2D map to efficiently extract local geometric and texture features, reducing computational cost.
• Integrate the fusion modules into an end-to-end 3D detection network for joint optimization of feature learning and detection.

Experimental results

Research questions

• RQ1 Can projection-aware feature learning reduce computational cost in 3D object detection while preserving geometric fidelity of LiDAR point clouds?
• RQ2 How can cross-modal misalignment between LiDAR points and image pixels be effectively mitigated during feature fusion?
• RQ3 To what extent can plug-and-play fusion modules like LiCamFuse and BiLiCamFuse improve detection accuracy without compromising inference speed?
• RQ4 What is the optimal projection resolution for balancing feature completeness and zero-point density in 2D LiDAR projections?
• RQ5 How does the proposed fusion strategy compare to existing methods in terms of accuracy, speed, and robustness on standard benchmarks?

Key findings

• FFPA-Net achieves 91.93% 3D detection mAP (IoU=0.7) on the KITTI validation set for easy samples, outperforming PointRCNN and EPNet.
• The network runs in only 59.40 ms per inference on an NVIDIA RTX 2080Ti GPU, significantly faster than many LiDAR-only or multimodal baselines.
• BiLiCamFuse achieves the highest mAP of 91.93% on easy samples, demonstrating the effectiveness of deep, bidirectional feature fusion.
• LiCamFuse improves fusion accuracy by incorporating Euclidean distance into soft query weights, reducing misalignment between modalities.
• The optimal projection resolution was found to be 40×275, balancing feature retention and computational efficiency.
• Ablation studies confirm that both LiCamFuse and BiLiCamFuse contribute significantly to performance, with BiLiCamFuse providing the best overall results at a modest speed cost.