[Paper Review] Enabling Efficient Deep Convolutional Neural Network-based Sensor Fusion for Autonomous Driving

Xiaoming Zeng, Zhendong Wang+1 more|arXiv (Cornell University)|Feb 22, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a feature-matching framework to improve deep convolutional neural network (DCNN)-based sensor fusion in autonomous driving by addressing feature-mismatching in element-wise fusion. It introduces a feature disparity metric, Fusion-filter for feature matching, Layer-sharing to reduce computational cost, and a Feature Disparity Loss to guide training—achieving higher accuracy on KITTI with lower inference overhead than state-of-the-art methods.

ABSTRACT

Autonomous driving demands accurate perception and safe decision-making. To achieve this, automated vehicles are now equipped with multiple sensors (e.g., camera, Lidar, etc.), enabling them to exploit complementary environmental context by fusing data from different sensing modalities. With the success of Deep Convolutional Neural Network(DCNN), the fusion between DCNNs has been proved as a promising strategy to achieve satisfactory perception accuracy. However, mainstream existing DCNN fusion schemes conduct fusion by directly element-wisely adding feature maps extracted from different modalities together at various stages, failing to consider whether the features being fused are matched or not. Therefore, we first propose a feature disparity metric to quantitatively measure the degree of feature disparity between the feature maps being fused. We then propose Fusion-filter as a feature-matching techniques to tackle the feature-mismatching issue. We also propose a Layer-sharing technique in the deep layer that can achieve better accuracy with less computational overhead. Together with the help of the feature disparity to be an additional loss, our proposed technologies enable DCNN to learn corresponding feature maps with similar characteristics and complementary visual context from different modalities to achieve better accuracy. Experimental results demonstrate that our proposed fusion technique can achieve better accuracy on KITTI dataset with less computational resources demand.

Motivation & Objective

• To address the issue of feature-mismatching in conventional DCNN-based middle fusion, where feature maps from different modalities are fused without considering their semantic correspondence.
• To quantify feature disparity between fused feature maps using a novel metric that measures visual characteristic differences.
• To improve fusion accuracy by introducing Fusion-filter, a learnable mechanism that aligns features from different modalities before element-wise summation.
• To reduce computational overhead in deep layers by proposing Layer-sharing, which reuses filters across modality branches based on feature similarity.
• To enhance training stability and feature alignment by incorporating Feature Disparity Loss as an auxiliary objective during backpropagation.

Proposed method

• Introduces a feature disparity (FD) metric to quantitatively measure the difference in visual characteristics between feature maps from different sensors (e.g., RGB and LiDAR).
• Proposes Fusion-filter, a learnable module that learns to match features across modalities before fusion, ensuring that only semantically aligned features are combined.
• Designs a Layer-sharing architecture that shares convolutional filters in deeper layers across modality branches, reducing model parameters and MACs.
• Introduces Feature Disparity Loss as an additional training objective that minimizes the FD between fused feature maps, promoting alignment during optimization.
• Employs a multi-stage fusion strategy with crossing skip connections, combining early, middle, and late fusion benefits through structured feature integration.
• Uses the KITTI dataset for evaluation, training models with and without the proposed components to isolate their effects on accuracy and efficiency.

Experimental results

Research questions

• RQ1 Does feature-mismatching in element-wise fusion of multi-modal features degrade perception accuracy in autonomous driving?
• RQ2 Can a quantitative feature disparity metric effectively measure the mismatch between features from different sensing modalities?
• RQ3 Can a learnable fusion mechanism (Fusion-filter) improve feature matching and thus enhance segmentation accuracy?
• RQ4 Can Layer-sharing in deep layers reduce computational cost without sacrificing accuracy?
• RQ5 Does incorporating Feature Disparity Loss during training lead to better feature alignment and improved performance?

Key findings

• The proposed Feature Disparity Loss significantly improves model performance, with BaseLoss achieving the best F-score in the challenging UU road scene, outperforming the baseline.
• AllFilter_U, which uses unidirectional Fusion-filter, outperforms the baseline in all three road scenes and ranks in the top three for three out of five metrics.
• AllFilter_B, with bidirectional Fusion-filter, achieves better accuracy than the baseline in UM and UMM scenes, with more balanced segmentation results.
• BaseSharing achieves the best accuracy among all models in the UMM scene with the lowest computational cost in terms of parameters and MACs.
• WeightedSharing, which dynamically adjusts fusion weights, outperforms the baseline in all three road scenes and achieves the best performance on three metrics in the UU scenario, despite having fewer parameters.
• The ablation study confirms that models trained with Feature Disparity Loss consistently outperform their counterparts without it, validating the effectiveness of the proposed loss.