[Paper Review] Multi-modal Sensor Fusion for Auto Driving Perception: A Survey

Finkenzeller, Andreas, Roberts, Andrew+4 more|arXiv (Cornell University)|Feb 6, 2022

Advanced Neural Network ApplicationsComputer Science90 citations

TL;DR

This survey proposes a novel taxonomy for multi-modal sensor fusion in autonomous driving perception, categorizing methods into two major classes—strong-fusion and weak-fusion—with four sub-classes based on fusion stage and feature representation. It provides a systematic review of 50+ papers, analyzes data formats, identifies key challenges like domain bias and resolution mismatch, and outlines future research directions in self-supervised learning and multi-source information integration.

ABSTRACT

Environmental perception and 3D object detection are key factors for advancing autonomous driving and require robust security measures to ensure optimal performance and safety. However, established methods often focus only on protecting the involved data and overlook synchronization and timing aspects, which are equally crucial for ensuring profound system security. For instance, multi-modal sensor fusion techniques for object detection can be affected by input desynchronization resulting from random communication delays or malicious cyber attacks, as these techniques combine various sensor inputs to extract shared features present in their data streams simultaneously. Current research acknowledges the importance of temporal alignment in this context. However, the presented studies typically assume genuine system behavior and neglect the potential threat of malicious attacks, as the suggested solutions lack strategies to prevent intentional data misalignment. Additionally, they do not adequately address how sensor input desynchronization affects fusion performance in depth. This paper investigates how desynchronization attacks impact sensor fusion algorithms for 3D object detection. We evaluate how varying sensor delays affect the detection performance and link our findings to the internal architecture of the sensor fusion algorithms and the influence of specific traffic scenarios and their dynamics. We compiled four datasets covering typical traffic scenarios for our empirical evaluation and tested them on four representative fusion algorithms. Our results show that all evaluated algorithms are vulnerable to input desynchronization, as the performance declines with increasing sensor delays, highlighting the existing lack of resilience to desynchronization attacks. Furthermore, we observe that the Light Detection and Ranging (LiDAR) sensor is significantly more susceptible to delays than the camera. Finally, our experiments indicate that the chosen fusion architecture correlates with the system’s resilience against desynchronization, as our results demonstrate that the early fusion approach provides greater robustness than others.

Motivation & Objective

• To address the limitations of traditional three-class fusion taxonomies (early, deep, late fusion) that lack clarity in feature representation and fail to capture asymmetric fusion patterns.
• To provide a comprehensive review of over 50 recent papers on multi-modal perception using LiDAR and camera sensors for tasks like 3D object detection and semantic segmentation.
• To identify persistent challenges in multi-modal fusion, including domain bias, resolution discrepancies, and information loss during feature transformation.
• To propose future research directions, such as self-supervised representation learning and leveraging temporal, spatial, and contextual information across multiple sensors and frames.
• To offer a systematic, stage-based classification framework that better reflects the evolving complexity of modern fusion architectures.

Proposed method

• Proposes a new taxonomy dividing fusion methods into two major classes: strong-fusion and weak-fusion, with strong-fusion further subdivided into early-fusion, deep-fusion, late-fusion, and asymmetry-fusion based on the fusion stage and feature representation.
• Analyzes and compares data formats and representations for LiDAR (point clouds, BEV projections, voxel grids) and camera (RGB images, feature maps), highlighting differences in resolution, structure, and information content.
• Evaluates fusion mechanisms such as concatenation, element-wise multiplication, and bilinear pooling, arguing for more sophisticated operations to bridge semantic gaps between modalities.
• Introduces a stage-based classification system that clarifies feature-level fusion by explicitly defining the role of each modality at different processing stages, avoiding the symmetry assumption of traditional methods.
• Proposes that future fusion models should integrate multi-source information—including semantic segmentation, lane detection, and temporal sequences—through joint learning frameworks.
• Advocates for self-supervised and contrastive learning techniques to exploit inherent cross-modal correspondence in real-world scenes, improving feature alignment without relying on paired annotations.

Experimental results

Research questions

• RQ1 How can a more systematic and precise taxonomy be developed for multi-modal sensor fusion in autonomous driving perception, beyond the traditional early/deep/late fusion classification?
• RQ2 What are the key differences in data representation and feature characteristics between LiDAR point clouds and camera images, and how do they affect fusion performance?
• RQ3 What are the main unresolved challenges in multi-modal fusion, such as domain bias, resolution mismatch, and information loss during projection?
• RQ4 How can future fusion models better leverage multi-source information, including semantic, spatial, and temporal context, to improve robustness and accuracy?
• RQ5 What role can self-supervised learning play in enhancing cross-modal feature alignment and reducing reliance on large-scale annotated datasets?

Key findings

• The proposed taxonomy—strong-fusion and weak-fusion with four sub-classes—offers a clearer, more accurate classification of current fusion methods than traditional early/deep/late fusion, especially in capturing asymmetric fusion patterns.
• LiDAR and camera data exhibit significant differences in resolution, structure, and information content, with LiDAR suffering from low spatial density and cameras facing occlusion and lighting challenges.
• Information loss is substantial during dimensionality reduction, such as projecting 3D point clouds into 2D bird’s-eye view (BEV), which can be mitigated by designing fusion-friendly, high-dimensional representations.
• Current fusion operations like concatenation and element-wise multiplication are insufficient for handling large distribution discrepancies between modalities, suggesting a need for advanced mechanisms like bilinear pooling.
• Domain bias due to sensor type, weather, location, and season limits model generalization, and future models must adaptively integrate diverse data sources to improve robustness.
• Self-supervised and contrastive learning methods show strong potential for improving cross-modal feature alignment and reducing annotation dependency, offering a promising path for future research.