[Paper Review] End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

Yin Zhou, Pei Sun+7 more|arXiv (Cornell University)|Oct 15, 2019

Advanced Neural Network ApplicationsComputer Science174 citations

TL;DR

Proposes an end-to-end multi-view fusion (MVF) framework with dynamic voxelization to fuse BEV and perspective views for improved 3D object detection from LiDAR, achieving superior accuracy over single-view baselines on Waymo and KITTI datasets.

ABSTRACT

Recent work on 3D object detection advocates point cloud voxelization in birds-eye view, where objects preserve their physical dimensions and are naturally separable. When represented in this view, however, point clouds are sparse and have highly variable point density, which may cause detectors difficulties in detecting distant or small objects (pedestrians, traffic signs, etc.). On the other hand, perspective view provides dense observations, which could allow more favorable feature encoding for such cases. In this paper, we aim to synergize the birds-eye view and the perspective view and propose a novel end-to-end multi-view fusion (MVF) algorithm, which can effectively learn to utilize the complementary information from both. Specifically, we introduce dynamic voxelization, which has four merits compared to existing voxelization methods, i) removing the need of pre-allocating a tensor with fixed size; ii) overcoming the information loss due to stochastic point/voxel dropout; iii) yielding deterministic voxel embeddings and more stable detection outcomes; iv) establishing the bi-directional relationship between points and voxels, which potentially lays a natural foundation for cross-view feature fusion. By employing dynamic voxelization, the proposed feature fusion architecture enables each point to learn to fuse context information from different views. MVF operates on points and can be naturally extended to other approaches using LiDAR point clouds. We evaluate our MVF model extensively on the newly released Waymo Open Dataset and on the KITTI dataset and demonstrate that it significantly improves detection accuracy over the comparable single-view PointPillars baseline.

Motivation & Objective

• Motivate leveraging complementary information from BEV and perspective views of the same LiDAR to improve 3D object detection.
• Develop an end-to-end MVF architecture that operates at the point level for effective cross-view feature fusion.
• Introduce dynamic voxelization to preserve all points and enable deterministic voxel embeddings.
• Demonstrate that MVF with dynamic voxelization outperforms single-view baselines on Waymo Open Dataset and KITTI.

Proposed method

• Embed each LiDAR point into a high-dimensional feature space. Apply dynamic voxelization in both BEV (cartesian) and perspective (spherical) views to establish bi-directional point-voxel mappings.
• Compute view-dependent features with per-view FC layers and aggregate voxel information via max pooling.
• Fuse per-point features from: (i) BEV voxel features, (ii) perspective voxel features, and (iii) the point's own features, to produce enhanced point embeddings.
• Process voxel feature maps with a convolutional tower to capture contextual information while preserving resolution.
• Train with the same loss as SECOND and PointPillars, using focal loss for classification and SmoothL1 for regression; optimize with Adam and cosine learning rate decay.

Experimental results

Research questions

• RQ1 Can dual-view (BEV and perspective) representations of the same LiDAR point cloud provide complementary contextual information that improves 3D object detection?
• RQ2 Does dynamic voxelization outperform traditional hard voxelization in preserving information and stabilizing detection?
• RQ3 How does MVF compare to single-view baselines on large-scale and standard benchmarks (Waymo Open Dataset and KITTI) for automotive and pedestrian detection?
• RQ4 Is the MVF approach generalizable to other LiDAR-based detectors beyond the baseline used?

Key findings

• MVF with dynamic voxelization consistently improves detection accuracy over HV+SV and DV+SV baselines on Waymo vehicle and pedestrian tasks.
• Dynamic voxelization preserves all points and voxels, yielding deterministic voxel embeddings and reduced information loss.
• Combining BEV and perspective views provides complementary context, with larger gains at longer ranges and for small/occluded objects like pedestrians.
• On Waymo dataset, MVF achieves higher BEV and 3D AP than HV+SV and DV+SV across distance ranges (0-30m, 30-50m, and beyond 50m).
• On KITTI, MVF attains competitive 3D car detection performance, surpassing HV+SV and DV+SV in easy/moderate/hard settings.
• MVF demonstrates favorable latency characteristics compared to baseline methods, enabling practical real-time inference.