[论文解读] Joint 3D Proposal Generation and Object Detection from View Aggregation

Jason S. Ku, Melissa Mozifian+3 more|arXiv (Cornell University)|Dec 6, 2017

Advanced Neural Network Applications参考文献 18被引用 135

一句话总结

AVOD 在一个两阶段网络（RPN 和第二阶段检测器）中融合高分辨率的 LiDAR BEV 和 RGB 图像特征，生成 3D 候选框并准确检测定向的 3D 边界框，在 KITTI 数据集上达到最先进的结果，并实现实时性能。

ABSTRACT

We present AVOD, an Aggregate View Object Detection network for autonomous driving scenarios. The proposed neural network architecture uses LIDAR point clouds and RGB images to generate features that are shared by two subnetworks: a region proposal network (RPN) and a second stage detector network. The proposed RPN uses a novel architecture capable of performing multimodal feature fusion on high resolution feature maps to generate reliable 3D object proposals for multiple object classes in road scenes. Using these proposals, the second stage detection network performs accurate oriented 3D bounding box regression and category classification to predict the extents, orientation, and classification of objects in 3D space. Our proposed architecture is shown to produce state of the art results on the KITTI 3D object detection benchmark while running in real time with a low memory footprint, making it a suitable candidate for deployment on autonomous vehicles. Code is at: https://github.com/kujason/avod

研究动机与目标

• Bridge the gap between 2D detection progress and 3D detection by leveraging multimodal data (LiDAR and images).
• Develop a high-resolution, low-footprint feature extractor for BEV and image spaces.
• Design a multimodal RPN that achieves high recall for small object classes in road scenes.
• Propose a compact, physically-consistent 3D box encoding and explicit orientation regression.
• Demonstrate real-time performance and robustness on KITTI under challenging conditions.

提出的方法

• Generate six-channel BEV maps from voxelized LiDAR data with height and density channels.
• Use a high-resolution feature extractor with an encoder–decoder (FPN-inspired) to produce shared feature maps for both views.
• Implement a multimodal fusion RPN that projects 3D anchors into BEV and image feature maps, applies 1×1 convolutions for dimensionality reduction, and fuses Crops via crop-and-resize to predict 3D proposals.
• Employ axis-aligned 3D anchors sampled in BEV, with recall-focused training and 2D BEV IoU-based pruning of anchors.
• Use a second-stage detector with a 4-corner box encoding plus top and bottom height offsets, and an explicit orientation vector (cosθ, sinθ) regression to resolve orientation ambiguities.
• Train RPN and detector jointly end-to-end with multitask losses (Smooth L1 for box parameters, cross-entropy for objectness/classification) and 2D BEV NMS for proposals.

实验结果

研究问题

• RQ1 Can multimodal fusion of high-resolution BEV LiDAR features and RGB image features improve 3D proposal recall and final 3D detections in autonomous driving?
• RQ2 Does a high-resolution feature extractor combined with a multiview RPN enable better localization and orientation estimation for small objects in road scenes?
• RQ3 What is the impact of a compact 4-corner 3D box encoding plus explicit orientation regression on 3D detection performance and orientation accuracy?
• RQ4 Is the AVOD approach capable of real-time inference with a small memory footprint on standard hardware while maintaining state-of-the-art accuracy?

主要发现

• The Feature Pyramid fusion RPN achieves 86% 3D recall for cars with only 10 proposals per frame.
• AVOD outperforms 3DOP and Mono3D in 3D proposal recall across car, pedestrian, and cyclist classes.
• In KITTI validation, AVOD with Feature Pyramid delivers state-of-the-art 3D AP and BEV AP on cars, and strong results for pedestrians with substantial gains from the high-resolution extractor.
• On KITTI test set, AVOD (Feature Pyramid) achieves leading 3D AP and BEV AP for cars and pedestrians, with competitive results for cyclists and favorable runtime (0.1s per frame) on TITAN Xp.
• The proposed 4-corner plus top/bottom height encoding with explicit orientation regression improves orientation accuracy and reduces ambiguity compared to prior encodings.
• The high-resolution feature extractor significantly boosts performance for small classes (pedestrians, cyclists) with manageable increases in computation and memory.