[Paper Review] Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection

Zhixin Wang, Kui Jia|arXiv (Cornell University)|Mar 5, 2019

Advanced Neural Network Applications30 references162 citations

TL;DR

Frustum ConvNet (F-ConvNet) introduces sliding frustums to group local points from 2D region proposals, aggregates them with PointNet into frustum-level features, fuses with a fully convolutional network for end-to-end oriented 3D box estimation, and achieves state-of-the-art results on SUN-RGBD and KITTI.

ABSTRACT

In this work, we propose a novel method termed \emph{Frustum ConvNet (F-ConvNet)} for amodal 3D object detection from point clouds. Given 2D region proposals in an RGB image, our method first generates a sequence of frustums for each region proposal, and uses the obtained frustums to group local points. F-ConvNet aggregates point-wise features as frustum-level feature vectors, and arrays these feature vectors as a feature map for use of its subsequent component of fully convolutional network (FCN), which spatially fuses frustum-level features and supports an end-to-end and continuous estimation of oriented boxes in the 3D space. We also propose component variants of F-ConvNet, including an FCN variant that extracts multi-resolution frustum features, and a refined use of F-ConvNet over a reduced 3D space. Careful ablation studies verify the efficacy of these component variants. F-ConvNet assumes no prior knowledge of the working 3D environment and is thus dataset-agnostic. We present experiments on both the indoor SUN-RGBD and outdoor KITTI datasets. F-ConvNet outperforms all existing methods on SUN-RGBD, and at the time of submission it outperforms all published works on the KITTI benchmark. Code has been made available at: {\url{https://github.com/zhixinwang/frustum-convnet}.}

Motivation & Objective

• Leverage 2D region proposals to guide amodal 3D object detection from point clouds.
• Introduce a sequence of sliding frustums to group local points and learn frustum-level features.
• Fuse frustum-level features with a fully convolutional network for end-to-end 3D box estimation.
• Extend the framework with multi-resolution frustum features and a refinement strategy.
• Demonstrate dataset-agnostic performance on indoor SUN-RGBD and outdoor KITTI datasets.

Proposed method

• Generate a sequence of overlapping frustums per 2D region proposal by sliding along the frustum axis.
• Apply PointNet to each frustum with relative coordinates to obtain frustum-level feature vectors.
• Arrange frustum features into a 2D feature map and process with a multi-block FCN to fuse features across frustums and regress 3D boxes.
• Optionally employ a multi-resolution frustum feature integration variant to capture multiple scales.
• Optionally apply a final refinement step by re-running the network on points inside an expanded predicted 3D box.
• Train with a multi-task loss including classification with focal loss, regression (center and size/angle offsets), and a corner loss.

Experimental results

Research questions

• RQ1 Can sliding frustums improve local point feature aggregation for amodal 3D detection?
• RQ2 Does FCN-based fusion of frustum-level features enable accurate end-to-end oriented 3D box estimation?
• RQ3 Do multi-resolution frustum features and refined 3D-space usage improve detection accuracy?
• RQ4 How well does F-ConvNet generalize across indoor SUN-RGBD and outdoor KITTI datasets?

Key findings

• F-ConvNet achieves state-of-the-art performance on SUN-RGBD for amodal 3D object detection.
• On KITTI, F-ConvNet outperforms all published works at the time of submission for the evaluated tasks.
• Ablation studies confirm the efficacy of frustum-based grouping, PointNet-based frustum features, FCN fusion, multi-resolution variants, focal loss, and the final refinement.
• Dataset-agnostic design enables effective detection without reliance on prior environmental knowledge.