[Paper Review] VIN: Voxel-based Implicit Network for Joint 3D Object Detection and Segmentation for Lidars

Yuanxin Zhong, Minghan Zhu+1 more|arXiv (Cornell University)|Jul 7, 2021

Remote Sensing and LiDAR Applications52 references4 citations

TL;DR

This paper proposes VIN (Voxel-based Implicit Network), a unified 3D object detection and semantic segmentation framework for LiDAR point clouds that leverages a lightweight semantic branch attached to a voxel-based detector to generate implicit semantic representations. The method achieves state-of-the-art performance on nuScenes-lidarseg with minimal computational overhead and demonstrates strong weakly supervised learning capability using only 0.1% labeled semantic points.

ABSTRACT

A unified neural network structure is presented for joint 3D object detection and point cloud segmentation in this paper. We leverage rich supervision from both detection and segmentation labels rather than using just one of them. In addition, an extension based on single-stage object detectors is proposed based on the implicit function widely used in 3D scene and object understanding. The extension branch takes the final feature map from the object detection module as input, and produces an implicit function that generates semantic distribution for each point for its corresponding voxel center. We demonstrated the performance of our structure on nuScenes-lidarseg, a large-scale outdoor dataset. Our solution achieves competitive results against state-of-the-art methods in both 3D object detection and point cloud segmentation with little additional computation load compared with object detection solutions. The capability of efficient weakly supervision semantic segmentation of the proposed method is also validated by experiments.

Motivation & Objective

• To develop a unified deep learning framework that jointly performs 3D object detection and semantic segmentation on LiDAR point clouds.
• To minimize computational overhead by reusing features from a single-stage 3D detector for semantic segmentation.
• To enable effective weakly supervised semantic segmentation using only a small fraction (0.1%) of annotated semantic labels.
• To resolve inconsistencies between 3D bounding box predictions and point-wise semantic labels through a novel training strategy.
• To demonstrate the utility of the implicit function for dense semantic map generation and downsampled point cloud inference.

Proposed method

• The method extends a single-stage 3D object detector by adding a semantic branch that learns an implicit function from the final feature map of the backbone network.
• The implicit function predicts semantic distributions at any 3D coordinate by querying the feature map, enabling semantic inference at arbitrary spatial locations.
• The semantic branch is trained using weak supervision from bounding box labels and sparse semantic annotations, reducing reliance on dense point-level labels.
• A consistency loss is introduced to align predicted semantic labels with the 3D bounding box predictions, reducing misalignment between detection and segmentation outputs.
• The framework supports efficient inference on downsampled point clouds by querying the implicit function at unmeasured locations, preserving semantic accuracy.
• Dense semantic maps are generated by querying the implicit function on a regular grid, enabling bird’s-eye view semantic representations.

Experimental results

Research questions

• RQ1 Can a unified network architecture effectively perform both 3D object detection and semantic segmentation on LiDAR point clouds with minimal computational cost?
• RQ2 To what extent can semantic segmentation be effectively supervised using only bounding box annotations and a small fraction of point-level labels?
• RQ3 How can inconsistencies between 3D detection boxes and point-wise semantic labels be minimized during joint training?
• RQ4 Can the implicit function learned by the semantic branch generalize to downsampled or incomplete point clouds while maintaining segmentation accuracy?
• RQ5 Can the implicit function enable efficient generation of dense semantic maps for downstream tasks like scene understanding or mapping?

Key findings

• VIN achieves state-of-the-art performance on the nuScenes-lidarseg benchmark for both 3D object detection and semantic segmentation, outperforming existing methods.
• With only 0.1% of semantic labels, the model achieves mIoU of 73.7 on fully observed point clouds and 73.7 even when using only 16 beams (32% of original point cloud density).
• The model maintains high segmentation accuracy (70.2 mIoU) when inferring on 16-beam downsampled inputs, outperforming nearest-neighbor baseline (55.9 mIoU) under the same conditions.
• Inference speed is nearly identical to the base detector (5.9 FPS vs. 6.0 FPS on a single 2080Ti), demonstrating minimal overhead for joint segmentation.
• The implicit function enables accurate semantic prediction in regions with no LiDAR measurements when the input is downsampled, showing robustness to sparse inputs.
• Dense semantic maps generated via implicit function queries show coherent semantic boundaries and are useful for visual interpretation and downstream tasks.