[Paper Review] H3DNet: 3D Object Detection Using Hybrid Geometric Primitives

Zaiwei Zhang, Bo Sun+2 more|arXiv (Cornell University)|Jun 10, 2020

3D Shape Modeling and Analysis53 references17 citations

TL;DR

H3DNet proposes a novel 3D object detection framework that regresses a hybrid set of geometric primitives—bounding box (BB) centers, face centers, and edge centers—from a colorless 3D point cloud. By defining a differentiable distance function to optimize object proposals continuously, and using a matching and refinement module to classify and fine-tune detections, H3DNet achieves state-of-the-art performance on ScanNet (67.2% mAP@0.25) and SUN RGB-D (60.1% mAP@0.25), with significant gains on challenging categories.

ABSTRACT

We introduce H3DNet, which takes a colorless 3D point cloud as input and outputs a collection of oriented object bounding boxes (or BB) and their semantic labels. The critical idea of H3DNet is to predict a hybrid set of geometric primitives, i.e., BB centers, BB face centers, and BB edge centers. We show how to convert the predicted geometric primitives into object proposals by defining a distance function between an object and the geometric primitives. This distance function enables continuous optimization of object proposals, and its local minimums provide high-fidelity object proposals. H3DNet then utilizes a matching and refinement module to classify object proposals into detected objects and fine-tune the geometric parameters of the detected objects. The hybrid set of geometric primitives not only provides more accurate signals for object detection than using a single type of geometric primitives, but it also provides an overcomplete set of constraints on the resulting 3D layout. Therefore, H3DNet can tolerate outliers in predicted geometric primitives. Our model achieves state-of-the-art 3D detection results on two large datasets with real 3D scans, ScanNet and SUN RGB-D.

Motivation & Objective

• To improve 3D object detection in complex, real-world scenes with irregular point cloud inputs and variable object counts.
• To address the challenge of noisy or imprecise geometric cues in 3D detection by leveraging an overcomplete set of geometric primitives.
• To enhance detection accuracy by integrating diverse geometric signals (centers, faces, edges) and enabling robust, continuous optimization of object proposals.
• To develop a unified, end-to-end deep learning framework that jointly regresses geometric primitives, generates proposals, and refines detections with semantic labels.
• To demonstrate that hybrid geometric primitive regression leads to more robust and accurate 3D object detection than single-primitive or non-geometric approaches.

Proposed method

• H3DNet regresses three types of geometric primitives: BB centers, BB face centers, and BB edge centers, from dense pointwise descriptors in a 3D point cloud.
• It defines a parametric distance function that measures the distance between a candidate 3D bounding box and the predicted geometric primitives, enabling continuous optimization of object proposals.
• Local minima of the distance function correspond to high-fidelity object proposals, which are refined through a differentiable optimization process.
• A matching and refinement module classifies proposals as detected objects, predicts geometric offsets (center, size, orientation), and assigns semantic labels using aggregated latent features from matched primitives.
• The network employs multiple descriptor computation towers to process diverse geometric cues and supports variable numbers of primitives per object.
• Feature aggregation across matched primitives enhances detection confidence and geometric accuracy by combining complementary geometric and semantic signals.

Experimental results

Research questions

• RQ1 Can a hybrid set of geometric primitives (centers, faces, edges) improve 3D object detection performance compared to using a single primitive type?
• RQ2 Does continuous optimization of object proposals via a differentiable distance function lead to higher-quality detections than discrete or non-optimized proposal generation?
• RQ3 Can an overcomplete set of geometric primitives improve robustness to prediction errors and outliers in real-world 3D scans?
• RQ4 How does the integration of multiple primitive types affect detection accuracy on challenging object categories with distinct geometric structures?
• RQ5 To what extent does feature aggregation from matched geometric primitives enhance classification and refinement of detected objects?

Key findings

• H3DNet achieves 67.2% mAP@0.25 on ScanNet, representing an 8.5% relative improvement over prior state-of-the-art methods that use only 3D point positions.
• On SUN RGB-D, H3DNet achieves 60.1% mAP@0.25, a 2.4% relative improvement over the same baseline methods.
• Performance gains are especially pronounced on difficult categories: window (38.1% → 51.9%), door (47.3% → 61.0%), and shower-curtain (57.1% → 75.3%) on ScanNet.
• Removing refinement of geometric parameters (center, size, angle, existence, or semantic scores) leads to a 14.6% drop in mAP@0.5 on ScanNet, demonstrating the critical role of refinement.
• Using only edge or face primitives results in suboptimal performance on certain categories—e.g., edge-only detection fails on shower-curtain, while face-only detection underperforms on windows—highlighting the benefit of hybrid primitives.
• Adding more descriptor computation towers improves performance (e.g., mAP@0.25 increases from 64.4 to 67.2), but gains diminish beyond four towers, indicating diminishing returns.