[Paper Review] V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection

Yichao Shen, Zigang Geng+7 more|arXiv (Cornell University)|Aug 8, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes V-DETR, a 3D object detection framework based on DETR with a novel 3D Vertex Relative Position Encoding (3DV-RPE) that improves attention localization by encoding relative positions of points to predicted 3D box vertices. The method achieves state-of-the-art performance on ScanNetV2 and SUN RGB-D, boosting AP_{25}/AP_{50} to 77.8%/66.0%, significantly outperforming prior 3DETR and fully convolutional methods.

ABSTRACT

We introduce a highly performant 3D object detector for point clouds using the DETR framework. The prior attempts all end up with suboptimal results because they fail to learn accurate inductive biases from the limited scale of training data. In particular, the queries often attend to points that are far away from the target objects, violating the locality principle in object detection. To address the limitation, we introduce a novel 3D Vertex Relative Position Encoding (3DV-RPE) method which computes position encoding for each point based on its relative position to the 3D boxes predicted by the queries in each decoder layer, thus providing clear information to guide the model to focus on points near the objects, in accordance with the principle of locality. In addition, we systematically improve the pipeline from various aspects such as data normalization based on our understanding of the task. We show exceptional results on the challenging ScanNetV2 benchmark, achieving significant improvements over the previous 3DETR in $ m{AP}_{25}$/$ m{AP}_{50}$ from 65.0\%/47.0\% to 77.8\%/66.0\%, respectively. In addition, our method sets a new record on ScanNetV2 and SUN RGB-D datasets.Code will be released at http://github.com/yichaoshen-MS/V-DETR.

Motivation & Objective

• To address the poor localization of attention in 3D object detection using DETR, where queries often attend to distant points, violating the locality principle.
• To improve inductive bias learning in Transformers for 3D point clouds despite limited training data.
• To design a position encoding scheme that provides clear, consistent positional cues based on relative positions to predicted 3D box vertices.
• To enhance model performance through systematic pipeline improvements, including object-based normalization and architectural refinements.
• To establish a strong DETR-based baseline for 3D indoor object detection that outperforms existing fully convolutional and attention-based methods.

Proposed method

• Introduces 3DV-RPE, which computes position encoding for each 3D point based on its relative offset to the eight vertices of the 3D box predicted by the query in each decoder layer.
• Applies canonical object space transformation to rotate each object so that its 3D box aligns with the coordinate system, ensuring consistent position encoding across orientations.
• Uses a sparse 3D ResNet-34 with FPN neck as the backbone encoder, followed by a Transformer decoder with cross-attention using 3DV-RPE.
• Employs object-based normalization to parameterize 3D bounding boxes, improving stability compared to scene-based normalization.
• Adapts recent 2D DETR advancements, such as query initialization and training schedules, to the 3D point cloud setting.
• Avoids voxel expansion in the FPN neck, preserving original 3D surface details and improving performance over methods relying on such operations.

Experimental results

Research questions

• RQ1 Can a learnable position encoding based on relative positions to predicted 3D box vertices improve attention localization in 3D object detection?
• RQ2 Does transforming points and boxes into a canonical object space enhance the consistency and generalization of position encoding across object rotations?
• RQ3 How does 3DV-RPE compare to other attention modulation techniques like 3D box masks or plain positional encoding in terms of performance and attention distribution?
• RQ4 To what extent do pipeline-level improvements—such as object-based normalization and training schedule—contribute to performance gains in 3D DETR?
• RQ5 Can a pure DETR-based architecture outperform fully convolutional 3D detection methods on challenging benchmarks like ScanNetV2 and SUN RGB-D?

Key findings

• V-DETR achieves 77.8% AP_{25} and 66.0% AP_{50} on the ScanNetV2 benchmark, a significant improvement over the previous 3DETR baseline of 65.0% and 47.0%.
• The method sets a new state-of-the-art record on both ScanNetV2 and SUN RGB-D datasets, demonstrating strong generalization across domains.
• 3DV-RPE reduces attention to irrelevant regions outside the ground-truth box, as visualized in attention maps, with weights concentrated on sparse object boundaries.
• Object-based normalization improves AP_{50} by 3.9 percentage points compared to scene-based normalization, highlighting its importance for stability in 3D detection.
• Voxel expansion degrades performance in DETR-based models, confirming that V-DETR preserves surface detail better than methods relying on such operations.
• Even with longer training (540 epochs), the performance gap between 3DV-RPE and 3D box mask remains, confirming the superiority of 3DV-RPE.