[Paper Review] Diffusion-SS3D: Diffusion Model for Semi-supervised 3D Object Detection

Cheng-Ju Ho, Chen-Hsuan Tai+3 more|arXiv (Cornell University)|Dec 5, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Diffusion-SS3D, a novel semi-supervised 3D object detection framework that leverages a diffusion model to refine noisy pseudo-labels by denoising corrupted 3D object sizes and class label distributions. By integrating the diffusion model into a teacher-student framework, it generates higher-quality pseudo-labels, achieving state-of-the-art performance with a 6%+ mAP@0.5 improvement over prior methods on ScanNet and SUN RGB-D datasets.

ABSTRACT

Semi-supervised object detection is crucial for 3D scene understanding, efficiently addressing the limitation of acquiring large-scale 3D bounding box annotations. Existing methods typically employ a teacher-student framework with pseudo-labeling to leverage unlabeled point clouds. However, producing reliable pseudo-labels in a diverse 3D space still remains challenging. In this work, we propose Diffusion-SS3D, a new perspective of enhancing the quality of pseudo-labels via the diffusion model for semi-supervised 3D object detection. Specifically, we include noises to produce corrupted 3D object size and class label distributions, and then utilize the diffusion model as a denoising process to obtain bounding box outputs. Moreover, we integrate the diffusion model into the teacher-student framework, so that the denoised bounding boxes can be used to improve pseudo-label generation, as well as the entire semi-supervised learning process. We conduct experiments on the ScanNet and SUN RGB-D benchmark datasets to demonstrate that our approach achieves state-of-the-art performance against existing methods. We also present extensive analysis to understand how our diffusion model design affects performance in semi-supervised learning.

Motivation & Objective

• To address the challenge of generating high-quality pseudo-labels in semi-supervised 3D object detection due to the vast and diverse 3D geometric space.
• To improve the reliability of pseudo-labels for unlabeled 3D point clouds, especially in cases of low model recall or poor initial predictions.
• To explore the use of diffusion models as a denoising mechanism for 3D object size and class label distributions to enhance pseudo-label quality.
• To integrate the diffusion model within a teacher-student framework to iteratively refine pseudo-labels and improve semi-supervised learning performance.

Proposed method

• Corrupt 3D object size and class label distributions with Gaussian noise to generate noisy pseudo-labels as input to the diffusion model.
• Use a diffusion model to iteratively denoise the corrupted object sizes and class labels, producing refined, more plausible bounding box predictions.
• Integrate the diffusion model into a teacher-student framework: the student model is trained on labeled data and pseudo-labels, while the teacher uses diffusion sampling to generate improved pseudo-labels.
• Apply farthest point sampling (FPS) to select representative 3D points as centers for generating initial noisy bounding box candidates, improving coverage and recall.
• Use exponential moving average (EMA) to update the teacher model weights, stabilizing pseudo-label generation during training.
• During inference, employ the trained diffusion sampler to generate final predictions by denoising random initial object sizes and class labels.

Experimental results

Research questions

• RQ1 Can a diffusion model effectively improve the quality of pseudo-labels in semi-supervised 3D object detection by denoising corrupted object sizes and class labels?
• RQ2 How does incorporating diffusion-based denoising affect the recall and accuracy of pseudo-labeled 3D bounding boxes compared to standard teacher-student methods?
• RQ3 Does the use of farthest point sampling (FPS) for center selection improve the performance and stability of the diffusion-based pseudo-label generation process?
• RQ4 To what extent does the diffusion model enhance model generalization and convergence speed in low-data regimes on indoor 3D detection benchmarks?
• RQ5 Can the diffusion model maintain consistent performance improvements over extended training epochs in semi-supervised 3D detection?

Key findings

• Diffusion-SS3D achieves a 6%+ improvement in mAP@0.5 over prior state-of-the-art methods on the ScanNet 5% split, reaching 30.93% mAP@0.5 at epoch 1000.
• The model maintains stable and consistent performance gains over time, with mAP@0.5 increasing from 22.42% at epoch 800 to 22.42% at epoch 1000, indicating robust learning.
• The diffusion component improves pseudo-label quality, as evidenced by a 44.17% recall@0.5 on unlabeled data at epoch 1000, significantly higher than the 35.25% recall of the 3DIoUMatch baseline.
• Using farthest point sampling (FPS) for center selection yields better performance than random sampling, with a 27.9% mAP@0.5 when combined with the diffusion model.
• The ablation study confirms that the diffusion model contributes significantly to performance, with a 43.5% mAP@0.25 and 27.9% mAP@0.5 when using FPS and the diffusion component.
• The method demonstrates strong generalization, achieving state-of-the-art results on both ScanNet and SUN RGB-D benchmarks, with consistent improvements across multiple evaluation metrics.