[Paper Review] 3DOS: Towards 3D Open Set Learning -- Benchmarking and Understanding Semantic Novelty Detection on Point Clouds

Antonio Alliegro, Francesco Cappio Borlino+1 more|arXiv (Cornell University)|Jul 23, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper introduces 3DOS, the first benchmark for 3D Open Set Learning, evaluating methods on synthetic-to-synthetic, real-to-real, and synthetic-to-real settings to assess semantic novelty detection on point clouds. It finds that simple methods like Cosine proto and MSP with PointNet++ often outperform complex state-of-the-art 2D Open Set techniques, highlighting the challenge of domain shift and the robustness of PointNet++ in real-world scenarios.

ABSTRACT

In recent years there has been significant progress in the field of 3D learning on classification, detection and segmentation problems. The vast majority of the existing studies focus on canonical closed-set conditions, neglecting the intrinsic open nature of the real-world. This limits the abilities of robots and autonomous systems involved in safety-critical applications that require managing novel and unknown signals. In this context exploiting 3D data can be a valuable asset since it provides rich information about the geometry of perceived objects and scenes. With this paper we provide the first broad study on 3D Open Set learning. We introduce 3DOS: a novel testbed for semantic novelty detection that considers several settings with increasing difficulties in terms of semantic (category) shift, and covers both in-domain (synthetic-to-synthetic, real-to-real) and cross-domain (synthetic-to-real) scenarios. Moreover, we investigate the related 2D Open Set literature to understand if and how its recent improvements are effective on 3D data. Our extensive benchmark positions several algorithms in the same coherent picture, revealing their strengths and limitations. The results of our analysis may serve as a reliable foothold for future tailored 3D Open Set methods.

Motivation & Objective

• To address the lack of standardized evaluation for 3D Open Set Learning, particularly in real-world deployment scenarios involving distribution shifts.
• To benchmark existing 2D Open Set methods on 3D point cloud data to assess their transferability and effectiveness.
• To identify the most robust architectures and methods for handling semantic and domain shifts in 3D point cloud classification.
• To provide a comprehensive, reproducible benchmark that enables future development of tailored 3D Open Set learning methods.

Proposed method

• Proposes 3DOS, a multi-track benchmark covering Synthetic, Real-to-Real, and Synthetic-to-Real settings to evaluate increasing levels of distribution shift.
• Employs multiple backbone networks—PointNet++, DGCNN, and others—to analyze the impact of architecture choice on Open Set performance.
• Applies a range of OOD detection techniques, including MSP, ODIN, Energy scores, GradNorm, ReAct, and distance-based methods like Cosine proto and CE(L²).
• Introduces SubArcFace and mixup-based data augmentation to improve generalization under domain shift.
• Uses AUROC and FPR95 as primary metrics to evaluate detection performance across all benchmarks.
• Conducts ablation studies on backbone choice and method performance, including closed-set accuracy comparisons.

Experimental results

Research questions

• RQ1 How do state-of-the-art 2D Open Set methods perform when applied to 3D point cloud data under varying distribution shifts?
• RQ2 Which backbone architecture—PointNet++ or DGCNN—demonstrates superior robustness to domain and semantic shifts in 3D Open Set learning?
• RQ3 To what extent does the combination of semantic shift and domain shift degrade the performance of existing Open Set methods in 3D?
• RQ4 Can simple representation learning methods like MSP or Cosine proto outperform complex, state-of-the-art 2D-based OOD detection techniques in 3D?
• RQ5 What is the relative difficulty of the three benchmark tracks—Synthetic, Real-to-Real, and Synthetic-to-Real—in terms of model generalization?

Key findings

• PointNet++ consistently outperforms DGCNN across all benchmarks, demonstrating superior robustness to noise and domain shift in real-world data.
• The MSP baseline with PointNet++ achieves 8.9 pp higher AUROC and 7.4 pp higher FPR95 than its DGCNN counterpart in the Synthetic-to-Real benchmark.
• Cosine proto achieves the best overall performance on both DGCNN and PointNet++ in the Real-to-Real benchmark, outperforming more complex methods.
• SubArcFace achieves the highest average AUROC across all backbones and settings, showing strong generalization under domain shift.
• The improvement of advanced methods over MSP is most pronounced in the Synthetic benchmark (+3.1 AUROC), but diminishes significantly in cross-domain settings (only +0.6 AUROC in Real-to-Real), indicating limited gains in real-world deployment.
• Distance-based methods like CE(L²) and Cosine proto are less sensitive to point density variations than baseline methods, suggesting inherent robustness to sparsity.