[Paper Review] Segmenting 3D Hybrid Scenes via Zero-Shot Learning.

Bo Liu, Shuang Deng+2 more|arXiv (Cornell University)|Jul 1, 2021

3D Shape Modeling and Analysis24 references4 citations

TL;DR

This paper proposes PFNet, a GAN-based zero-shot learning framework that synthesizes point features for unseen 3D object classes by leveraging semantic relationships between seen and unseen classes, enabling accurate semantic segmentation in hybrid 3D scenes. The method achieves state-of-the-art performance on two newly introduced benchmarks derived from S3DIS and ScanNet, demonstrating effectiveness in zero-shot 3D scene understanding.

ABSTRACT

This work is to tackle the problem of point cloud semantic segmentation for 3D hybrid scenes under the framework of zero-shot learning. Here by hybrid, we mean the scene consists of both seen-class and unseen-class 3D objects, a more general and realistic setting in application. To our knowledge, this problem has not been explored in the literature. To this end, we propose a network to synthesize point features for various classes of objects by leveraging the semantic features of both seen and unseen object classes, called PFNet. The proposed PFNet employs a GAN architecture to synthesize point features, where the semantic relationship between seen-class and unseen-class features is consolidated by adapting a new semantic regularizer, and the synthesized features are used to train a classifier for predicting the labels of the testing 3D scene points. Besides we also introduce two benchmarks for algorithmic evaluation by re-organizing the public S3DIS and ScanNet datasets under six different data splits. Experimental results on the two benchmarks validate our proposed method, and we hope our introduced two benchmarks and methodology could be of help for more research on this new direction.

Motivation & Objective

• Address the lack of research on 3D hybrid scene semantic segmentation, where scenes contain both seen-class and unseen-class objects.
• Enable zero-shot generalization to unseen 3D object classes in realistic, mixed-scene environments.
• Develop a method that synthesizes point features for unseen classes using semantic embeddings from seen classes.
• Introduce two new evaluation benchmarks based on S3DIS and ScanNet with six distinct data splits to standardize future research.

Proposed method

• Propose PFNet, a GAN architecture that generates synthetic point features for unseen 3D object classes using semantic embeddings from seen classes.
• Incorporate a novel semantic regularizer to model and consolidate the semantic relationships between seen-class and unseen-class features.
• Train a classifier using the synthesized features to predict semantic labels for 3D scene points during inference.
• Utilize semantic embeddings from both seen and unseen classes as conditional inputs to the generator network.
• Leverage the discriminator to ensure the realism and discriminability of synthesized point features.
• Re-organize public S3DIS and ScanNet datasets into six distinct data splits to create two new benchmarks for zero-shot 3D segmentation evaluation.

Experimental results

Research questions

• RQ1 Can a zero-shot learning framework effectively generalize to unseen 3D object classes in hybrid scenes containing both seen and unseen classes?
• RQ2 How effective is the proposed semantic regularizer in capturing cross-class semantic relationships between seen and unseen 3D objects?
• RQ3 To what extent does the synthesized point feature generation improve semantic segmentation accuracy in zero-shot 3D scene understanding?
• RQ4 How robust are the proposed benchmarks in evaluating zero-shot 3D segmentation models across different data splits?

Key findings

• The proposed PFNet achieves state-of-the-art performance on the two newly introduced benchmarks derived from S3DIS and ScanNet.
• The integration of the semantic regularizer significantly improves the quality of synthesized point features and downstream segmentation accuracy.
• The method successfully generalizes to unseen 3D object classes in hybrid scenes, demonstrating effective zero-shot generalization.
• The two new benchmarks with six data splits provide a standardized and rigorous evaluation protocol for future zero-shot 3D segmentation research.