[Paper Review] Towards 3D Scene Understanding by Referring Synthetic Models

Runnan Chen, Xinge Zhu+6 more|arXiv (Cornell University)|Mar 20, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes Referring Transfer Learning (RTL), a method that enables 3D scene understanding by transferring knowledge from synthetic 3D models to real-world scans without real-scene annotations. It introduces physical data alignment and convex-hull regularized feature alignment to bridge the model-to-scene and synthetic-to-real domain gaps, achieving 46.08% and 55.49% mAP on ScanNet and S3DIS using only synthetic models from ModelNet.

ABSTRACT

Promising performance has been achieved for visual perception on the point cloud. However, the current methods typically rely on labour-extensive annotations on the scene scans. In this paper, we explore how synthetic models alleviate the real scene annotation burden, i.e., taking the labelled 3D synthetic models as reference for supervision, the neural network aims to recognize specific categories of objects on a real scene scan (without scene annotation for supervision). The problem studies how to transfer knowledge from synthetic 3D models to real 3D scenes and is named Referring Transfer Learning (RTL). The main challenge is solving the model-to-scene (from a single model to the scene) and synthetic-to-real (from synthetic model to real scene's object) gap between the synthetic model and the real scene. To this end, we propose a simple yet effective framework to perform two alignment operations. First, physical data alignment aims to make the synthetic models cover the diversity of the scene's objects with data processing techniques. Then a novel extbf{convex-hull regularized feature alignment} introduces learnable prototypes to project the point features of both synthetic models and real scenes to a unified feature space, which alleviates the domain gap. These operations ease the model-to-scene and synthetic-to-real difficulty for a network to recognize the target objects on a real unseen scene. Experiments show that our method achieves the average mAP of 46.08\% and 55.49\% on the ScanNet and S3DIS datasets by learning the synthetic models from the ModelNet dataset. Code will be publicly available.

Motivation & Objective

• To reduce the annotation burden in 3D scene understanding by leveraging synthetic 3D models instead of real-scene annotations.
• To address the model-to-scene generalization gap, where a network learns from single models but infers on complex, crowded scenes.
• To mitigate the synthetic-to-real domain gap caused by geometric and distributional differences between synthetic models and real scans.
• To enable zero-shot recognition of unseen object categories by transferring knowledge from synthetic models to real scenes.
• To develop a simple yet effective framework that supports efficient knowledge transfer for downstream 3D perception tasks.

Proposed method

• Physical data alignment applies data augmentation techniques—rotation, scaling, cropping, and mix-up with scene points—to make synthetic models more representative of real-world object variations.
• A novel convex-hull regularized feature alignment module projects point features into a unified feature space using learnable prototypes to form a convex hull, reducing domain shift.
• The learnable prototypes act as base structural elements that define a compact, closure feature space, ensuring feature distributions are constrained and comparable across domains.
• Contrastive learning is used with synthetic models as positive pairs and other synthetic models as negative samples to improve feature generalization.
• The framework employs a Minkowski U-Net backbone with a temperature-controlled attention mechanism to refine feature alignment and improve robustness.
• The method is designed as a pretext pretraining task, enabling performance gains on downstream 3D segmentation tasks.

Experimental results

Research questions

• RQ1 Can a neural network recognize specific object categories in a real, unseen 3D scene using only labeled synthetic 3D models and no real-scene annotations?
• RQ2 How can the model-to-scene gap—where a model trained on isolated objects generalizes to complex, cluttered scenes—be effectively mitigated?
• RQ3 To what extent can feature alignment across synthetic and real domains reduce the synthetic-to-real domain shift in 3D semantic segmentation?
• RQ4 What data augmentation strategies best simulate real-world object variations in synthetic models to improve generalization?
• RQ5 Can the proposed method generalize to unseen object categories not present in the synthetic training set?

Key findings

• The proposed RTL framework achieves 46.08% mAP on the ScanNet dataset and 55.49% mAP on the S3DIS dataset using only synthetic models from ModelNet, with no real-scene annotations.
• Physical data alignment improves mAP by 31.26% compared to baseline, with random scaling and rotation being critical for performance.
• Excluding data augmentation (noDA) reduces mAP to 23.44%, highlighting the importance of realistic data augmentation for domain generalization.
• Using scene points as negative samples for contrastive learning degrades performance to 23.04% mAP, indicating that such artifacts hinder generalization.
• The convex-hull regularized feature alignment module improves mAP by approximately 4% when added to the physical data alignment baseline.
• The optimal number of prototypes is 128, and setting the temperature hyperparameter λ to 0.5 yields the best performance, with ablation showing sensitivity to these hyperparameters.