[Paper Review] GNeSF: Generalizable Neural Semantic Fields

Hanlin Chen, Li Chen+3 more|arXiv (Cornell University)|Oct 24, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes GNeSF, a generalizable neural semantic field that enables 3D semantic segmentation for novel scenes using only 2D supervision. By leveraging multi-view image features, a soft voting mechanism with view difference encoding, and a visibility module, GNeSF avoids per-scene optimization and achieves state-of-the-art performance with 2D annotations alone, outperforming methods requiring 3D supervision.

ABSTRACT

3D scene segmentation based on neural implicit representation has emerged recently with the advantage of training only on 2D supervision. However, existing approaches still requires expensive per-scene optimization that prohibits generalization to novel scenes during inference. To circumvent this problem, we introduce a generalizable 3D segmentation framework based on implicit representation. Specifically, our framework takes in multi-view image features and semantic maps as the inputs instead of only spatial information to avoid overfitting to scene-specific geometric and semantic information. We propose a novel soft voting mechanism to aggregate the 2D semantic information from different views for each 3D point. In addition to the image features, view difference information is also encoded in our framework to predict the voting scores. Intuitively, this allows the semantic information from nearby views to contribute more compared to distant ones. Furthermore, a visibility module is also designed to detect and filter out detrimental information from occluded views. Due to the generalizability of our proposed method, we can synthesize semantic maps or conduct 3D semantic segmentation for novel scenes with solely 2D semantic supervision. Experimental results show that our approach achieves comparable performance with scene-specific approaches. More importantly, our approach can even outperform existing strong supervision-based approaches with only 2D annotations. Our source code is available at: https://github.com/HLinChen/GNeSF.

Motivation & Objective

• To address the limitation of existing neural semantic fields that require per-scene optimization, restricting generalization to novel scenes.
• To enable high-quality 3D semantic segmentation for unseen scenes without retraining or 3D supervision.
• To improve generalization by avoiding overfitting to scene-specific geometric and semantic details.
• To develop a framework that leverages 2D semantic maps and multi-view features for robust 3D semantic prediction.

Proposed method

• The framework uses multi-view image features instead of positional encoding to prevent overfitting to scene-specific geometry and semantics.
• A soft voting mechanism aggregates 2D semantic probabilities from multiple views to predict 3D point semantics, weighted by view relevance.
• View difference encoding is introduced to prioritize nearby views over distant ones in the voting process.
• A visibility module detects and filters out semantic information from occluded or unreliable views.
• The 3D geometry (density or SDF) is predicted via a 3D MLP from interpolated features in a geometry encoding volume.
• The final semantic prediction is computed as a weighted combination of projected 2D semantics, with weights derived from predicted geometry.

Experimental results

Research questions

• RQ1 Can a neural semantic field be trained to generalize across novel scenes without per-scene optimization?
• RQ2 How can 2D semantic supervision be effectively leveraged to predict 3D semantic fields without 3D annotations?
• RQ3 What role does view proximity play in aggregating multi-view semantic information for 3D prediction?
• RQ4 Can a visibility module improve robustness by filtering occluded view contributions?
• RQ5 How does using image features instead of positional encoding affect generalization and performance?

Key findings

• GNeSF achieves a 3D mIoU of 43.8% on the Replica dataset, outperforming all prior methods using only 2D supervision.
• On ScanNet, GNeSF achieves 55.1% mIoU, surpassing the previous best method with 3D supervision (51.5%).
• The ablation study shows that adding the visibility module improves mIoU by 0.9% over the baseline, confirming its effectiveness.
• Using probability distributions instead of raw logits improves mIoU by 0.6% compared to logit-based voting.
• GNeSF outperforms S4R (36.9% mIoU) by a significant margin, demonstrating the superiority of its soft voting mechanism.