[Paper Review] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Xiao Fu, Shangzhan Zhang+6 more|arXiv (Cornell University)|Mar 29, 2022

Remote Sensing and LiDAR Applications4 citations

TL;DR

Panoptic NeRF proposes a 3D-to-2D label transfer method that leverages NeRF to jointly optimize geometry and semantics, using coarse 3D bounding primitives and noisy 2D semantic predictions to generate high-quality, multi-view consistent panoptic segmentation labels. It achieves state-of-the-art performance on KITTI-360, improving both accuracy and consistency by fusing 3D and 2D supervisory signals through dual semantic fields and semantically guided geometry optimization.

ABSTRACT

Large-scale training data with high-quality annotations is critical for training semantic and instance segmentation models. Unfortunately, pixel-wise annotation is labor-intensive and costly, raising the demand for more efficient labeling strategies. In this work, we present a novel 3D-to-2D label transfer method, Panoptic NeRF, which aims for obtaining per-pixel 2D semantic and instance labels from easy-to-obtain coarse 3D bounding primitives. Our method utilizes NeRF as a differentiable tool to unify coarse 3D annotations and 2D semantic cues transferred from existing datasets. We demonstrate that this combination allows for improved geometry guided by semantic information, enabling rendering of accurate semantic maps across multiple views. Furthermore, this fusion process resolves label ambiguity of the coarse 3D annotations and filters noise in the 2D predictions. By inferring in 3D space and rendering to 2D labels, our 2D semantic and instance labels are multi-view consistent by design. Experimental results show that Panoptic NeRF outperforms existing label transfer methods in terms of accuracy and multi-view consistency on challenging urban scenes of the KITTI-360 dataset.

Motivation & Objective

• To address the high cost and labor intensity of pixel-wise semantic and instance segmentation annotation in urban scenes.
• To improve label transfer accuracy from coarse 3D bounding primitives to dense 2D panoptic labels using weak supervision from noisy 2D predictions.
• To ensure multi-view and spatio-temporal consistency in the generated 2D labels by performing inference in 3D space.
• To resolve label ambiguity at overlapping 3D bounding primitive regions and reduce noise in 2D predictions through joint optimization.
• To enable end-to-end, fully automated label transfer without human post-processing, leveraging differentiable rendering and semantic guidance.

Proposed method

• The method uses a NeRF-based radiance field with dual semantic fields: one fixed from 3D bounding primitives and one learned from 2D predictions.
• It performs semantically guided geometry optimization by fixing the semantic field and refining geometry using 2D semantic cues.
• A joint geometry and semantic optimization step fuses 3D primitive labels and 2D noisy predictions via a learned semantic field to resolve ambiguities.
• The model is trained using a combination of 3D supervision ($\mathcal{L}^{\text{3D}}_{\mathbf{s}}$), 2D supervision ($\mathcal{L}^{\text{2D}}_{\mathbf{S}}$), and photometric loss ($\mathcal{L}_p$), with ray masking to improve sampling.
• Volume rendering is used to project 3D-learned semantic fields into 2D images, producing consistent panoptic labels across views.
• The method employs a novel sampling strategy that improves geometry reconstruction under sparse input views.

Experimental results

Research questions

• RQ1 Can 3D-to-2D label transfer be improved by jointly optimizing geometry and semantics in 3D space rather than in 2D?
• RQ2 Can semantically guided geometry optimization enhance reconstruction quality when only sparse views are available?
• RQ3 Can the fusion of coarse 3D bounding primitives and noisy 2D predictions resolve label ambiguity and reduce noise in 2D predictions?
• RQ4 Does 3D inference lead to inherently multi-view consistent 2D panoptic labels?
• RQ5 How does the proposed method compare to existing 2D-to-2D and 3D-to-2D label transfer baselines in accuracy and consistency?

Key findings

• Panoptic NeRF achieves a mIoU of 81.4 and accuracy of 94.5 on the KITTI-360 dataset, outperforming existing 3D-to-2D and 2D-to-2D label transfer methods.
• The ablation study shows that removing the 2D supervision loss ($\mathcal{L}^{\text{2D}}_{\mathbf{S}}$) reduces mIoU to 70.7, demonstrating its critical role in refining the learned semantic field.
• The fixed semantic field ($s_{\beta}$) significantly improves geometry, as removing it increases depth error and blurs object boundaries.
• The complete model with all components achieves the lowest photometric loss (5.23) and highest mIoU (81.4), confirming the effectiveness of the full optimization pipeline.
• Using pre-trained 2D models like Deeplab and PSPNet as pseudo GTs, Panoptic NeRF still outperforms them, with mIoU of 79.3 and 75.9 respectively, showing robustness to noisy 2D inputs.
• The method achieves globally consistent instance labeling across views, with each object assigned a unique instance ID via 3D bounding primitives.