[Paper Review] Recognizing Scenes from Novel Viewpoints

Shengyi Qian, Alexander Kirillov+5 more|arXiv (Cornell University)|Dec 2, 2021

Advanced Vision and Imaging65 references4 citations

TL;DR

This paper proposes ViewSeg, an end-to-end model that learns a 3D scene representation from a few RGB images and camera poses, enabling zero-shot semantic segmentation and depth prediction from novel viewpoints without access to the novel view's RGB image. The method achieves state-of-the-art performance on Hypersim and Replica datasets by jointly optimizing for semantics and geometry using only 2D multi-view annotations and camera poses, demonstrating strong generalization to complex, unseen indoor scenes.

ABSTRACT

Humans can perceive scenes in 3D from a handful of 2D views. For AI agents, the ability to recognize a scene from any viewpoint given only a few images enables them to efficiently interact with the scene and its objects. In this work, we attempt to endow machines with this ability. We propose a model which takes as input a few RGB images of a new scene and recognizes the scene from novel viewpoints by segmenting it into semantic categories. All this without access to the RGB images from those views. We pair 2D scene recognition with an implicit 3D representation and learn from multi-view 2D annotations of hundreds of scenes without any 3D supervision beyond camera poses. We experiment on challenging datasets and demonstrate our model's ability to jointly capture semantics and geometry of novel scenes with diverse layouts, object types and shapes.

Motivation & Objective

• To enable AI agents to recognize and understand scenes from novel viewpoints using only a few 2D images and camera poses.
• To jointly learn semantic understanding and 3D geometry of scenes without explicit 3D annotations beyond camera poses.
• To bridge the gap between 2D scene understanding, 3D scene reconstruction, and novel view synthesis for complex indoor environments.
• To develop a model that generalizes to diverse layouts, object types, and shapes in real-world scenes.

Proposed method

• ViewSeg uses a multi-view 2D supervision setup with RGB images, semantic segmentations, and camera poses as input.
• It employs a 3D implicit neural representation (e.g., based on NeRF-style volume rendering) to implicitly encode scene geometry and semantics.
• The model predicts semantic segmentation and depth for a novel target view by querying the 3D representation using the target camera's pose.
• It is trained end-to-end with a multi-task loss combining semantic segmentation and depth prediction, using only 2D annotations and camera poses.
• The architecture integrates a 2D backbone (e.g., DeepLabV3+) for feature extraction and a 3D rendering head for differentiable view generation.
• Fine-tuning on target datasets like Replica improves performance, showing strong zero-shot generalization from pre-training on Hypersim.

Experimental results

Research questions

• RQ1 Can a model learn to perform semantic segmentation from novel viewpoints using only 2D images and camera poses, without any 3D supervision?
• RQ2 Does joint optimization for semantics and geometry improve both 3D understanding and segmentation accuracy compared to separate or sequential approaches?
• RQ3 How well does the model generalize to complex, real-world indoor scenes with diverse layouts and object types?
• RQ4 To what extent does semantic supervision improve depth estimation in the absence of ground-truth depth during training?

Key findings

• ViewSeg achieves a mean Intersection over Union (mIoU) of 30.2 on the Replica test set after fine-tuning, significantly outperforming baseline methods.
• On the Hypersim dataset, ViewSeg achieves an mIoU of 30.2 and a depth Rel (relative error) of 0.130, demonstrating strong performance on challenging, diverse scenes.
• The model improves depth prediction accuracy when trained with semantic supervision, indicating that semantics enhance geometric understanding.
• Fine-tuning on Replica improves mIoU from 13.2 (no fine-tuning) to 30.2, showing strong adaptation to new scene statistics.
• The oracle model (supervised on Replica) achieves mIoU of 56.2, indicating that ViewSeg’s zero-shot performance is competitive despite no 3D supervision.
• Ablation studies confirm that semantic supervision enhances depth estimation, with a 40% reduction in depth error (Rel from 0.222 to 0.130) when semantic loss is included.