[Paper Review] ProSGNeRF: Progressive Dynamic Neural Scene Graph with Frequency Modulated Foundation Model in Urban Scenes

Tianchen Deng, Wang, Yanbo+6 more|arXiv (Cornell University)|Dec 14, 2023

Advanced Vision and Imaging7 citations

TL;DR

ProSGNeRF proposes a progressive dynamic neural scene graph with a frequency-modulated autoencoder to enable high-fidelity novel view synthesis in large-scale urban scenes with fast-moving vehicles and sparse views. By dynamically allocating local scene graphs over temporal windows and using frequency-aware latent encoding, it achieves state-of-the-art performance in view synthesis, object manipulation, and scene roaming on KITTI and VKITTI datasets.

ABSTRACT

Implicit neural representation has demonstrated promising results in 3D reconstruction on various scenes. However, existing approaches either struggle to model fast-moving objects or are incapable of handling large-scale camera ego-motions in urban environments. This leads to low-quality synthesized views of the large-scale urban scenes. In this paper, we aim to jointly solve the problems caused by large-scale scenes and fast-moving vehicles, which are more practical and challenging. To this end, we propose a progressive scene graph network architecture to learn the local scene representations of dynamic objects and global urban scenes. The progressive learning architecture dynamically allocates a new local scene graph trained on frames within a temporal window, with the window size automatically determined, allowing us to scale up the representation to arbitrarily large scenes. Besides, according to our observations, the training views of dynamic objects are relatively sparse according to rapid movements, which leads to a significant decline in reconstruction accuracy for dynamic objects. Therefore, we utilize a foundation model network to encode the latent code. Specifically, we leverage the generalization capability of the visual foundation model DINOv2 to extract appearance and shape codes, and train the network on a large-scale urban scene object dataset to enhance its prior modeling ability for handling sparse-view dynamic inputs. In parallel, we introduce a frequency-modulated module that regularizes the frequency spectrum of objects, thereby addressing the challenge of modeling sparse image inputs from a frequency-domain perspective. Experimental results demonstrate that our method achieves state-of-the-art view synthesis accuracy, object manipulation, and scene roaming ability in various scenes.

Motivation & Objective

• To address the challenge of view synthesis in large-scale urban scenes with multiple fast-moving objects and significant camera ego-motion.
• To improve reconstruction accuracy for dynamic objects under sparse view observations common in real-world urban datasets.
• To reduce dependency on human-annotated masks by leveraging promptable segmentation (e.g., SAM) for automatic object and far-field mask generation.
• To maintain geometric consistency in large scenes using LiDAR point cloud projection supervision.
• To enable flexible scene editing, including object manipulation and novel scene graph construction, without requiring observed training data for such configurations.

Proposed method

• The method employs a neural scene graph that decomposes the scene into background, dynamic objects, and far-field components, each with dedicated neural networks for representation.
• A progressive scheme dynamically instantiates and maintains local scene graphs over a temporal window, enabling scalability to arbitrarily large scenes.
• A frequency auto-encoder network encodes shape and appearance features of dynamic objects and performs frequency modulation to regularize latent representations under sparse view inputs.
• The system uses Segment Anything Model (SAM) for zero-shot, prompt-based instance segmentation to generate accurate masks for dynamic objects and far-field regions, reducing annotation burden.
• LiDAR point cloud projections are used to supervise geometry consistency, ensuring accurate spatial alignment in large-scale urban scenes.
• A multi-component loss function includes depth loss and KL divergence loss to improve geometric accuracy and latent space regularization.

Experimental results

Research questions

• RQ1 Can a progressive neural scene graph architecture effectively scale to large-scale urban scenes with complex camera ego-motions and multiple dynamic objects?
• RQ2 How can implicit neural representations be enhanced under sparse view observations for dynamic urban objects?
• RQ3 To what extent can prompt-based segmentation models like SAM reduce the need for manual annotations in dynamic scene reconstruction?
• RQ4 Does frequency modulation in the latent space improve the representation of dynamic objects with limited training views?
• RQ5 Can the proposed method support flexible scene editing, such as novel object pose manipulation and scene graph construction, beyond training data?

Key findings

• The full ProSGNeRF model achieves a PSNR of 30.31, SSIM of 0.891, and LPIPS of 0.055 on the KITTI tracking 0006 sequence, outperforming prior methods.
• Removing the progressive scene graph leads to a significant drop in PSNR (27.21), demonstrating its critical role in handling large-scale ego-motion.
• Replacing the frequency auto-encoder with a standard MLP reduces PSNR to 28.32, confirming its effectiveness in handling sparse views of dynamic objects.
• Omitting the depth loss or KL divergence loss results in degraded performance (PSNR 28.93 and 29.45, respectively), showing their importance for geometric accuracy.
• The method enables novel view synthesis of unseen object arrangements and rotations not present during training, demonstrating strong generalization and editing capability.
• On the VKITTI dataset, ProSGNeRF achieves state-of-the-art image reconstruction accuracy, with PSNR of 0.372 and LPIPS of 0.088, surpassing concurrent methods.