[Paper Review] NPBG++: Accelerating Neural Point-Based Graphics

Ruslan Rakhimov, Andrei-Timotei Ardelean+2 more|arXiv (Cornell University)|Mar 24, 2022

Advanced Vision and Imaging4 citations

TL;DR

NPBG++ accelerates novel view synthesis by predicting view-dependent neural descriptors for 3D point clouds in a single forward pass through input images, eliminating per-scene optimization. It achieves real-time rendering with high image quality, outperforming prior methods in fitting and rendering speed while maintaining competitive perceptual quality (LPIPS ↓ 0.154 on DTU scans).

ABSTRACT

We present a new system (NPBG++) for the novel view synthesis (NVS) task that achieves high rendering realism with low scene fitting time. Our method efficiently leverages the multiview observations and the point cloud of a static scene to predict a neural descriptor for each point, improving upon the pipeline of Neural Point-Based Graphics in several important ways. By predicting the descriptors with a single pass through the source images, we lift the requirement of per-scene optimization while also making the neural descriptors view-dependent and more suitable for scenes with strong non-Lambertian effects. In our comparisons, the proposed system outperforms previous NVS approaches in terms of fitting and rendering runtimes while producing images of similar quality.

Motivation & Objective

• Address the long fitting time of existing neural point-based graphics methods that require per-scene optimization.
• Enable real-time rendering of high-quality novel views from sparse input images and point clouds.
• Improve rendering quality for non-Lambertian surfaces by introducing view-dependent neural descriptors.
• Reduce sensitivity to geometric imperfections such as sparse point clouds, noise, and limited input views.
• Ensure rotational equivariance in the rendering pipeline through lightweight 2D alignment stages.

Proposed method

• Predict neural descriptors for each 3D point using a single forward pass through all source images, bypassing per-scene optimization.
• Employ an online, permutation-invariant aggregation method that combines features from multiple views using constant memory via online linear regression.
• Model view-dependent appearance by learning basis function coefficients (e.g., MLP with m=6) for each point’s descriptor.
• Introduce two lightweight 2D alignment stages: one for input images and one for output renderings, to correct for in-plane rotation misalignments.
• Use a U-Net-based renderer that performs multi-scale rasterization of the point cloud with learned descriptors.
• Apply spherical harmonics and MLP-based basis functions for descriptor representation, with ablation showing MLP (m=6) yields best results.

Experimental results

Research questions

• RQ1 Can we eliminate per-scene optimization in neural point-based graphics while maintaining high rendering quality?
• RQ2 How can we efficiently model view-dependent appearance in point-based neural rendering without increasing fitting time?
• RQ3 To what extent does input and output image alignment improve performance in neural point-based rendering pipelines?
• RQ4 How robust is the system to geometric imperfections such as sparse point clouds, noise, and limited input views?
• RQ5 Can lightweight 2D alignment stages effectively restore rotational equivariance in descriptor-based rendering?

Key findings

• NPBG++ achieves a time-to-render (fitting + rendering) of 1.2 seconds per image on the 'hotdog' scene, significantly faster than NPBG and IBRNet.
• With both input and output image alignment, NPBG++ achieves LPIPS ↓ 0.154 on DTU scans, outperforming NPBG (LPIPS ↓ 0.125) when aligned.
• The MLP-based aggregation with m=6 basis functions yields the best results: PSNR ↑ 23.11, SSIM ↑ 0.766, LPIPS ↓ 0.501 on ScanNet scenes.
• The system shows graceful degradation under geometric imperfections: it maintains >80% filled pixels even with 50% point cloud sparsity and performs robustly under noise and few-view scenarios.
• Ablation shows that using only input or output alignment degrades performance, confirming the necessity of both for distribution alignment.
• The proposed online aggregation method enables efficient, constant-memory feature fusion from any number of views, supporting scalable and view-dependent descriptor learning.