[Paper Review] ProbNeRF: Uncertainty-Aware Inference of 3D Shapes from 2D Images

Matthew D. Hoffman, Tuan Anh Lê+5 more|arXiv (Cornell University)|Oct 27, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

ProbNeRF proposes a probabilistic neural radiance field model that enables uncertainty-aware 3D shape and appearance reconstruction from single or few 2D images. By combining a variational autoencoder with Hamiltonian Monte Carlo inference over full NeRF weights—using an exact renderer, annealed HMC, and hypernetwork-based decoding—ProbNeRF generates diverse, coherent, and high-fidelity 3D hypotheses consistent with input views.

ABSTRACT

The problem of inferring object shape from a single 2D image is underconstrained. Prior knowledge about what objects are plausible can help, but even given such prior knowledge there may still be uncertainty about the shapes of occluded parts of objects. Recently, conditional neural radiance field (NeRF) models have been developed that can learn to infer good point estimates of 3D models from single 2D images. The problem of inferring uncertainty estimates for these models has received less attention. In this work, we propose probabilistic NeRF (ProbNeRF), a model and inference strategy for learning probabilistic generative models of 3D objects' shapes and appearances, and for doing posterior inference to recover those properties from 2D images. ProbNeRF is trained as a variational autoencoder, but at test time we use Hamiltonian Monte Carlo (HMC) for inference. Given one or a few 2D images of an object (which may be partially occluded), ProbNeRF is able not only to accurately model the parts it sees, but also to propose realistic and diverse hypotheses about the parts it does not see. We show that key to the success of ProbNeRF are (i) a deterministic rendering scheme, (ii) an annealed-HMC strategy, (iii) a hypernetwork-based decoder architecture, and (iv) doing inference over a full set of NeRF weights, rather than just a low-dimensional code.

Motivation & Objective

• Address the challenge of uncertainty in 3D shape reconstruction from single or few 2D images, where traditional NeRFs produce only point estimates.
• Overcome the limitations of existing conditional NeRFs that fail to represent shape and appearance uncertainty, especially for occluded or unseen parts.
• Develop a generative model that learns plausible 3D object priors and enables posterior inference over full NeRF representations, not just latent codes.
• Enable robust, diverse, and high-fidelity 3D reconstructions by combining exact rendering, annealed HMC, and weight-level inference over NeRF parameters.

Proposed method

• Train ProbNeRF as a variational autoencoder (VAE) with an invertible RealNVP flow to model a prior over latent codes, followed by a hypernetwork to generate full NeRF weights.
• Use an exact, deterministic renderer (based on Chen et al., 2022) instead of Monte Carlo rendering to ensure compatibility with HMC and reduce variance in gradient estimates.
• Perform test-time inference via Hamiltonian Monte Carlo (HMC) over the full set of NeRF weights, enabling exploration of diverse, plausible 3D shape hypotheses.
• Apply a temperature-annealing strategy in HMC to improve mixing and convergence in the presence of non-log-concave likelihoods and isolated posterior modes.
• Employ a two-stage hypernetwork-based decoder to generate compact, high-fidelity NeRFs, reducing per-pixel rendering cost and enabling efficient iterative inference.
• Treat both the latent code and the raw NeRF weights as random variables in the posterior, avoiding the bottleneck of low-dimensional codes and enabling high-fidelity reconstruction.

Experimental results

Research questions

• RQ1 Can we model shape and appearance uncertainty in 3D reconstruction from single 2D images using a probabilistic NeRF framework?
• RQ2 How can we achieve high-fidelity 3D reconstruction while maintaining diversity in generated hypotheses for unseen object parts?
• RQ3 What inference strategy enables robust posterior sampling over full NeRF weight spaces, especially when likelihoods are non-log-concave?
• RQ4 How do exact rendering and annealed HMC compare to standard Monte Carlo rendering and HMC in terms of sampling quality and convergence?
• RQ5 Does inferring over full NeRF weights, rather than just latent codes, lead to better reconstruction fidelity and diversity?

Key findings

• ProbNeRF produces diverse, coherent, and high-fidelity 3D shape hypotheses from a single 2D image, with samples that are consistent with the input view and vary meaningfully in unseen parts.
• The use of an exact renderer enables stable and efficient HMC sampling, whereas standard Monte Carlo rendering leads to high rejection rates and poor mixing.
• Annealed HMC significantly improves convergence and exploration of isolated posterior modes, especially in non-log-concave likelihood landscapes.
• Inferring over full NeRF weights rather than just latent codes eliminates the bottleneck of low-dimensional representations, enabling higher reconstruction fidelity.
• The two-stage hypernetwork-based decoder reduces per-pixel rendering cost and improves efficiency, making iterative test-time inference feasible.
• HMC-generated samples show high per-pixel variance in occluded regions, indicating meaningful uncertainty modeling, while amortized inference (e.g., VAE) produces nearly identical samples with no diversity.