[Paper Review] HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Keunhong Park, Utkarsh Sinha+6 more|arXiv (Cornell University)|Jun 24, 2021

Advanced Vision and Imaging45 references81 citations

TL;DR

HyperNeRF extends NeRF with a higher-dimensional ambient space and deformable slicing to model topological changes in dynamic scenes, improving interpolation between moments and novel-view synthesis.

ABSTRACT

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at https://hypernerf.github.io.

Motivation & Objective

• Address topological variations in dynamic scenes beyond continuous deformations.
• Leverage a higher-dimensional (ambient) space to represent slices of a template NeRF.
• Enable smooth interpolation between scene moments and high-quality novel-view synthesis.
• Promote efficient representation of topology-changing phenomena via deformable slicing surfaces.

Proposed method

• Represent each observation frame as a nonplanar slice through a hyper-space NeRF (HyperNeRF).
• Extend the template NeRF to a higher-dimensional domain (x, w) with W ambient dimensions.
• Use deformable slicing surfaces H(x, ωi) to map spatial points to ambient coordinates, enabling topology changes.
• Parameterize deformation T(x, ωi) and slicing H with MLPs and latent codes ωi (deformation) and ψi (appearance).
• Apply windowed positional encoding to ambient coordinates to stagger the use of ambient dimensions and stabilize optimization.
• Train end-to-end with an L2 photometric loss, using a coarse-to-fine windowed encoding schedule and standard NeRF rendering.

Experimental results

Research questions

• RQ1 Can a higher-dimensional ambient space with deformable slicing capture topology changes in dynamic NeRF scenes?
• RQ2 Does HyperNeRF produce better interpolation between observed scene moments and improve novel-view synthesis at fixed moments?
• RQ3 How does deformable slicing compare to axis-aligned slicing for representing topological variations in NeRFs?

Key findings

• HyperNeRF achieves sharper, higher-quality renderings with fewer artifacts than prior methods on both interpolation and novel-view synthesis tasks.
• Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis (LPIPS).
• Using deformable slicing surfaces (DS) yields better reconstruction quality and smoother state interpolations than axis-aligned slicing planes (AP).
• Delaying ambient dimension usage through windowed encoding improves optimization stability and avoids premature reliance on ambient coordinates.
• HyperNeRF can reconstruct photorealistic free-viewpoint renderings of topologically varying scenes from monocular video without depth sensors.