[Paper Review] Neural Volumes: Learning Dynamic Renderable Volumes from Images

Stephen Lombardi, Tomas Simon+4 more|arXiv (Cornell University)|Jun 18, 2019

Advanced Vision and Imaging59 references22 citations

TL;DR

This paper proposes Neural Volumes, a learning-based method that reconstructs dynamic, renderable 3D scenes from multi-view video using an end-to-end trained encoder-decoder network with differentiable ray marching. By learning a dynamic, irregular voxel grid via a warp field, it achieves high-resolution, artifact-free novel view synthesis without explicit mesh reconstruction or tracking, enabling realistic rendering of complex phenomena like hair and smoke.

ABSTRACT

Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occlusion, and biological motion. Mesh-based reconstruction and tracking often fail in these cases, and other approaches (e.g., light field video) typically rely on constrained viewing conditions, which limit interactivity. We circumvent these difficulties by presenting a learning-based approach to representing dynamic objects inspired by the integral projection model used in tomographic imaging. The approach is supervised directly from 2D images in a multi-view capture setting and does not require explicit reconstruction or tracking of the object. Our method has two primary components: an encoder-decoder network that transforms input images into a 3D volume representation, and a differentiable ray-marching operation that enables end-to-end training. By virtue of its 3D representation, our construction extrapolates better to novel viewpoints compared to screen-space rendering techniques. The encoder-decoder architecture learns a latent representation of a dynamic scene that enables us to produce novel content sequences not seen during training. To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications where the highest resolution is required, using facial performance capture as a case in point.

Motivation & Objective

• To address the challenge of reconstructing dynamic scenes with complex phenomena such as thin structures, translucency, and topological changes that traditional mesh-based methods fail to capture.
• To enable end-to-end training of a 3D volumetric representation directly from 2D multi-view images without explicit geometry reconstruction or tracking.
• To improve resolution and reduce grid artifacts in volumetric representations by learning a dynamic irregular grid structure using a warp field during ray marching.
• To support high-fidelity rendering of fine details, such as facial features and hair, by combining neural volumetric representations with surface-based models.
• To enable generalization to novel viewpoints and dynamic content generation through a learned latent space, even without explicit temporal modeling.

Proposed method

• The method uses an encoder-decoder neural network to map multi-view RGB images into a 3D volume representation of opacity and color at each spatial position.
• A differentiable ray-marching operation computes rendered images by integrating color and opacity along rays, enabling backpropagation through the rendering process.
• An implicit warp field is learned during training to dynamically deform a base voxel grid, increasing effective resolution and reducing grid-like artifacts.
• The framework supports joint optimization of geometry and appearance by leveraging view-dependent neural rendering, allowing compensation for reconstruction inaccuracies.
• The method can be extended to incorporate high-resolution surface models (e.g., textured meshes) for regions requiring fine detail, such as facial skin and eyes.
• Training is supervised via an image-space loss function, enabling end-to-end optimization without explicit supervision on geometry or motion.

Experimental results

Research questions

• RQ1 Can a neural network learn a dynamic, 3D volumetric representation directly from multi-view video without explicit mesh reconstruction or tracking?
• RQ2 Can differentiable ray marching enable end-to-end training of a volumetric representation that generalizes well to novel viewpoints?
• RQ3 Can a learned warp field improve the effective resolution and reduce artifacts in voxel-based volumetric representations?
• RQ4 How well can the method reconstruct challenging dynamic scenes such as moving hair, translucent objects, and smoke compared to traditional methods?
• RQ5 Can the latent space of the encoder-decoder network generate plausible dynamic content not seen during training?

Key findings

• The method successfully reconstructs complex dynamic scenes such as moving hair, fuzzy toys, and smoke with high perceptual quality and consistent novel view synthesis.
• The use of a differentiable ray-marching operation enables end-to-end training and allows gradients to propagate through the rendering process, improving convergence and reconstruction quality.
• The learned warp field significantly reduces grid-like artifacts and improves apparent resolution, outperforming standard voxel grids in visual fidelity.
• The method generalizes well to novel viewpoints, even for near-field scenes where viewpoint variation is high, due to its 3D-aware volumetric representation.
• Combining the neural volume with a textured mesh for facial regions improves detail fidelity, achieving higher-quality results than volume-only or mesh-only approaches.
• The latent space of the encoder-decoder network enables plausible generation of dynamic content sequences not in the training data, though temporal consistency depends on input dynamics.