[Paper Review] Fast Training of Neural Lumigraph Representations using Meta Learning

Alexander W. Bergman, Petr Kellnhofer+1 more|arXiv (Cornell University)|Jun 28, 2021

Advanced Vision and Imaging13 references4 citations

TL;DR

MetaNLR++ accelerates training of neural lumigraph representations by combining a coordinate-based neural shape network with CNN-based image feature aggregation and meta learning to achieve high-quality novel view synthesis in minutes rather than hours. The method enables real-time rendering by extracting implicit surface representations from optimized, meta-learned priors on shape and features.

ABSTRACT

Novel view synthesis is a long-standing problem in machine learning and computer vision. Significant progress has recently been made in developing neural scene representations and rendering techniques that synthesize photorealistic images from arbitrary views. These representations, however, are extremely slow to train and often also slow to render. Inspired by neural variants of image-based rendering, we develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. To push representation convergence times down to minutes, we leverage meta learning to learn neural shape and image feature priors which accelerate training. The optimized shape and image features can then be extracted using traditional graphics techniques and rendered in real time. We show that MetaNLR++ achieves similar or better novel view synthesis results in a fraction of the time that competing methods require.

Motivation & Objective

• To address the long training times of neural scene representations in novel view synthesis.
• To enable real-time rendering of high-quality synthesized views from sparse 2D input images.
• To reduce dependency on slow pre-processing steps like SfM or MVS by end-to-end optimization of proxy shape and features.
• To accelerate convergence using meta learning for efficient initialization of neural shape and image feature networks.
• To balance image quality, training speed, and rendering efficiency in neural rendering pipelines.

Proposed method

• A coordinate-based neural network is used to represent the proxy shape, optimized end-to-end with image feature encoders and decoders.
• 2D CNNs extract features from input images, which are aggregated on the learned surface using a differentiable re-projection mechanism.
• Meta learning is applied to learn effective initializations for the shape network, feature encoders, and aggregation function to accelerate convergence.
• The final implicit surface representation is extracted and rendered using traditional graphics techniques for real-time performance.
• A ray-mask loss with object masks supervises shape learning to ensure accurate geometry during early training phases.
• The framework avoids volume rendering by directly aggregating features on the surface, reducing computational cost.

Experimental results

Research questions

• RQ1 Can meta learning significantly reduce training time for neural lumigraph representations without sacrificing image quality?
• RQ2 Can end-to-end optimization of shape and feature networks eliminate reliance on pre-computed SfM/MVS pipelines?
• RQ3 Can a surface-based neural rendering approach achieve real-time inference while maintaining photorealistic novel view synthesis?
• RQ4 How does the integration of neural shape priors and image feature priors via meta learning affect convergence speed and final performance?
• RQ5 What trade-offs exist between shape detail and feature-based representation capacity in neural rendering?

Key findings

• MetaNLR++ achieves novel view synthesis with PSNR of 37.55 (LPIPS 0.034) on the NLR dataset, outperforming NLR++ (PSNR 35.54, LPIPS 0.046).
• Training time is reduced to under 30 minutes for high-quality results, compared to hours or days for competing methods.
• The method enables real-time rendering by extracting and rendering the implicit surface representation after training.
• Meta learning reduces convergence time by learning effective initializations for shape and feature networks.
• The approach maintains high image quality even with only six training views, demonstrating strong generalization on interpolated viewpoints.
• The method shows improved performance over baseline NLR++ in both PSNR and LPIPS metrics, indicating better perceptual quality.