[Paper Review] Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines

Ben Mildenhall, Pratul P. Srinivasan+5 more|arXiv (Cornell University)|May 2, 2019

Advanced Vision and ImagingComputer Science48 references107 citations

TL;DR

We expand each input view into a local light field via multiplane images (MPI) and blend adjacent MPIs to synthesize novel views, providing prescriptive sampling guidelines that can reach Nyquist-quality results with up to 4000× fewer input views.

ABSTRACT

We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

Motivation & Objective

• Extend plenoptic sampling theory to prescribe input view sampling density for reliable high-quality view synthesis.
• Develop a practical pipeline that expands irregular view grids into local light fields using MPIs and blends them for continuous view reconstruction.
• Show that deep learning can predict MPIs from a small neighborhood of views to achieve state-of-the-art results.
• Demonstrate the method with AR apps and real-time rendering across mobile and desktop platforms.

Proposed method

• Promote each sampled view to a local MPI with D depth planes using a 3D CNN that adapts D to the input sampling rate.
• Render novel views by alpha compositing RGB-Alpha MPI planes from back to front and blending renderings from neighboring MPIs.
• Extend plenoptic sampling to handle occlusions by predicting per-layer opacities and using alpha-aware blending (Equation 8).
• Derive a sampling bound that reduces required view density by a factor of D (and D^2 for two viewing directions) under occlusions (Equations 4–6, 7).
• Train the pipeline with synthetic (SUNCG, UnrealCV) and real handheld data, supervised by image reconstruction losses and perceptual losses, with staged training for efficiency.

Experimental results

Research questions

• RQ1 How densely must input views be captured to reliably render high-quality novel views with local light field fusion?
• RQ2 Can a deep network predict MPIs from a small, irregular set of views while maintaining consistency across multiple novel viewpoints?
• RQ3 How does the number of MPI depth planes D affect view synthesis quality and required sampling density?
• RQ4 Does alpha-guided blending of multiple MPIs improve handling of occlusions and non-Lambertian effects compared to single-MPI approaches?
• RQ5 Can the approach achieve Nyquist-level perceptual quality with substantially reduced input views in real-time AR/VR apps?

Key findings

• The method can achieve perceptual quality equivalent to Nyquist sampling while using up to ~4000× fewer input views (e.g., 64^2 views) when using 64 MPI planes.
• Promoting each view to an MPI with D planes reduces the required camera sampling interval by a factor of D (and D^2 for two-view directions) under occlusions.
• Alpha-aware blending of multiple MPI renderings yields higher fidelity and fewer artifacts than simple averaging or single-MPI rendering.
• A 3D CNN can adapt the number of depth planes to the input sampling rate, enabling flexible, scale-aware view synthesis.
• The pipeline supports real-time rendering on desktop and mobile and is validated with both synthetic and real-world handheld data.