[Paper Review] DeepView: View Synthesis with Learned Gradient Descent

John P. Flynn, Michael Broxton+6 more|arXiv (Cornell University)|Jun 18, 2019

Advanced Vision and Imaging34 references21 citations

TL;DR

DeepView proposes a learned gradient descent approach to reconstruct high-quality multiplane images (MPIs) from sparse input views, enabling state-of-the-art view synthesis with improved occlusion reasoning. By modeling visibility in gradients, the method achieves superior results on complex scenes—especially at edges, reflections, and high-depth-complexity regions—outperforming prior methods on the Kalantari light field dataset and a new public dataset, Spaces.

ABSTRACT

We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of sparse camera viewpoints. The resulting method incorporates occlusion reasoning, improving performance on challenging scene features such as object boundaries, lighting reflections, thin structures, and scenes with high depth complexity. We show that our method achieves high-quality, state-of-the-art results on two datasets: the Kalantari light field dataset, and a new camera array dataset, Spaces, which we make publicly available.

Motivation & Objective

• Address the ill-posed inverse problem of reconstructing multiplane images (MPIs) from sparse camera viewpoints.
• Improve view synthesis quality in challenging scenarios such as object boundaries, reflections, thin structures, and high depth complexity.
• Overcome limitations of standard gradient descent in MPI optimization by learning a data-driven update rule that incorporates scene priors implicitly.
• Introduce a new, large-scale real-world dataset, Spaces, for training and evaluating view synthesis methods.
• Demonstrate that learned gradient descent can produce high-quality, real-time renderable MPIs with minimal iterations.

Proposed method

• Use a deep neural network to learn the gradient update rule in an iterative optimization process, replacing standard gradient descent with a learned update function.
• Process input images through a CNN to generate an initial MPI, then refine it via a few iterations of learned gradient descent.
• Model visibility between input views and MPI layers explicitly in the gradient computation, enabling accurate occlusion reasoning.
• Leverage the intuitive form of gradients as visibility-aware error signals, allowing the network to learn representations that stay on the manifold of natural scenes.
• Use max-pooling across input views to enable flexibility in the number of input cameras, suggesting potential for future generalization beyond fixed view counts.
• Train the model end-to-end using a differentiable renderer to optimize MPI parameters against input view measurements.

Experimental results

Research questions

• RQ1 Can learned gradient descent improve MPI reconstruction quality in view synthesis, especially for challenging scene features?
• RQ2 How does visibility-aware gradient computation enhance occlusion reasoning in MPI optimization compared to standard gradient descent?
• RQ3 To what extent can a learned update rule implicitly encode scene priors, reducing overfitting and visual artifacts?
• RQ4 Can the method generalize to diverse real-world scenes with high depth complexity, reflections, and thin structures?
• RQ5 How does the performance of DeepView compare to existing state-of-the-art methods on both standard and newly introduced benchmark datasets?

Key findings

• DeepView achieves state-of-the-art view synthesis performance on the Kalantari light field dataset and the new public Spaces dataset.
• The method produces high-quality results with sharp edges, accurate reflections, and correct handling of thin structures and complex foliage.
• Compared to Soft3D, DeepView reduces softness and edge artifacts, particularly in regions of high depth complexity.
• The model reconstructs diffuse reflections by using transparent alpha values in the MPI to allow visibility through to distant planes.
• The depth visualizations derived from the MPI show high accuracy around object boundaries, though less so in smooth regions due to lack of explicit depth supervision.
• The method runs in approximately 50 seconds on a P100 GPU for 4 input views, enabling real-time rendering of novel viewpoints from the optimized MPI.