[Paper Review] Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes

Zhengqi Li, Simon Niklaus+2 more|arXiv (Cornell University)|Nov 26, 2020

Advanced Vision and Imaging76 references32 citations

TL;DR

The paper introduces Neural Scene Flow Fields, a time-variant neural representation for monocular videos that jointly models appearance, geometry, and 3D scene motion to synthesize novel views in space and time.

ABSTRACT

We present a method to perform novel view and time synthesis of dynamic scenes, requiring only a monocular video with known camera poses as input. To do this, we introduce Neural Scene Flow Fields, a new representation that models the dynamic scene as a time-variant continuous function of appearance, geometry, and 3D scene motion. Our representation is optimized through a neural network to fit the observed input views. We show that our representation can be used for complex dynamic scenes, including thin structures, view-dependent effects, and natural degrees of motion. We conduct a number of experiments that demonstrate our approach significantly outperforms recent monocular view synthesis methods, and show qualitative results of space-time view synthesis on a variety of real-world videos.

Motivation & Objective

• Motivation: enable novel view synthesis and time interpolation from a single monocular video with known camera poses.
• Goal: model dynamic scenes as a continuous function of space and time that includes 3D motion (scene flow) and appearance/density.
• Aim: optimize a neural representation to fit input views while handling disocclusions and motion boundaries.

Proposed method

• Represent dynamic scenes as Neural Scene Flow Fields that output color, density, forward and backward 3D scene flow, and disocclusion weights.
• Use a time-extended MLP to map (x, d, i) to (c, sigma, F_i, W_i).
• Optimize with a temporal photometric consistency loss by warping neighboring time views into the target time via predicted scene flow.
• Incorporate disocclusion weights to handle motion boundaries and apply an L1 regularization and cycle-consistency on scene flow.
• Combine dynamic and static scene representations with a learned blending weight v to improve rendering in static regions.
• Enable space-time interpolation by a splatting-based plane-sweep volume rendering that blends time i and i+1 using predicted flow.
• Train per scene with COLMAP-derived camera parameters, optional data-driven priors for initialization, and multi-term losses including data priors and geometric consistency.

Experimental results

Research questions

• RQ1 Can monocular video with known camera poses be used to synthesize both novel views and novel times of dynamic scenes?
• RQ2 Does a neural, dense 3D scene flow field better capture motion boundaries and disocclusions than static or purely view-based methods?
• RQ3 Does integrating a static and a dynamic scene representation improve rendering quality in static regions while preserving dynamic fidelity?
• RQ4 Can time interpolation be achieved through predicted 3D scene flow and splatted rendering rather than simple frame blending?

Key findings

• The proposed dynamic representation outperforms state-of-the-art monocular and multi-view baselines on the Nvidia Dynamic Scenes dataset for both novel view synthesis and joint view-time synthesis.
• Adding a static scene component improves rendering quality, especially in static regions, and yields up to about 30% improvement in some settings.
• In ablations, the full model with static integration and all losses achieves the best SSIM/PSNR/LPIPS across dynamic and full-scene evaluations.
• Space-time rendering with scene-flow-based warping yields superior intermediate-time results compared to simple 2D frame interpolation or purely static methods.