[Paper Review] NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos

Yi-Ling Qiao, Alexander Gao+1 more|arXiv (Cornell University)|Oct 22, 2022

Model Reduction and Neural Networks8 citations

TL;DR

NeuPhysics proposes a method to reconstruct editable 3D geometry and physics from monocular videos using a decoupled neural field representation: a time-invariant signed distance function (SDF) for static geometry and a time-conditioned deformation field for motion. By enabling two-way conversion between neural fields and hexahedral meshes, it allows interactive editing and differentiable physics simulation, achieving state-of-the-art results in mesh and video reconstruction with end-to-end optimization via a cycle-consistency loss.

ABSTRACT

We present a method for learning 3D geometry and physics parameters of a dynamic scene from only a monocular RGB video input. To decouple the learning of underlying scene geometry from dynamic motion, we represent the scene as a time-invariant signed distance function (SDF) which serves as a reference frame, along with a time-conditioned deformation field. We further bridge this neural geometry representation with a differentiable physics simulator by designing a two-way conversion between the neural field and its corresponding hexahedral mesh, enabling us to estimate physics parameters from the source video by minimizing a cycle consistency loss. Our method also allows a user to interactively edit 3D objects from the source video by modifying the recovered hexahedral mesh, and propagating the operation back to the neural field representation. Experiments show that our method achieves superior mesh and video reconstruction of dynamic scenes compared to competing Neural Field approaches, and we provide extensive examples which demonstrate its ability to extract useful 3D representations from videos captured with consumer-grade cameras.

Motivation & Objective

• To enable end-to-end learning of 3D geometry and physics parameters from a single monocular RGB video.
• To address the under-constrained nature of monocular 3D reconstruction by decoupling static geometry from dynamic motion.
• To bridge neural implicit representations with differentiable physics simulation through a two-way mesh-field conversion pipeline.
• To support interactive editing of 3D scenes by modifying explicit hexahedral meshes that propagate changes back to the neural field.
• To eliminate reliance on manual 3D model initialization by jointly optimizing geometry and physics from video input.

Proposed method

• The scene is represented as a time-invariant signed distance function (SDF) for geometry, combined with separate neural fields for appearance, rigidity, and time-conditioned motion displacement.
• Motion is modeled via a time-conditioned deformation field that maps 3D coordinates in a frame back to their reference frame positions, enabling point-wise correspondence across time.
• A two-way conversion between implicit neural fields and explicit hexahedral meshes is established, allowing users to edit the mesh and propagate changes to the neural field.
• Physics parameters are optimized by minimizing a cycle-consistency loss between the differentiable physics simulator and the motion field, ensuring alignment between simulated and observed dynamics.
• The rigidity map is used to segment dynamic foreground objects from the background, with optional user-defined bounding boxes for improved foreground extraction.
• The framework integrates differentiable rendering (NeuS) and physics simulation (differentiable engine), enabling end-to-end training with joint optimization of geometry, appearance, and dynamics.

Experimental results

Research questions

• RQ1 Can a monocular video alone be used to recover a complete, editable 3D scene with accurate geometry and physically plausible dynamics?
• RQ2 How can neural fields be effectively decoupled into static geometry and dynamic motion to improve reconstruction stability and quality?
• RQ3 To what extent can a differentiable physics simulator be integrated with neural implicit representations without requiring manual 3D model initialization?
• RQ4 Can a two-way mesh-field conversion pipeline enable interactive editing of neural fields through explicit mesh manipulation?
• RQ5 How does cycle-consistency-based physics optimization compare to direct Chamfer distance minimization in terms of speed and accuracy?

Key findings

• The proposed cycle-consistency loss reduces total optimization time from ~115 seconds (using Chamfer distance) to ~67 seconds, while maintaining comparable reconstruction and simulation quality.
• The method achieves superior mesh and video reconstruction quality compared to competing neural field approaches, both quantitatively and qualitatively.
• The rigidity map with a threshold of 0.2 provides a robust foreground-background separation, and manual bounding boxes further improve segmentation when needed.
• Motion magnitude alone is insufficient for dynamic object detection due to small displacements in cyclic motions, making the rigidity map a more reliable criterion.
• The two-way mesh-field conversion enables direct manipulation of 3D scenes—such as adding, deleting, or deforming objects—while preserving consistent neural field representations.
• The framework successfully enables physically-based animation and simulation from raw monocular video, demonstrating its potential for digital twin and interactive 3D content creation.