[论文解读] Recent Trends in 3D Reconstruction of General Non-Rigid Scenes

Raza Yunus, Jan Eric Lenssen+8 more|arXiv (Cornell University)|Mar 22, 2024

3D Surveying and Cultural HeritageEarth and Planetary Sciences被引用 3

一句话总结

本文综述了使用神经隐式表示在单目和多视角输入下对一般非刚性场景进行3D重建的最新进展。重点介绍了在形变建模、场景分解、编辑和生成先验方面的进展，关键贡献包括可微分渲染、基于物理的约束以及用于4D场景合成的扩散模型。

ABSTRACT

Reconstructing models of the real world, including 3D geometry, appearance, and motion of real scenes, is essential for computer graphics and computer vision. It enables the synthesizing of photorealistic novel views, useful for the movie industry and AR/VR applications. It also facilitates the content creation necessary in computer games and AR/VR by avoiding laborious manual design processes. Further, such models are fundamental for intelligent computing systems that need to interpret real-world scenes and actions to act and interact safely with the human world. Notably, the world surrounding us is dynamic, and reconstructing models of dynamic, non-rigidly moving scenes is a severely underconstrained and challenging problem. This state-of-the-art report (STAR) offers the reader a comprehensive summary of state-of-the-art techniques with monocular and multi-view inputs such as data from RGB and RGB-D sensors, among others, conveying an understanding of different approaches, their potential applications, and promising further research directions. The report covers 3D reconstruction of general non-rigid scenes and further addresses the techniques for scene decomposition, editing and controlling, and generalizable and generative modeling. More specifically, we first review the common and fundamental concepts necessary to understand and navigate the field and then discuss the state-of-the-art techniques by reviewing recent approaches that use traditional and machine-learning-based neural representations, including a discussion on the newly enabled applications. The STAR is concluded with a discussion of the remaining limitations and open challenges.

研究动机与目标

• 提供对一般非刚性场景3D重建最新趋势的全面概述，涵盖重建、分解、编辑和生成建模。
• 分析神经隐式表示和可微分渲染在实现3D几何、外观和运动端到端优化中的作用。
• 识别在可泛化和生成建模密集动态场景方面存在的开放挑战，特别是时间与空间一致性方面。
• 探索整合归纳偏置（如基于物理的约束和预训练特征）以提升场景理解能力和可控性。
• 概述未来研究方向，包括自监督分解和实时可行的方法（如高斯点云渲染）。

提出的方法

• 使用神经隐式表示（如神经辐射场NeRF）从RGB和RGB-D输入中联合优化3D几何、外观和运动。
• 采用可微分渲染，实现从图像或视频直接对场景表示进行端到端优化。
• 通过可微分模拟器集成基于物理的先验，以强制实现真实动态，从而在约束条件下提高重建精度。
• 探索基于扩散的生成模型，并通过从2D扩散模型（如Stable Diffusion、ControlNet）蒸馏生成4D场景。
• 应用自监督学习实现场景分解，如骨架或结构发现，而无需依赖掩码或模板。
• 利用预训练特征和分割模型实现组合式建模并提升编辑能力。

Figure 1 : Capture Trajectories. A frame can consist of images from a single or multiple cameras installed on a rig. The camera or rig can move along a forward-facing, a 360-circle, or a freeform trajectory. Image source: [ WLC ∗ 23 ] .

实验结果

研究问题

• RQ1 如何有效利用神经隐式表示从单目或多视角输入中重建密集的非刚性形变场景？
• RQ2 基于物理的约束在提升非刚性3D重建的准确性和真实性方面发挥什么作用？
• RQ3 如何将基于扩散的生成模型适配以大规模生成一致的4D（3D + 时间）场景重建？
• RQ4 当前方法在无监督处理一般性、非结构化动态场景方面存在哪些局限性？
• RQ5 自监督或弱监督方法如何在无需真实标注的情况下实现非刚性场景的分解与编辑？

主要发现

• NeRF等神经隐式方法通过在端到端可微分框架中联合优化几何、外观和运动，实现了最先进的视图合成质量。
• 基于物理的约束可提高非刚性场景中的重建精度，但受限于物理模拟器中的简化假设。
• 基于扩散的生成模型在4D场景合成方面展现出前景，但目前生成低分辨率场景仍需数小时，表明存在可扩展性挑战。
• 从2D扩散模型（如Stable Diffusion）蒸馏为3D生成建模提供了一条可行路径，但空间和时间一致性仍是关键挑战。
• 自监督分解方法正成为在无需标注掩码或模板的情况下发现场景结构和骨架的新兴途径。
• 实时可行的方法（如高斯点云渲染）和基于扩散的先验正成为实现可扩展且可控的3D重建的关键未来方向。

Figure 2 : Common Scene Representations for Non-Rigid Reconstruction. Scene representations can be discrete, such as point clouds, meshes, and grids, or continuous, such as MLPs or Transformers. Both paradigms can optionally be combined, where feature embeddings for continuous neural representations