[论文解读] NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

Bangbang Yang, Chong Bao+5 more|arXiv (Cornell University)|Jul 25, 2022

3D Shape Modeling and Analysis被引用 22

一句话总结

NeuMesh 在网格支架上对神经隐式场进行编码，每个顶点有解耦的几何和纹理编码，支持网格引导的几何编辑和区域特定纹理编辑，包括纹理交换、填充和绘画。它使用可学习的符号指示、蒸馏微调训练以及具有空间感知的纹理优化，以提高编辑保真度。

ABSTRACT

Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction. However, existing neural rendering methods for editing purposes offer limited functionality, e.g., rigid transformation, or not applicable for fine-grained editing for general objects from daily lives. In this paper, we present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices, which facilitates a set of editing functionalities, including mesh-guided geometry editing, designated texture editing with texture swapping, filling and painting operations. To this end, we develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation, distillation and fine-tuning mechanism to make a steady convergence, and the spatial-aware optimization strategy to realize precise texture editing. Extensive experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability. Code is available on the project webpage: https://zju3dv.github.io/neumesh/.

研究动机与目标

• Motivate editing-capable neural rendering that aligns with traditional mesh-based workflows.
• Propose a mesh-anchored, vertex-bound latent field with disentangled geometry and appearance for local editing.
• Enable non-rigid geometry editing and fine-grained texture editing through new techniques and training schemes.
• Maintain high rendering quality while providing editable control over geometry and textures.

提出的方法

• Represent the scene as vertex-bounded latent codes (geometry l^g and texture l^t) on a mesh scaffold with learnable sign indicators for each vertex.
• Interpolate codes and projected distances from K nearest mesh vertices along a ray to form local inputs to separate geometry and radiance decoders for SDF s and color c.
• Use a distillation and fine-tuning training scheme from a pretrained coordinate-based implicit field to stabilize optimization and improve detail.
• Introduce a learnable sign indicator n per vertex to compute interpolated signed distance h(x) and enhance spatial distinguishability without requiring watertight meshes.
• Apply a spatial-aware optimization strategy for texture painting to selectively fine-tune affected texture codes based on painted rays and a dilated paint mask.
• Provide mesh-guided geometry editing by deforming the mesh alone, which automatically deforms the implicit field, maintaining consistency via relative normal rotation adjustments.
• Enable designatable texture editing via texture code swapping, texture filling from template textures, and painting from 2D views, with region-local control on the mesh.

实验结果

研究问题

• RQ1 Can a mesh-based neural implicit representation support high-quality rendering while enabling intuitive editing operations on geometry and texture?
• RQ2 How can geometry and texture be disentangled and edited locally without sacrificing rendering fidelity or introducing artifacts?
• RQ3 What mechanisms (e.g., learnable sign indicators, distillation, spatial-aware optimization) enable stable training and precise texture edits across views?
• RQ4 How well does mesh-guided editing transfer to real and synthetic data compared to previous neural rendering methods?

主要发现

• NeuMesh achieves competitive rendering quality compared to NeuS and NeuTex on DTU and NeRF 360 synthetic datasets.
• The proposed texture editing capabilities include texture swapping, texture filling with template textures, and texture painting from 2D views, with improved locality and view-consistency.
• Mesh-guided geometry editing produces smoother, more surface-consistent deformations than naïve field-warping approaches.
• A learnable sign indicator per vertex and the distillation+fine-tuning training scheme are crucial for stable training and higher image quality.
• Spatial-aware texture optimization reduces artifacts when painting from a single view and enables precise, region-specific texture edits.
• Quantitative results show Ours achieving PSNR 28.289 (DTU) and 30.945 (NeRF360) with SSIM 0.921 and 0.951, and LPIPS 0.117 and 0.043 respectively, outperforming NeuTex and NeuS on key metrics.