[Paper Review] NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

Bangbang Yang, Chong Bao+5 more|arXiv (Cornell University)|Jul 25, 2022

3D Shape Modeling and Analysis22 citations

TL;DR

NeuMesh encodes a neural implicit field on a mesh scaffold with disentangled geometry and texture codes per vertex, enabling mesh-guided geometry editing and region-specific texture editing including texture swapping, filling, and painting. It uses learnable sign indicators, distillation-fine-tuning training, and spatially-aware texture optimization to improve editing fidelity.

ABSTRACT

Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction. However, existing neural rendering methods for editing purposes offer limited functionality, e.g., rigid transformation, or not applicable for fine-grained editing for general objects from daily lives. In this paper, we present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices, which facilitates a set of editing functionalities, including mesh-guided geometry editing, designated texture editing with texture swapping, filling and painting operations. To this end, we develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation, distillation and fine-tuning mechanism to make a steady convergence, and the spatial-aware optimization strategy to realize precise texture editing. Extensive experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability. Code is available on the project webpage: https://zju3dv.github.io/neumesh/.

Motivation & Objective

• Motivate editing-capable neural rendering that aligns with traditional mesh-based workflows.
• Propose a mesh-anchored, vertex-bound latent field with disentangled geometry and appearance for local editing.
• Enable non-rigid geometry editing and fine-grained texture editing through new techniques and training schemes.
• Maintain high rendering quality while providing editable control over geometry and textures.

Proposed method

• Represent the scene as vertex-bounded latent codes (geometry l^g and texture l^t) on a mesh scaffold with learnable sign indicators for each vertex.
• Interpolate codes and projected distances from K nearest mesh vertices along a ray to form local inputs to separate geometry and radiance decoders for SDF s and color c.
• Use a distillation and fine-tuning training scheme from a pretrained coordinate-based implicit field to stabilize optimization and improve detail.
• Introduce a learnable sign indicator n per vertex to compute interpolated signed distance h(x) and enhance spatial distinguishability without requiring watertight meshes.
• Apply a spatial-aware optimization strategy for texture painting to selectively fine-tune affected texture codes based on painted rays and a dilated paint mask.
• Provide mesh-guided geometry editing by deforming the mesh alone, which automatically deforms the implicit field, maintaining consistency via relative normal rotation adjustments.
• Enable designatable texture editing via texture code swapping, texture filling from template textures, and painting from 2D views, with region-local control on the mesh.

Experimental results

Research questions

• RQ1 Can a mesh-based neural implicit representation support high-quality rendering while enabling intuitive editing operations on geometry and texture?
• RQ2 How can geometry and texture be disentangled and edited locally without sacrificing rendering fidelity or introducing artifacts?
• RQ3 What mechanisms (e.g., learnable sign indicators, distillation, spatial-aware optimization) enable stable training and precise texture edits across views?
• RQ4 How well does mesh-guided editing transfer to real and synthetic data compared to previous neural rendering methods?

Key findings

• NeuMesh achieves competitive rendering quality compared to NeuS and NeuTex on DTU and NeRF 360 synthetic datasets.
• The proposed texture editing capabilities include texture swapping, texture filling with template textures, and texture painting from 2D views, with improved locality and view-consistency.
• Mesh-guided geometry editing produces smoother, more surface-consistent deformations than naïve field-warping approaches.
• A learnable sign indicator per vertex and the distillation+fine-tuning training scheme are crucial for stable training and higher image quality.
• Spatial-aware texture optimization reduces artifacts when painting from a single view and enables precise, region-specific texture edits.
• Quantitative results show Ours achieving PSNR 28.289 (DTU) and 30.945 (NeRF360) with SSIM 0.921 and 0.951, and LPIPS 0.117 and 0.043 respectively, outperforming NeuTex and NeuS on key metrics.