[Paper Review] DSM-Net: Disentangled Structured Mesh Net for Controllable Generation of Fine Geometry

Jie Yang, Kaichun Mo+3 more|arXiv (Cornell University)|Aug 12, 2020

3D Shape Modeling and Analysis23 citations

TL;DR

DSM-Net is a deep generative model that learns a disentangled structured mesh representation for 3D shapes by jointly modeling geometry and structure through hierarchical variational autoencoders with bijective mappings, enabling controllable generation such as structure interpolation while preserving geometry, and achieving state-of-the-art quality in synthesized shapes.

ABSTRACT

3D shape generation is a fundamental operation in computer graphics. While significant progress has been made, especially with recent deep generative models, it remains a challenge to synthesize high-quality geometric shapes with rich detail and complex structure, in a controllable manner. To tackle this, we introduce DSM-Net, a deep neural network that learns a disentangled structured mesh representation for 3D shapes, where two key aspects of shapes, geometry and structure, are encoded in a synergistic manner to ensure plausibility of the generated shapes, while also being disentangled as much as possible. This supports a range of novel shape generation applications with intuitive control, such as interpolation of structure (geometry) while keeping geometry (structure) unchanged. To achieve this, we simultaneously learn structure and geometry through variational autoencoders (VAEs) in a hierarchical manner for both, with bijective mappings at each level. In this manner we effectively encode geometry and structure in separate latent spaces, while ensuring their compatibility: the structure is used to guide the geometry and vice versa. At the leaf level, the part geometry is represented using a conditional part VAE, to encode high-quality geometric details, guided by the structure context as the condition. Our method not only supports controllable generation applications, but also produces high-quality synthesized shapes, outperforming state-of-the-art methods.

Motivation & Objective

• To address the challenge of generating high-quality, detailed 3D shapes with complex structure in a controllable manner.
• To disentangle the representation of geometry and structural layout in 3D shapes while maintaining their compatibility.
• To enable novel generation applications such as structure interpolation with fixed geometry.
• To improve shape plausibility and geometric detail through conditional modeling guided by structural context.
• To outperform existing methods in generating high-fidelity 3D meshes with intuitive control.

Proposed method

• DSM-Net employs a hierarchical variational autoencoder (VAE) framework to model structure and geometry separately yet compatibly.
• Bijective mappings are applied at each level of the hierarchy to ensure invertible and disentangled encoding of geometry and structure.
• At the leaf level, part geometry is modeled via a conditional VAE conditioned on the structural context for high-fidelity detail generation.
• Structure guides geometry generation, and geometry informs structure learning, creating a synergistic, bidirectional relationship.
• The method uses a mesh-based representation that supports fine geometric detail and structural coherence.
• The network is trained end-to-end to optimize both disentanglement and shape plausibility.

Experimental results

Research questions

• RQ1 Can geometry and structure in 3D shapes be effectively disentangled in a latent space while preserving compatibility?
• RQ2 Can hierarchical VAEs with bijective mappings enable controllable 3D shape generation with intuitive user controls?
• RQ3 Does conditioning geometry on structural context improve the quality and fidelity of generated shapes?
• RQ4 Can the model achieve state-of-the-art performance in generating complex, detailed 3D meshes?
• RQ5 Is interpolation of structure while preserving geometry feasible and plausible in the generated outputs?

Key findings

• DSM-Net successfully disentangles geometry and structure in 3D shapes through hierarchical VAEs with bijective mappings.
• The model enables novel controllable generation tasks, such as interpolating structure while keeping geometry fixed.
• Conditional part VAEs at the leaf level produce high-quality geometric details guided by structural context.
• The synergistic interaction between structure and geometry modeling enhances shape plausibility and realism.
• DSM-Net outperforms state-of-the-art methods in generating high-fidelity 3D meshes with complex structure and fine details.
• The method achieves superior results in both qualitative and quantitative evaluations, demonstrating improved generation quality.