[Paper Review] SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation

Juil Koo, Seung-Woo Yoo+2 more|arXiv (Cornell University)|Mar 21, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

SALAD proposes a cascaded latent diffusion model for 3D shape generation and manipulation using a part-level implicit representation, enabling state-of-the-art generation quality and zero-shot part-level editing without fine-tuning. By decoupling extrinsic and intrinsic part embeddings and applying diffusion in a two-phase cascade, it achieves superior performance in shape generation, part mixing, refinement, and text-guided completion.

ABSTRACT

We present a cascaded diffusion model based on a part-level implicit 3D representation. Our model achieves state-of-the-art generation quality and also enables part-level shape editing and manipulation without any additional training in conditional setup. Diffusion models have demonstrated impressive capabilities in data generation as well as zero-shot completion and editing via a guided reverse process. Recent research on 3D diffusion models has focused on improving their generation capabilities with various data representations, while the absence of structural information has limited their capability in completion and editing tasks. We thus propose our novel diffusion model using a part-level implicit representation. To effectively learn diffusion with high-dimensional embedding vectors of parts, we propose a cascaded framework, learning diffusion first on a low-dimensional subspace encoding extrinsic parameters of parts and then on the other high-dimensional subspace encoding intrinsic attributes. In the experiments, we demonstrate the outperformance of our method compared with the previous ones both in generation and part-level completion and manipulation tasks.

Motivation & Objective

• To develop a 3D diffusion model that achieves state-of-the-art generation quality while enabling zero-shot part-level manipulation.
• To address the limitation of existing 3D diffusion models in editing and completion due to lack of structural and spatial information in latent codes.
• To design a cascaded diffusion framework that effectively learns high-dimensional part embeddings by separating extrinsic and intrinsic components.
• To enable flexible, user-driven shape editing via text conditioning and part selection without retraining.
• To demonstrate the model's versatility across diverse manipulation tasks including part mixing, refinement, and text-guided completion.

Proposed method

• Employs a part-level implicit representation where each part is encoded by a disentangled embedding vector capturing both extrinsic (position, orientation) and intrinsic (geometry details) parameters.
• Introduces a two-phase cascaded diffusion pipeline: first generating extrinsic parameters in a low-dimensional subspace, then producing intrinsic latent codes conditioned on the extrinsics.
• Uses a Transformer-based architecture with timestep-conditioned self-attention to ensure permutation invariance and global communication across parts in the set-based representation.
• Applies noise scheduling and reverse denoising in the latent space to generate complete 3D shapes from random noise.
• Conditions the reverse diffusion process on text embeddings via AdaLN to enable text-guided generation and completion.
• Integrates with a text-driven semantic part segmentation model (GAUSSGLOT) to allow interactive, text-driven selection and editing of individual parts.

Experimental results

Research questions

• RQ1 Can a 3D diffusion model trained on part-level representations achieve state-of-the-art generation quality while enabling zero-shot part-level editing?
• RQ2 How does separating extrinsic and intrinsic part embeddings improve the quality and controllability of 3D shape generation?
• RQ3 To what extent can a latent diffusion model with disentangled part representations support diverse manipulation tasks like part mixing, refinement, and text-guided completion?
• RQ4 Can the model generalize to conditional generation tasks such as text-to-3D shape synthesis without fine-tuning?
• RQ5 How does the cascaded diffusion design improve training stability and generation quality in high-dimensional part embedding spaces?

Key findings

• SALAD achieves SOTA performance in 3D shape generation, outperforming SPAGHETTI and other baselines on CD and EMD metrics across all shape classes.
• In part mixing and refinement, SALAD improves output quality, with a 3.21% reduction in CD and 1.83% reduction in EMD on average compared to SPAGHETTI.
• For text-guided generation, SALAD achieves a 4.043 FPD score, significantly outperforming AutoSDF’s 31.53, indicating higher fidelity and realism.
• In text-guided part completion, SALAD successfully completes selected parts based on textual descriptions, as validated by qualitative results and neural evaluator preference (42.22 NEP vs. 38.98 for AutoSDF).
• The model demonstrates strong zero-shot capabilities in editing, with noticeable improvements in 1-NNA metrics across all classes, indicating better local shape consistency.
• The cascaded diffusion design enables effective learning in high-dimensional latent spaces, improving generation quality without increasing computational cost.