[Paper Review] ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model

Fukun Yin, Xin Chen+7 more|arXiv (Cornell University)|Nov 29, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

ShapeGPT proposes a unified multi-modal language model that integrates 3D shapes, text, and images into a single sequence-to-sequence framework using shape tokens derived from a VQ-VAE. By leveraging instruction-based generation across a three-stage training scheme, ShapeGPT achieves state-of-the-art performance on diverse tasks including text-to-shape, shape editing, and shape completion, demonstrating strong generalization and multimodal alignment in a single architecture.

ABSTRACT

The advent of large language models, enabling flexibility through instruction-driven approaches, has revolutionized many traditional generative tasks, but large models for 3D data, particularly in comprehensively handling 3D shapes with other modalities, are still under-explored. By achieving instruction-based shape generations, versatile multimodal generative shape models can significantly benefit various fields like 3D virtual construction and network-aided design. In this work, we present ShapeGPT, a shape-included multi-modal framework to leverage strong pre-trained language models to address multiple shape-relevant tasks. Specifically, ShapeGPT employs a word-sentence-paragraph framework to discretize continuous shapes into shape words, further assembles these words for shape sentences, as well as integrates shape with instructional text for multi-modal paragraphs. To learn this shape-language model, we use a three-stage training scheme, including shape representation, multimodal alignment, and instruction-based generation, to align shape-language codebooks and learn the intricate correlations among these modalities. Extensive experiments demonstrate that ShapeGPT achieves comparable performance across shape-relevant tasks, including text-to-shape, shape-to-text, shape completion, and shape editing.

Motivation & Objective

• Address the lack of unified, instruction-driven models for 3D shape generation that integrate multiple modalities such as text, images, and 3D shapes.
• Overcome the limitations of task-specific models that treat shape generation, captioning, and editing as isolated problems.
• Develop a holistic framework capable of handling diverse shape-relevant tasks—such as text-to-shape, shape editing, and completion—under a single, unified architecture.
• Enable flexible, instruction-guided generation by mapping 3D shapes into a language-compatible token space using a shape-specific VQ-VAE.
• Establish a multi-modal pre-training and fine-tuning pipeline that aligns shape, image, and text representations within a single language model.

Proposed method

• Discretize 3D shapes into fixed-length shape tokens using a shape-aware vector quantized variational autoencoder (VQ-VAE), enabling integration into a language model's token space.
• Represent images using a visual transformer and a visual-language perceiver to align visual features with the language model's latent space.
• Construct multi-modal paragraphs by combining shape tokens, textual descriptions, and image embeddings into a unified sequence for end-to-end sequence-to-sequence modeling.
• Implement a three-stage training scheme: (1) pre-training on shape-only data for basic shape generation and captioning, (2) multi-modal fine-tuning on image-shape-text triplets to align modalities, and (3) instruction-based generation to learn task-specific behaviors.
• Use T5-base as the core language model to process the multimodal sequences and generate outputs in a unified format across tasks.
• Apply contrastive loss with CLIP to align generated captions with ground-truth annotations and use reconstruction metrics (IoU, CD, F-score) for shape completion and editing tasks.

Experimental results

Research questions

• RQ1 Can a unified multi-modal language model effectively handle diverse 3D shape generation tasks—including text-to-shape, shape-to-text, shape editing, and shape completion—using a single architecture?
• RQ2 How does the integration of shape tokens via VQ-VAE affect the model’s ability to learn and generate coherent 3D shapes in a language model framework?
• RQ3 To what extent does instruction-based training improve generalization across multiple shape-relevant tasks compared to task-specific fine-tuning?
• RQ4 What is the impact of pre-training on shape-language alignment and the model’s ability to generate semantically and geometrically plausible shapes?
• RQ5 How do sequence length, model size, and pre-training affect performance in multi-modal 3D generation tasks?

Key findings

• ShapeGPT achieves state-of-the-art performance on text-to-shape, shape-to-text, shape completion, and shape editing tasks, with an IoU of 0.570, CD of 1.297, and F-score of 0.396 on the 512-token setting.
• The ablation study confirms that 512 shape tokens (8×8×8 grid) yield better performance than shorter sequences (64 tokens, 4×4×4 grid), indicating higher resolution improves geometric fidelity.
• Using the T5-base model (220M parameters) outperforms T5-small (60M parameters), demonstrating that larger model capacity enhances generation quality under the same training setup.
• Pre-training significantly improves model performance, as models without pre-training fail to generate coherent shapes or align modalities, highlighting its necessity for learning shape-language grammar.
• The unified sequence-to-sequence framework enables seamless task transfer without architectural changes, showing that ShapeGPT can handle diverse tasks within a single inference pipeline.
• Quantitative results show that ShapeGPT outperforms non-pretrained models and baseline methods across all evaluated metrics, confirming the effectiveness of the three-stage training scheme.