[Paper Review] ISS: Image as Stepping Stone for Text-Guided 3D Shape Generation

Zhengzhe Liu, Peng Dai+3 more|arXiv (Cornell University)|Sep 9, 2022

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes ISS, a novel text-guided 3D shape generation framework that uses 2D images as a stepping stone to bridge the semantic gap between text and 3D shapes without requiring paired text-shape datasets. By leveraging CLIP's joint text-image embeddings and a pre-trained single-view reconstruction model, ISS performs two-stage feature-space alignment to generate high-fidelity, consistent 3D shapes in under 85 seconds, outperforming prior methods in fidelity, consistency, and stylization capabilities.

ABSTRACT

Text-guided 3D shape generation remains challenging due to the absence of large paired text-shape data, the substantial semantic gap between these two modalities, and the structural complexity of 3D shapes. This paper presents a new framework called Image as Stepping Stone (ISS) for the task by introducing 2D image as a stepping stone to connect the two modalities and to eliminate the need for paired text-shape data. Our key contribution is a two-stage feature-space-alignment approach that maps CLIP features to shapes by harnessing a pre-trained single-view reconstruction (SVR) model with multi-view supervisions: first map the CLIP image feature to the detail-rich shape space in the SVR model, then map the CLIP text feature to the shape space and optimize the mapping by encouraging CLIP consistency between the input text and the rendered images. Further, we formulate a text-guided shape stylization module to dress up the output shapes with novel textures. Beyond existing works on 3D shape generation from text, our new approach is general for creating shapes in a broad range of categories, without requiring paired text-shape data. Experimental results manifest that our approach outperforms the state-of-the-arts and our baselines in terms of fidelity and consistency with text. Further, our approach can stylize the generated shapes with both realistic and fantasy structures and textures.

Motivation & Objective

• To address the challenge of text-guided 3D shape generation without relying on large-scale paired text-shape datasets.
• To close the substantial semantic gap between text and 3D shapes, which hinders fidelity and consistency in existing methods.
• To enable generation of diverse 3D shapes across broad categories, including realistic and fantasy structures and textures.
• To reduce inference time significantly compared to prior optimization-based methods like Dream Fields.
• To achieve strong CLIP consistency between input text and generated shape views, improving semantic alignment.

Proposed method

• Introduces a two-stage feature-space-alignment framework: first mapping CLIP image features to a detail-rich 3D shape space via a pre-trained single-view reconstruction (SVR) model with multi-view supervision.
• Second, fine-tunes the mapper using CLIP consistency loss between the input text and rendered images of the generated shape to improve text-shape alignment.
• Employs a CLIP2Shape mapper that learns to map CLIP image and text features into the same shape feature space, leveraging the joint embedding space of CLIP.
• Utilizes a pre-trained SVR model to inject strong 3D shape priors into the generation process, enabling high-fidelity reconstruction.
• Applies a text-guided shape stylization module to generate novel structures and textures beyond the training distribution.
• Optimizes the shape using a combination of occupancy, color, background, and CLIP consistency losses to ensure multi-view consistency and semantic alignment.

Experimental results

Research questions

• RQ1 Can 2D images serve as an effective stepping stone to bridge the semantic gap between text and 3D shapes in zero-shot 3D generation?
• RQ2 Can a two-stage feature-space-alignment strategy improve text-to-3D shape consistency without paired text-shape data?
• RQ3 Can the proposed method generate high-fidelity 3D shapes across diverse categories, including realistic and fantasy designs, without fine-tuning on specific shapes?
• RQ4 How does the inference speed of the proposed method compare to optimization-based baselines like Dream Fields?
• RQ5 To what extent does CLIP consistency loss improve alignment between input text and generated 3D shape views?

Key findings

• The proposed ISS framework generates 3D shapes in under 85 seconds per instance, significantly faster than Dream Fields, which requires over 72 minutes per shape.
• ISS achieves superior FID scores on ShapeNet and CO3D benchmarks compared to state-of-the-art methods, indicating higher generation fidelity.
• Human perceptual evaluation confirms that ISS-generated shapes are more consistent with text descriptions than baselines like CLIP-Forge and Dream Fields.
• The method successfully generates both realistic and fantasy-style 3D shapes with coherent structures and textures, demonstrating broad applicability across categories.
• Ablation studies show that the two-stage alignment and CLIP consistency loss are critical for improving text-shape alignment and reducing semantic gap.
• The framework generalizes well beyond ShapeNet categories, producing plausible shapes even for rare or unseen object types not present in the training data.