[Paper Review] X-Mesh: Towards Fast and Accurate Text-driven 3D Stylization via Dynamic Textual Guidance

Yiwei Ma, Xiaioqing Zhang+6 more|arXiv (Cornell University)|Mar 28, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

X-Mesh proposes a text-driven 3D stylization framework that uses a Text-guided Dynamic Attention Module (TDAM) to inject dynamic textual semantics into vertex feature extraction, enabling faster convergence and more accurate, semantically aligned stylization. It achieves state-of-the-art performance with a 29.26 MES score and 88.53 ITS, outperforming prior methods in both quality and speed on the new MIT-30 benchmark.

ABSTRACT

Text-driven 3D stylization is a complex and crucial task in the fields of computer vision (CV) and computer graphics (CG), aimed at transforming a bare mesh to fit a target text. Prior methods adopt text-independent multilayer perceptrons (MLPs) to predict the attributes of the target mesh with the supervision of CLIP loss. However, such text-independent architecture lacks textual guidance during predicting attributes, thus leading to unsatisfactory stylization and slow convergence. To address these limitations, we present X-Mesh, an innovative text-driven 3D stylization framework that incorporates a novel Text-guided Dynamic Attention Module (TDAM). The TDAM dynamically integrates the guidance of the target text by utilizing text-relevant spatial and channel-wise attentions during vertex feature extraction, resulting in more accurate attribute prediction and faster convergence speed. Furthermore, existing works lack standard benchmarks and automated metrics for evaluation, often relying on subjective and non-reproducible user studies to assess the quality of stylized 3D assets. To overcome this limitation, we introduce a new standard text-mesh benchmark, namely MIT-30, and two automated metrics, which will enable future research to achieve fair and objective comparisons. Our extensive qualitative and quantitative experiments demonstrate that X-Mesh outperforms previous state-of-the-art methods.

Motivation & Objective

• To address the lack of textual semantic guidance during vertex attribute prediction in existing text-driven 3D stylization methods, which leads to inconsistent results and slow convergence.
• To overcome the limitations of subjective, non-reproducible user studies in evaluating stylized 3D assets by introducing standardized benchmarks and automated metrics.
• To develop a framework that achieves high-quality stylization with significantly reduced training time, enabling practical real-time or near-real-time 3D content creation.
• To establish a fair and reproducible evaluation protocol for future research in text-driven 3D stylization through a new benchmark and two automated metrics.

Proposed method

• Introduces a Text-guided Dynamic Attention Module (TDAM) that dynamically generates spatial and channel-wise attention weights based on textual features to guide vertex feature extraction.
• Uses CLIP-based text encoding to generate query vectors that condition the attention modules, ensuring the predicted vertex attributes are semantically aligned with the input text.
• Employs a multi-stage training pipeline where vertex attributes (color, displacement, etc.) are predicted with attention-guided features, supervised by CLIP loss between rendered images and the text prompt.
• Constructs the MIT-30 benchmark, consisting of 30 mesh categories each paired with 5 diverse text prompts, to standardize evaluation across methods.
• Proposes two automated evaluation metrics: Multi-view Expert Score (MES) for stylization quality and Iteration for Target Score (ITS) for convergence speed.
• Applies fixed camera angles to render 24 views of each stylized mesh for consistent evaluation, enabling objective comparison across models.

Experimental results

Research questions

• RQ1 Can dynamic textual guidance during vertex attribute prediction significantly improve the semantic consistency and visual quality of text-driven 3D stylization?
• RQ2 How does the integration of text-aware attention mechanisms affect convergence speed compared to conventional text-independent MLPs?
• RQ3 Can a standardized benchmark and automated metrics replace subjective user studies in evaluating 3D stylization quality, ensuring reproducibility and fairness?
• RQ4 To what extent does the proposed framework handle complex or ambiguous text prompts without collapsing or losing semantic details?
• RQ5 Does the proposed method achieve superior performance in both stylization quality and training efficiency compared to state-of-the-art baselines?

Key findings

• X-Mesh achieves a Multi-view Expert Score (MES) of 29.26 on the MIT-30 benchmark, representing a 0.41 absolute improvement over the previous SOTA method, Text2Mesh (28.85).
• The Iteration for Target Score (ITS) of X-Mesh is 88.53, significantly lower than Text2Mesh (173.27) and TANGO (795.47), indicating much faster convergence.
• With TDAM, X-Mesh reaches stable results in under 200 iterations (approx. 3 minutes), while the baseline without TDAM requires over 500 iterations (over 8 minutes).
• The loss curve for X-Mesh with TDAM decreases faster than the baseline, confirming improved training efficiency and faster convergence.
• In complex prompt scenarios, X-Mesh with TDAM preserves geometric integrity and captures fine-grained details (e.g., 'black collar', 'colorful gloves'), whereas the baseline model collapses or ignores such attributes.
• The TDAM module enables the model to generalize better to diverse and complex prompts, demonstrating robustness and semantic awareness in attribute prediction.