[Paper Review] MATLABER: Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR

Xudong Xu, Zhaoyang Lyu+2 more|arXiv (Cornell University)|Aug 18, 2023

Image Processing and 3D ReconstructionComputer Science3 citations

TL;DR

MATLABER proposes a material-aware text-to-3D generation framework using a latent BRDF auto-encoder trained on real-world BRDF datasets to disentangle material from environment lighting. By predicting latent BRDF codes instead of raw BRDF parameters, it achieves high-fidelity, realistic materials that enable relighting and material editing, outperforming prior methods in realism, detail, and disentanglement.

ABSTRACT

Based on powerful text-to-image diffusion models, text-to-3D generation has made significant progress in generating compelling geometry and appearance. However, existing methods still struggle to recover high-fidelity object materials, either only considering Lambertian reflectance, or failing to disentangle BRDF materials from the environment lights. In this work, we propose Material-Aware Text-to-3D via LAtent BRDF auto-EncodeR ( extbf{MATLABER}) that leverages a novel latent BRDF auto-encoder for material generation. We train this auto-encoder with large-scale real-world BRDF collections and ensure the smoothness of its latent space, which implicitly acts as a natural distribution of materials. During appearance modeling in text-to-3D generation, the latent BRDF embeddings, rather than BRDF parameters, are predicted via a material network. Through exhaustive experiments, our approach demonstrates the superiority over existing ones in generating realistic and coherent object materials. Moreover, high-quality materials naturally enable multiple downstream tasks such as relighting and material editing. Code and model will be publicly available at \url{https://sheldontsui.github.io/projects/Matlaber}.

Motivation & Objective

• To address the limitation of existing text-to-3D methods in generating high-fidelity, realistic materials that are disentangled from environment lighting.
• To leverage large-scale real-world BRDF collections (e.g., TwoShotBRDF) to learn a smooth, natural distribution of materials in latent space.
• To enable downstream applications such as relighting and material editing by generating accurate, editable BRDF representations.
• To improve appearance quality and coherence in text-to-3D synthesis by replacing direct BRDF prediction with latent code optimization.

Proposed method

• A novel latent BRDF auto-encoder is trained on large-scale real-world BRDF data (e.g., TwoShotBRDF) to learn a smooth, continuous latent space representing natural material distributions.
• The auto-encoder maps raw BRDFs to compact, disentangled latent codes, which serve as priors for material generation in text-to-3D pipelines.
• During text-to-3D generation, a material network predicts latent BRDF embeddings conditioned on text prompts, rather than predicting raw BRDF parameters.
• The method uses a diffusion-based 3D generation framework (e.g., SDS loss) with NeRF or DMTet representations, where appearance is optimized using predicted latent BRDF codes.
• The smoothness of the latent space ensures realistic and coherent material generation, even under varying lighting conditions.
• The framework supports relighting and material editing via interpolation of latent BRDF codes, enabling natural morphing between material types.

Experimental results

Research questions

• RQ1 Can a latent BRDF auto-encoder trained on real-world BRDF data effectively disentangle material from environment lighting in text-to-3D generation?
• RQ2 Does predicting latent BRDF codes instead of raw BRDF parameters lead to more realistic and coherent material generation?
• RQ3 Can the generated BRDF materials support practical downstream tasks such as relighting and material editing?
• RQ4 How does the proposed method compare to state-of-the-art text-to-3D models in terms of material realism, detail, and disentanglement?

Key findings

• MATLABER achieves superior performance in material realism, detail richness, and disentanglement compared to DreamFusion, Magic3D, and Fantasia3D, as confirmed by user studies.
• User studies with 80 participants showed that MATLABER outperformed all baselines in realism, detail, and disentanglement, with a significant margin in disentanglement scores.
• The method enables photorealistic relighting under various HDR environment maps, with consistent and natural appearance changes across different lighting conditions.
• Material interpolation via latent code blending produces smooth, visually plausible transitions between materials, such as from gold to silver on a goblet.
• The framework supports high-fidelity material editing and scene manipulation, enabling applications previously unattainable with prior text-to-3D models.
• Despite high-quality materials, artifacts appear under novel lighting conditions due to imperfect geometry, indicating a need for future work on geometry refinement.