[Paper Review] Harnessing the Power of Foundation Models for Accurate Material Classification

Qingran Lin, Fengwei Yang+1 more|arXiv (Cornell University)|Mar 18, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper proposes a dual-stream framework that combines synthetic data generation with vision-language priors to improve material classification, achieving state-of-the-art results on FMD, DMS-test, and Google-test datasets. It also shows synthetic data can generalize better across domains than real DMS data.

ABSTRACT

Material classification has emerged as a critical task in computer vision and graphics, supporting the assignment of accurate material properties to a wide range of digital and real-world applications. While traditionally framed as an image classification task, this domain faces significant challenges due to the scarcity of annotated data, limiting the accuracy and generalizability of trained models. Recent advances in vision-language foundation models (VLMs) offer promising avenues to address these issues, yet existing solutions leveraging these models still exhibit unsatisfying results in material recognition tasks. In this work, we propose a novel framework that effectively harnesses foundation models to overcome data limitations and enhance classification accuracy. Our method integrates two key innovations: (a) a robust image generation and auto-labeling pipeline that creates a diverse and high-quality training dataset with material-centric images, and automatically assigns labels by fusing object semantics and material attributes in text prompts; (b) a prior incorporation strategy to distill information from VLMs, combined with a joint fine-tuning method that optimizes a pre-trained vision foundation model alongside VLM-derived priors, preserving broad generalizability while adapting to material-specific features.Extensive experiments demonstrate significant improvements on multiple datasets. We show that our synthetic dataset effectively captures the characteristics of real world materials, and the integration of priors from vision-language models significantly enhances the final performance. The source code and dataset will be released.

Motivation & Objective

• Address scarcity of high-quality material annotations by generating a large, auto-labeled synthetic dataset focused on materials.
• Bridge visual and language priors to improve material recognition and generalization.
• Preserve foundation model priors while adapting to material-specific features through collaborative fine-tuning.

Proposed method

• Generate a diverse, auto-labeled material image dataset using diffusion models with semantic grounding and Grounding DINO for region-aware labeling.
• Fuse vision features from DINOv2 with language descriptors from GPT-4v encoded by CLIP in a dual-stream architecture.
• Fine-tune the MLP head jointly on synthetic data while keeping vision and language backbones frozen to preserve priors.
• Formulate material classification as cross-modal fusion of masked visual features and language embeddings with a joint latent space.
• Evaluate on FMD (10 classes), DMS-test (21 classes), and Google-test (21 classes) to demonstrate cross-dataset generalization.

Experimental results

Research questions

• RQ1 Can synthetic, semantically grounded image generation produce high-quality, auto-labeled material datasets?
• RQ2 Does integrating vision-based and language-based priors via a dual-stream architecture improve material classification beyond zero-shot VLMs?
• RQ3 How does joint fine-tuning of the head with preserved foundation-model priors affect performance and generalization across datasets?

Key findings

• The proposed method achieves 89% accuracy on the FMD dataset and 92% on the Google-test dataset, outperforming the state-of-the-art by significant margins.
• Zero-shot CLIP and GPT-4v fall well behind the proposed method on DMS-test and Google-test, highlighting limitations of generic VLMs for material tasks.
• Ablation shows both language and vision priors contribute, with dual-stream fusion yielding the best results across datasets.
• Synthetic data trained models outperform DMS-trained baselines in cross-domain settings (e.g., 0.46 vs 0.60 mIoU on Google-test and DMS-test, 0.81 vs 0.89 mIoU on FMD).
• Auto-labeled semantic guidance improves performance by about 3-4 percentage points in mIoU/mAcc over non-semantic variants.