[Paper Review] A Prompt-Engineered Large Language Model, Deep Learning Workflow for Materials Classification

Siyu Liu, Tongqi Wen+2 more|arXiv (Cornell University)|Jan 31, 2024

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper proposes a novel prompt-engineered large language model (LLM)-enhanced deep learning workflow for materials classification, leveraging textual knowledge distillation from LLMs to improve model accuracy. By generating synthetic text data via prompt engineering and fine-tuning a BERT model, the method achieved up to 463% higher accuracy on sparse metallic glass classification tasks compared to traditional models, with overall 97.7% accuracy.

ABSTRACT

Large language models (LLMs) have demonstrated rapid progress across a wide array of domains. Owing to the very large number of parameters and training data in LLMs, these models inherently encompass an expansive and comprehensive materials knowledge database, far exceeding the capabilities of individual researcher. Nonetheless, devising methods to harness the knowledge embedded within LLMs for the design and discovery of novel materials remains a formidable challenge. We introduce a general approach for addressing materials classification problems, which incorporates LLMs, prompt engineering, and deep learning. Utilizing a dataset of metallic glasses as a case study, our methodology achieved an improvement of up to 463% in prediction accuracy compared to conventional classification models. These findings underscore the potential of leveraging textual knowledge generated by LLMs for materials especially in the common situation where datasets are sparse, thereby promoting innovation in materials discovery and design.

Motivation & Objective

• To address the challenge of limited and sparse experimental data in inorganic materials classification, particularly for complex systems like metallic glasses.
• To develop a generalizable deep learning workflow that leverages textual knowledge from large language models (LLMs) instead of relying solely on experimental or simulation-based features.
• To improve classification accuracy—especially for low-resource categories—by integrating LLM-generated text data with fine-tuned deep learning models.
• To establish a universal, transferable framework for materials classification that is independent of specific data formats or feature engineering.
• To demonstrate the feasibility and superiority of using LLMs as knowledge sources for materials discovery, particularly in data-scarce domains.

Proposed method

• The workflow begins by defining a materials classification problem, such as identifying metallic glass formation categories (bulk, ribbon, non-ribbon).
• Prompt engineering is used to extract structured, task-specific knowledge from a pre-trained LLM, generating synthetic text data describing material compositions and their classifications.
• The generated text data is tokenized and fed into a fine-tuned BERT model, which learns to classify materials based on the LLM-derived textual features.
• Position embedding techniques are applied to enhance the model’s ability to capture sequence-level patterns in the text inputs.
• The fine-tuned BERT model is trained on the synthetic text-label pairs and evaluated on real experimental datasets of metallic glasses.
• The approach is validated on a dataset of 5,577 metallic glasses, with performance compared against conventional machine learning models.

Experimental results

Research questions

• RQ1 Can LLM-generated textual knowledge improve classification accuracy in materials science, especially when experimental data is scarce?
• RQ2 How effective is prompt engineering in extracting actionable, structured knowledge from LLMs for materials classification tasks?
• RQ3 To what extent can a fine-tuned BERT model trained on synthetic LLM-generated text outperform traditional ML models on low-resource materials classification problems?
• RQ4 Can this workflow be generalized across different materials and classification objectives without requiring extensive retraining or feature engineering?
• RQ5 What is the impact of position embeddings and prompt optimization on model performance in materials text classification?

Key findings

• The proposed workflow achieved an overall classification accuracy of 97.7% on a metallic glass dataset of 5,577 samples, significantly outperforming conventional models.
• For the smallest class—bulk metallic glasses (BMGs), representing only ~11% of the dataset—the model achieved a 463% improvement in accuracy compared to traditional machine learning models.
• The BERT model fine-tuned on LLM-generated text data showed a 48% increase in accuracy over baseline models for the full classification task.
• The method demonstrated robust performance even with sparse and imbalanced datasets, highlighting its potential for low-data materials discovery.
• The integration of prompt-engineered LLMs as a knowledge source enabled high-quality feature representation without manual feature engineering or reliance on simulation data.
• The workflow is generalizable and can be adapted to various materials classification problems, including stability, corrosion resistance, and optoelectronic properties.