[Paper Review] MaScQA: A Question Answering Dataset for Investigating Materials Science Knowledge of Large Language Models

Mohd Zaki, Jayadeva+2 more|arXiv (Cornell University)|Aug 17, 2023

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper introduces MaScQA, a question-answering dataset of 650 graduate-level materials science questions curated from the GATE exam, to evaluate large language models (LLMs) on domain-specific knowledge. GPT-4 achieves 62% zero-shot accuracy, but chain-of-thought prompting offers no significant improvement, with conceptual errors (64%) being the primary performance limiter over computational ones (36%).

ABSTRACT

Information extraction and textual comprehension from materials literature are vital for developing an exhaustive knowledge base that enables accelerated materials discovery. Language models have demonstrated their capability to answer domain-specific questions and retrieve information from knowledge bases. However, there are no benchmark datasets in the materials domain that can evaluate the understanding of the key concepts by these language models. In this work, we curate a dataset of 650 challenging questions from the materials domain that require the knowledge and skills of a materials student who has cleared their undergraduate degree. We classify these questions based on their structure and the materials science domain-based subcategories. Further, we evaluate the performance of GPT-3.5 and GPT-4 models on solving these questions via zero-shot and chain of thought prompting. It is observed that GPT-4 gives the best performance (~62% accuracy) as compared to GPT-3.5. Interestingly, in contrast to the general observation, no significant improvement in accuracy is observed with the chain of thought prompting. To evaluate the limitations, we performed an error analysis, which revealed conceptual errors (~64%) as the major contributor compared to computational errors (~36%) towards the reduced performance of LLMs. We hope that the dataset and analysis performed in this work will promote further research in developing better materials science domain-specific LLMs and strategies for information extraction.

Motivation & Objective

• To develop a benchmark dataset for evaluating the understanding of materials science concepts in large language models (LLMs).
• To assess the performance of general-purpose LLMs like GPT-3.5 and GPT-4 on complex, domain-specific questions requiring graduate-level knowledge.
• To investigate whether chain-of-thought prompting improves LLM performance on materials science reasoning tasks.
• To identify the primary factors limiting LLM performance in the materials science domain, particularly conceptual vs. computational errors.

Proposed method

• Curated 650 challenging materials science questions from the Graduate Aptitude Test in Engineering (GATE) exam, requiring knowledge equivalent to a completed undergraduate materials science degree.
• Classified questions based on structural complexity (e.g., multiple-choice, numerical, conceptual) and domain subcategories (e.g., thermodynamics, mechanical behavior, atomic structure).
• Evaluated GPT-3.5 and GPT-4 using zero-shot prompting and chain-of-thought (CoT) prompting to assess reasoning and accuracy.
• Performed error analysis to categorize incorrect responses into conceptual and computational errors, quantifying their contribution to model failure.
• Used API-based inference to obtain model predictions and compared them against gold-standard answers for evaluation.
• Identified recurring error patterns across domains such as stress-strain behavior, XRD analysis, phase transitions, and thermodynamic calculations.

Experimental results

Research questions

• RQ1 How well do general-purpose LLMs perform on complex, graduate-level materials science questions?
• RQ2 Can chain-of-thought prompting improve the reasoning and accuracy of LLMs on materials science QA tasks?
• RQ3 What are the primary factors—conceptual or computational—limiting LLM performance in the materials science domain?
• RQ4 In which materials science subdomains do LLMs show the weakest performance, and why?

Key findings

• GPT-4 achieved the highest accuracy of approximately 62% on the MaScQA dataset using zero-shot prompting, outperforming GPT-3.5.
• Chain-of-thought prompting did not yield a statistically significant improvement in performance, indicating limited benefit for this dataset.
• Conceptual errors accounted for 64% of incorrect responses, while computational errors made up 36%, highlighting a major gap in domain-specific reasoning.
• The highest error rates occurred in thermodynamics (46% incorrect), atomic/crystal structure (42% incorrect), and phase transitions (41% incorrect), indicating weak model understanding in these areas.
• LLMs struggled with X-ray diffraction interpretation, fracture mechanics, creep behavior, and magnetic property calculations, reflecting poor integration of theory and experimental concepts.
• The results suggest that fine-tuning LLMs on domain-specific data or developing specialized prompting strategies is necessary to improve performance in materials science applications.