[Paper Review] Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPT

Tong Xie, Yuwei Wan+9 more|arXiv (Cornell University)|Apr 5, 2023

Machine Learning in Materials Science15 citations

TL;DR

The paper introduces structured information inference (SII) and shows that fine-tuning GPT-3 on a perovskite solar cell review dataset yields high-accuracy, device-level information extraction and enables downstream data analysis and device-performance prediction. It demonstrates superior NER/RE/ER/II performance over GPT-3.5 and discusses dataset building and MDP tasks.

ABSTRACT

The amount of data has growing significance in exploring cutting-edge materials and a number of datasets have been generated either by hand or automated approaches. However, the materials science field struggles to effectively utilize the abundance of data, especially in applied disciplines where materials are evaluated based on device performance rather than their properties. This article presents a new natural language processing (NLP) task called structured information inference (SII) to address the complexities of information extraction at the device level in materials science. We accomplished this task by tuning GPT-3 on an existing perovskite solar cell FAIR (Findable, Accessible, Interoperable, Reusable) dataset with 91.8% F1-score and extended the dataset with data published since its release. The produced data is formatted and normalized, enabling its direct utilization as input in subsequent data analysis. This feature empowers materials scientists to develop models by selecting high-quality review articles within their domain. Additionally, we designed experiments to predict the electrical performance of solar cells and design materials or devices with targeted parameters using large language models (LLMs). Our results demonstrate comparable performance to traditional machine learning methods without feature selection, highlighting the potential of LLMs to acquire scientific knowledge and design new materials akin to materials scientists.

Motivation & Objective

• Address the challenge of extracting device-level information from unstructured materials literature.
• Define and implement a new NLP task called structured information inference (SII).
• Fine-tune GPT-3 on a perovskite solar cell FAIR dataset to produce structured, normalized outputs.
• Demonstrate how SII outputs can seed downstream analyses and device-level predictions.

Proposed method

• Transform a large materials science corpus into a plain-text schema suitable for GPT-3 fine-tuning.
• Create a fuzzy-matching pipeline to align schemas with underlying text and select high-match samples.
• Fine-tune the 175B GPT-3 model (davinci) on 31 key-value schema outputs across 4 task types (NER, ER, RE, II).
• Evaluate SII via multi-task metrics comparing outputs to expert-annotated targets, supplemented by manual domain expert evaluation.
• Compare fine-tuned GPT-3 with GPT-3.5 to establish gains in device-level information extraction and dedicated downstream tasks.

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM perform structured information inference that combines NER, ER, RE, and II at the device level?
• RQ2 How does the fine-tuned GPT-3 perform on SII tasks compared with GPT-3.5 in perovskite solar cell data?
• RQ3 Can the produced schema outputs be directly used as input for downstream data analysis and modeling?
• RQ4 Can the framework enable device-performance prediction (MDP) tasks from literature-derived data?

Key findings

• The fine-tuned model substantially outperforms GPT-3.5 on NER (total F1 91.8 vs 28.7) and RE tasks (e.g., A-B 89.39 F1 vs 6.67).
• On manual evaluation, the fine-tuned model achieves notably higher overall scores (94.1 vs 72.1 for GPT-3.5).
• RE results show strong performance across A-B, A-C, and ABC-D relations with F1-scores of 89.39, 82.33, and 68.49 respectively for the fine-tuned model, compared to much lower GPT-3.5 scores.
• II/ER results indicate high accuracy for II (91.80) and ER (69.23 for units, 87.18 for terminology) with the fine-tuned model.
• The study finds rapid learning of the schema with data efficiency; substantial gains are observed within the first 50 examples, with diminishing returns after about 100 examples.