[Paper Review] Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models

Maciej P. Polak, Shrey Modi+6 more|arXiv (Cornell University)|Feb 9, 2023

Machine Learning in Materials ScienceMaterials Science53 references9 citations

TL;DR

The paper presents a simple, model-agnostic workflow using large language models with optional human supervision to extract materials data from full-text papers, achieving high precision and recall for mid-sized databases with minimal coding.

ABSTRACT

Accurate and comprehensive material databases extracted from research papers are crucial for materials science and engineering, but their development requires significant human effort. With large language models (LLMs) transforming the way humans interact with text, LLMs provide an opportunity to revolutionize data extraction. In this study, we demonstrate a simple and efficient method for extracting materials data from full-text research papers leveraging the capabilities of LLMs combined with human supervision. This approach is particularly suitable for mid-sized databases and requires minimal to no coding or prior knowledge about the extracted property. It offers high recall and nearly perfect precision in the resulting database. The method is easily adaptable to new and superior language models, ensuring continued utility. We show this by evaluating and comparing its performance on GPT-3 and GPT-3.5/4 (which underlie ChatGPT), as well as free alternatives such as BART and DeBERTaV3. We provide a detailed analysis of the method's performance in extracting sentences containing bulk modulus data, achieving up to 90% precision at 96% recall, depending on the amount of human effort involved. We further demonstrate the method's broader effectiveness by developing a database of critical cooling rates for metallic glasses over twice the size of previous human curated databases.

Motivation & Objective

• Motivate efficient extraction of materials data from text to build mid-sized databases.
• Propose a flexible workflow that leverages general LLMs with minimal coding and optional human supervision.
• Show that the method achieves high precision and recall across multiple models and properties.
• Demonstrate applicability by constructing a database of bulk modulus sentences and a larger database of critical cooling rates for metallic glasses.

Proposed method

• Split papers into sentences and classify sentences as containing the target property using an LLM in zero-shot fashion.
• Optionally fine-tune the LLM with a small human-verified dataset to improve precision and recall (Step 2).
• Structurize data from positive sentences into full datapoints (material, value, unit, and optional temperature) with human-in-the-loop to ensure high precision (Step 3).
• Provide a lightweight workflow that requires little to no coding and is adaptable to different LLMs and properties.
• Compare performance across GPT-3, GPT-3.5/4, and open models like BART and DeBERTaV3, focusing on bulk modulus extraction as a benchmark.

Experimental results

Research questions

• RQ1 Can a simple zero-shot sentence classification by LLMs identify sentences containing a given material property in full-text papers?
• RQ2 How does optional human-in-the-loop fine-tuning affect precision and recall for mid-sized data extraction tasks?
• RQ3 What is the practical data throughput and human time cost for building a ~1000-entry database using this workflow?
• RQ4 Is the method adaptable to different properties and language models without substantial reengineering?

Key findings

• Zero-shot sentence classification with LLMs can identify sentences containing a target property with high recall; precision varies by model and prompt.
• Optional Step 2 human-assisted fine-tuning improves precision and recall, enabling near-perfect precision in the final structured data.
• Step 3 data structurization, guided by model probabilities and human review, yields almost perfect precision while maintaining high recall, enabling databases up to ~1000 entries within a workday.
• GPT-3/3.5/4 and open models (BART, DeBERTaV3) can be used, with performance varying by model and prompt; Chat models report 100% recall in their single reported point.
• For bulk modulus, the method achieved up to 90% precision at 96% recall depending on human effort and model used.
• Demonstrated broader utility by developing a database of critical cooling rates for metallic glasses that is twice the size of previous human-curated databases.