[Paper Review] Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

Maciej P. Polak, Dane Morgan|arXiv (Cornell University)|Mar 7, 2023

Machine Learning in Materials Science42 references16 citations

TL;DR

The paper presents ChatExtract, a zero-shot data extraction workflow using conversational LLMs and prompts to extract Material–Value–Unit triplets from papers, achieving high precision and recall and building materials databases for bulk modulus, critical cooling rates, and HEA yield strengths.

ABSTRACT

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.

Motivation & Objective

• Motivate automated data extraction from research papers to replace manual curation.
• Develop a zero-shot data extraction workflow that requires minimal up-front effort.
• Create a transferable set of prompts that identify data sentences and extract Material–Value–Unit triplets.
• Demonstrate the method on multiple properties and construct public databases.
• Evaluate how prompt design and conversational memory affect extraction accuracy.

Proposed method

• Use a two-stage workflow: Stage A for sentence relevancy classification and Stage B for data extraction from relevant sentences.
• Split data into single-valued and multi-valued cases to tailor extraction prompts.
• Incorporate explicit handling for missing data to curb hallucinations.
• Employ redundancy through uncertainty-inducing follow-up prompts within a single conversation to improve accuracy.
• Enforce a strict Yes/No answer format to simplify automation and reduce ambiguity.
• Leverage information retention in conversational LLMs to relate follow-up questions to prior responses.

Experimental results

Research questions

• RQ1 Can zero-shot, prompt-engineered conversational LLMs accurately extract Material–Value–Unit triplets from scientific text without fine-tuning?
• RQ2 How do single-valued versus multi-valued data in sentences affect extraction performance and error modes?
• RQ3 What is the impact of follow-up questions and conversational memory on precision and recall in data extraction?
• RQ4 Is ChatExtract transferable across different material properties and databases, including metals and alloys?

Key findings

• The best model (ChatGPT-4) achieved 90.8% precision and 87.7% recall on bulk modulus data.
• For critical cooling rates of bulk metallic glasses, ChatGPT-4 achieved 91.6% precision and 83.6% recall on a practical database construction example.
• Single-valued sentences yielded higher recall (100% with ChatGPT-4; 85.5% with ChatGPT-3.5) than multi-valued sentences (82.7% with ChatGPT-4; 55.9% with ChatGPT-3.5).
• Removing follow-up questions dramatically reduced overall precision (to 42.7% for ChatGPT-4 and 26.5% for ChatGPT-3.5).
• Starting a new conversation (no chat history) preserved precision with ChatGPT-3.5 but reduced recall, illustrating the importance of conversational retention.
• ChatExtract enabled creation of raw, cleaned, and standardized databases for bulk metallic glasses, high-entropy alloys, and HEA yield strengths, with standardized data achieving high precision (e.g., 91.9% precision for standardized critical cooling rates).