[论文解读] Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering

Maciej P. Polak, Dane Morgan|arXiv (Cornell University)|Mar 7, 2023

Machine Learning in Materials Science参考文献 42被引用 16

一句话总结

本论文提出 ChatExtract，一种零-shot 数据提取工作流，使用会话式大语言模型和提示从论文提取材料–值–单位三元组，达到高精度和高召回率，并为体积模量、临界冷却速率和HEA屈服强度构建材料数据库。

ABSTRACT

There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.

研究动机与目标

• 促进自动化从研究论文提取数据以替代人工整理。
• 开发一个零-shot数据提取工作流程，所需前期投入最小。
• 创建一套可转移的提示集合，识别数据句并提取材料–数值–单位三元组。
• 在多种属性上验证方法并构建公共数据库。
• 评估提示设计和对话记忆对提取准确度的影响。

提出的方法

• 采用两阶段工作流程：阶段A进行句子相关性分类，阶段B对相关句子进行数据提取。
• 将数据分为单值和多值情形，以定制提取提示。
• 对缺失数据进行显式处理，以抑制幻觉。
• 通过在单次对话中使用引发不确定性的跟进提示实现冗余，以提升准确性。
• 强制使用严格的是/否回应格式，以简化自动化并减少歧义。
• 利用对话式大模型中的信息保留，将后续问题与先前回答相关联。

实验结果

研究问题

• RQ1 零-shot、通过提示工程的对话式大模型是否能够在不进行微调的情况下从科学文本中准确提取材料–数值–单位三元组？
• RQ2 句子中的单值与多值数据对提取性能和错误模式有何影响？
• RQ3 跟进问题和对话记忆对数据提取的精确度和召回率有何影响？
• RQ4 ChatExtract能否在不同材料特性和数据库（包括金属及合金）之间实现可转移性？

主要发现

• 最佳模型（ChatGPT-4）在体积模量数据上达到90.8%准确率和87.7%召回率。
• 对于金属晶体金属玻璃的临界冷却速率，ChatGPT-4在一个实际数据库构建示例中达到91.6%准确率和83.6%召回率。
• 单值句子比多值句子产生更高的召回率（ChatGPT-4 100%；ChatGPT-3.5 85.5%）。
• 去除跟进问题会显著降低整体准确性（ChatGPT-4 42.7%；ChatGPT-3.5 26.5%）。
• 开始新对话（无聊天历史）在ChatGPT-3.5下保持了准确性但降低了召回，体现对话记忆的重要性。
• ChatExtract实现了原始、清洗和标准化数据库的创建，覆盖金属玻璃、高熵合金与HEA屈服强度，标准化数据达到高准确性（如标准化临界冷却速率的准确性91.9%）。