[Paper Review] Agent-based Learning of Materials Datasets from Scientific Literature

Mehrad Ansari, Seyed Mohamad Moosavi|arXiv (Cornell University)|Dec 18, 2023

Machine Learning in Materials ScienceMaterials Science12 citations

TL;DR

Introduces Eunomia, a chemistry-informed AI agent powered by GPT-4 that autonomously constructs structured materials datasets from unstructured literature, achieving competitive zero-shot performance vs fine-tuned baselines across three NLP information extraction tasks.

ABSTRACT

Advancements in machine learning and artificial intelligence are transforming materials discovery. Yet, the availability of structured experimental data remains a bottleneck. The vast corpus of scientific literature presents a valuable and rich resource of such data. However, manual dataset creation from these resources is challenging due to issues in maintaining quality and consistency, scalability limitations, and the risk of human error and bias. Therefore, in this work, we develop a chemist AI agent, powered by large language models (LLMs), to overcome these challenges by autonomously creating structured datasets from natural language text, ranging from sentences and paragraphs to extensive scientific research articles. Our chemist AI agent, Eunomia, can plan and execute actions by leveraging the existing knowledge from decades of scientific research articles, scientists, the Internet and other tools altogether. We benchmark the performance of our approach in three different information extraction tasks with various levels of complexity, including solid-state impurity doping, metal-organic framework (MOF) chemical formula, and property relations. Our results demonstrate that our zero-shot agent, with the appropriate tools, is capable of attaining performance that is either superior or comparable to the state-of-the-art fine-tuned materials information extraction methods. This approach simplifies compilation of machine learning-ready datasets for various materials discovery applications, and significantly ease the accessibility of advanced natural language processing tools for novice users in natural language. The methodology in this work is developed as an open-source software on https://github.com/AI4ChemS/Eunomia.

Motivation & Objective

• Motivate the need for extracting structured materials data from unstructured literature to accelerate ML-driven discovery.
• Develop an autonomous chemistry AI agent (Eunomia) that leverages LLMs and domain tools to create machine-learning-ready datasets without fine-tuning.
• Demonstrate zero-shot information extraction performance on three progressively complex materials NLP tasks.
• Showcase tool-augmented verification to reduce hallucinations and improve data quality.
• Provide open-source tooling and datasets to facilitate adoption by researchers and non-experts.

Proposed method

• Use a GPT-4 based agent (Eunomia) with planning and tool-use capabilities (ReAct) to extract data from text.
• Augment the LLM with chemistry-informed toolkits (Doc Search, Dataset Search, CSV Generator) to access literature, databases, and structure outputs.
• Implement a Chain-of-Verification (CoV) process to iteratively validate agent outputs against predefined criteria and reduce hallucinations.
• Benchmark three case studies (host-to-many dopants relation, MOF formula/guest species, MOF water-stability property) against a fine-tuned LLM baseline (LLM-NERRE).
• Represent outputs as structured datasets (CSV/JSON) and provide open-source code and deployment via a Streamlit app.

Experimental results

Research questions

• RQ1 Can a zero-shot, tool-augmented LLM-based agent reliably extract NER/Relation Extraction data from scientific texts in materials science?
• RQ2 How does Eunomia perform on progressively harder tasks (from single sentence to full paper) compared with a fine-tuned baseline?
• RQ3 Does the chain-of-verification technique reduce hallucinations and improve extraction accuracy and yield?
• RQ4 What is the practicality and usability of an open-source agent framework for non-experts to generate ML-ready datasets from literature?

Key findings

• Eunomia often matches or surpasses the fine-tuned baseline in case-study evaluations, especially when augmented with Chain-of-Verification.
• Case Study 1 (host-to-many dopants): Eunomia+CoV achieves the highest F1 for hosts and dopants (0.905 and 0.920 respectively) versus LLM-NERRE.
• Case Study 2 (MOF formula/guest species): Eunomia improves MOF formula F1 (0.606) over LLM-NERRE (0.424); for guest species, recalls are high (0.923) but precision is lower (0.429).
• Case Study 3 (MOF water stability): with CoV, yield 86.20% and ternary accuracy 0.91; without CoV, accuracy drops to 0.86 and yield to 82.70%.
• The approach enables rapid, zero-shot data extraction with domain-aware tools, reducing annotation burden and enabling human-in-the-loop oversight.
• All data and code are publicly available (GitHub) for reproducibility and reuse.