[Paper Review] CurateGPT: A flexible language-model assisted biocuration tool

Harry Caufield, Carlo Kroll+11 more|arXiv (Cornell University)|Oct 29, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

CurateGPT is a large language model (LLM)-powered biocuration tool that enhances expert curation by integrating generative AI with trusted knowledge bases and literature sources. It enables automated reasoning, ontology searching, and external data retrieval, significantly improving curation efficiency and traceability by linking claims to supporting evidence beyond the LLM’s training data.

ABSTRACT

Effective data-driven biomedical discovery requires data curation: a time-consuming process of finding, organizing, distilling, integrating, interpreting, annotating, and validating diverse information into a structured form suitable for databases and knowledge bases. Accurate and efficient curation of these digital assets is critical to ensuring that they are FAIR, trustworthy, and sustainable. Unfortunately, expert curators face significant time and resource constraints. The rapid pace of new information being published daily is exceeding their capacity for curation. Generative AI, exemplified by instruction-tuned large language models (LLMs), has opened up new possibilities for assisting human-driven curation. The design philosophy of agents combines the emerging abilities of generative AI with more precise methods. A curator's tasks can be aided by agents for performing reasoning, searching ontologies, and integrating knowledge across external sources, all efforts otherwise requiring extensive manual effort. Our LLM-driven annotation tool, CurateGPT, melds the power of generative AI together with trusted knowledge bases and literature sources. CurateGPT streamlines the curation process, enhancing collaboration and efficiency in common workflows. Compared to direct interaction with an LLM, CurateGPT's agents enable access to information beyond that in the LLM's training data and they provide direct links to the data supporting each claim. This helps curators, researchers, and engineers scale up curation efforts to keep pace with the ever-increasing volume of scientific data.

Motivation & Objective

• Address the growing gap between the volume of biomedical literature and the capacity of expert curators to manually curate it.
• Reduce the time and effort required for data curation by leveraging generative AI to automate reasoning, ontology lookup, and external knowledge integration.
• Ensure FAIR (Findable, Accessible, Interoperable, Reusable) and trustworthy data by providing traceable, evidence-backed annotations.
• Enhance collaboration among curators, researchers, and engineers by streamlining curation workflows through AI-assisted agents.
• Overcome limitations of direct LLM interaction by enabling access to up-to-date, external data sources beyond the model’s static training distribution.

Proposed method

• Design and implement CurateGPT as a modular agent-based system that combines instruction-tuned LLMs with external knowledge retrieval.
• Integrate agents capable of performing reasoning, querying biomedical ontologies (e.g., MONDO, HPO), and retrieving evidence from scientific literature and databases.
• Use retrieval-augmented generation (RAG) techniques to ground LLM outputs in real-time, up-to-date data sources beyond the model’s training data.
• Embed citation tracking and provenance logging to ensure each annotation claim is linked to its source material, enhancing transparency and trust.
• Support collaborative curation workflows by enabling versioning, audit trails, and structured output formats compatible with knowledge bases and databases.
• Deploy the system in a secure, scalable environment to support integration into existing curation pipelines and platforms.

Experimental results

Research questions

• RQ1 Can LLM-powered agents significantly reduce the time required for manual biocuration while maintaining high accuracy and traceability?
• RQ2 To what extent can AI agents enhance curation by accessing external knowledge sources beyond the LLM’s training data?
• RQ3 How does integrating evidence tracing and source linking improve the trustworthiness and reproducibility of curated data?
• RQ4 Can agent-based curation tools scale effectively to support large-scale, collaborative curation in biomedical knowledge bases?
• RQ5 How does CurateGPT compare to direct LLM prompting in terms of curation efficiency, accuracy, and reliability?

Key findings

• CurateGPT enables curators to access and integrate information from external sources such as ontologies and literature, extending beyond the LLM’s static training data.
• The system provides direct, traceable links to evidence for each claim, increasing transparency and auditability of curated annotations.
• By combining generative reasoning with structured knowledge retrieval, CurateGPT reduces manual effort in distilling and validating complex biological information.
• The agent-based architecture allows for modular, extensible workflows that support diverse curation tasks across different biological domains.
• Compared to direct LLM interaction, CurateGPT improves reliability and consistency by grounding responses in authoritative, up-to-date data sources.
• The tool enhances collaboration by enabling structured, versioned curation with clear provenance, supporting sustainable and FAIR data practices.