[Paper Review] Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge

Gilchan Park, Byung-Jun Yoon+4 more|arXiv (Cornell University)|Jul 17, 2023

Biomedical Text Mining and Ontologies8 citations

TL;DR

The paper systematically compares multiple large language models (LLMs) on bioNLP tasks: protein–protein interactions, pathways affected by low-dose radiation, and gene regulatory relations, identifying which models perform best across settings and data sources.

ABSTRACT

Background: Identification of the interactions and regulatory relations between biomolecules play pivotal roles in understanding complex biological systems and the mechanisms underlying diverse biological functions. However, the collection of such molecular interactions has heavily relied on expert curation in the past, making it labor-intensive and time-consuming. To mitigate these challenges, we propose leveraging the capabilities of large language models (LLMs) to automate genome-scale extraction of this crucial knowledge. Results: In this study, we investigate the efficacy of various LLMs in addressing biological tasks, such as the recognition of protein interactions, identification of genes linked to pathways affected by low-dose radiation, and the delineation of gene regulatory relationships. Overall, the larger models exhibited superior performance, indicating their potential for specific tasks that involve the extraction of complex interactions among genes and proteins. Although these models possessed detailed information for distinct gene and protein groups, they faced challenges in identifying groups with diverse functions and in recognizing highly correlated gene regulatory relationships. Conclusions: By conducting a comprehensive assessment of the state-of-the-art models using well-established molecular interaction and pathway databases, our study reveals that LLMs can identify genes/proteins associated with pathways of interest and predict their interactions to a certain extent. Furthermore, these models can provide important insights, marking a noteworthy stride toward advancing our understanding of biological systems through AI-assisted knowledge discovery.

Motivation & Objective

• Assess the effectiveness of diverse large language models (LLMs) in extracting molecular interactions and pathway knowledge from biomedical literature.
• Compare PPI recognition, LDR-affected pathway gene retrieval, and gene regulatory relation tasks across multiple LLMs.
• Identify which models excel for specific biological knowledge extraction tasks and discuss limitations and opportunities.

Proposed method

• Evaluate multiple LLMs including Galactica, Alpaca, RST, Falcon, MPT, LLaMA2 and domain-specialized BioGPT/BioMedLM on three bioNLP tasks.
• Use STRING, KEGG, and INDRA as data sources to construct evaluation sets for PPI, pathway genes, and gene regulatory relations.
• Vary the number of in-context examples (0–5 shots) and prompts to identify optimal prompting strategy per task.
• Run experiments on 4× NVIDIA A100 80GB GPUs with task-specific batch sizes.
• Report micro F1, macro F1, and full-match counts to quantify performance.

Figure 1: Confusion matrices for PPI Task2.

Experimental results

Research questions

• RQ1 Which LLMs best recognize protein–protein interactions in the STRING-derived human protein network?
• RQ2 Which models most accurately identify genes in human pathways affected by low-dose radiation exposure (LDR) using KEGG data?
• RQ3 How well do LLMs classify gene regulatory relations using INDRA DB text statements?
• RQ4 Does model size or domain specialization correlate with performance across tasks?
• RQ5 How do prompting strategies (shots) influence performance for each task?

Key findings

• LLaMA2-Chat (70B) yields the highest Micro F1 and Macro F1 in PPI Task1, with 159 full matches out of 1K.
• LLaMA2-Chat (7B) shows performance close to larger models such as MPT-Chat (30B) and Galactica (30B) in PPI Task1.
• MPT-Chat (7B) and MPT-Chat (30B) perform strongest in PPI Task2 (binary yes/no) with Micro F1 up to 0.9840 and 0.9350 respectively (5-shot).
• Pathway by LDR Task: Galactica (30B) and MPT-Chat (30B) most accurately predict genes, with BioMedLM and BioGPT-Large showing notable gains on domain-specific data.
• In INDRA Task, larger models (e.g., Galactica 30B, LLaMA2-Chat 70B, MPT-Chat 30B) outperform smaller BioGPT/BioMedLM models, indicating size and diverse training data aid reading comprehension for gene regulatory relations.
• Domain-specialized smaller models can outperform larger general models on certain specialized tasks, suggesting task-domain alignment matters.