[论文解读] Comparative Performance Evaluation of Large Language Models for Extracting Molecular Interactions and Pathway Knowledge

Gilchan Park, Byung-Jun Yoon+4 more|arXiv (Cornell University)|Jul 17, 2023

Biomedical Text Mining and Ontologies被引用 8

一句话总结

该论文系统地在生物NLP任务上比较了多种大型语言模型（LLMs）：蛋白质相互作用、低剂量放射影响的通路、以及基因调控关系，识别在不同设置和数据源下表现最好的模型。

ABSTRACT

Background: Identification of the interactions and regulatory relations between biomolecules play pivotal roles in understanding complex biological systems and the mechanisms underlying diverse biological functions. However, the collection of such molecular interactions has heavily relied on expert curation in the past, making it labor-intensive and time-consuming. To mitigate these challenges, we propose leveraging the capabilities of large language models (LLMs) to automate genome-scale extraction of this crucial knowledge. Results: In this study, we investigate the efficacy of various LLMs in addressing biological tasks, such as the recognition of protein interactions, identification of genes linked to pathways affected by low-dose radiation, and the delineation of gene regulatory relationships. Overall, the larger models exhibited superior performance, indicating their potential for specific tasks that involve the extraction of complex interactions among genes and proteins. Although these models possessed detailed information for distinct gene and protein groups, they faced challenges in identifying groups with diverse functions and in recognizing highly correlated gene regulatory relationships. Conclusions: By conducting a comprehensive assessment of the state-of-the-art models using well-established molecular interaction and pathway databases, our study reveals that LLMs can identify genes/proteins associated with pathways of interest and predict their interactions to a certain extent. Furthermore, these models can provide important insights, marking a noteworthy stride toward advancing our understanding of biological systems through AI-assisted knowledge discovery.

研究动机与目标

• 评估多样化大型语言模型（LLMs）在从生物医学文献中提取分子相互作用与通路知识方面的有效性。
• 比较PPI识别、LDR影响的通路基因检索以及基因调控关系任务在多台LLMs上的表现。
• 识别在特定生物学知识抽取任务中表现出色的模型，并讨论局限性与机会。

提出的方法

• 评估多种LLMs，包括Galactica、Alpaca、RST、Falcon、MPT、LLaMA2以及领域专门化的BioGPT/BioMedLM，在三个bioNLP任务上的表现。
• 以STRING、KEGG和INDRA作为数据源，构建PPI、通道基因与基因调控关系的评测集。
• 改变上下文示例数量（0–5 次示例）与提示，确认每个任务的最佳提示策略。
• 在4×NVIDIA A100 80GB GPU上进行实验，采用任务特定的批量大小。
• 报告微F1、宏F1及完全匹配计数以量化性能。

Figure 1: Confusion matrices for PPI Task2.

实验结果

研究问题

• RQ1 哪些LLM在STRING派生的人类蛋白质网络中能最好地识别蛋白质–蛋白质相互作用？
• RQ2 哪些模型在使用KEGG数据时能最准确地识别受低剂量放射暴露影响的人类通路基因？
• RQ3 LLMs在使用INDRA DB文本陈述对基因调控关系进行分类方面的表现如何？
• RQ4 模型规模或领域专长是否与跨任务的性能相关？
• RQ5 提示策略（shots）对每个任务的性能有何影响？

主要发现

• LLaMA2-Chat（70B）在PPI任务Task1中实现最高的Micro F1和Macro F1，1K 中有159个完全匹配项。
• LLaMA2-Chat（7B）在PPI Task1中的表现接近更大模型，如MPT-Chat（30B）和Galactica（30B）。
• MPT-Chat（7B）与MPT-Chat（30B）在PPI Task2（二元Yes/No）上表现最强，Micro F1分别高达0.9840和0.9350（5-shot）。
• LDR路径任务中，Galactica（30B）与MPT-Chat（30B）最准确地预测基因，BioMedLM与BioGPT-Large在领域特定数据上有显著提升。
• 在INDRA任务中，较大模型（如Galactica 30B、LLaMA2-Chat 70B、MPT-Chat 30B）优于较小的BioGPT/BioMedLM模型，表明规模与多样化训练数据有助于基因调控关系的阅读理解。
• 领域专门化的小型模型在某些专门任务上可超越大型通用模型，表明任务-领域对齐很重要。