[论文解读] Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text

Hasin Rehana, Nur Bengisu Çam+6 more|PubMed|Mar 30, 2023

Biomedical Text Mining and Ontologies参考文献 43被引用 20

一句话总结

该论文评估基于 GPT 和 BERT 的模型在蛋白质-蛋白质相互作用（PPI）提取上的表现，覆盖三个金标准语料库，结果显示 BERT 变体总体表现最佳，而 GPT-4 展现出具有竞争力的结果。

ABSTRACT

Detecting protein-protein interactions (PPIs) is crucial for understanding genetic mechanisms, disease pathogenesis, and drug design. However, with the fast-paced growth of biomedical literature, there is a growing need for automated and accurate extraction of PPIs to facilitate scientific knowledge discovery. Pre-trained language models, such as generative pre-trained transformers (GPT) and bidirectional encoder representations from transformers (BERT), have shown promising results in natural language processing (NLP) tasks. We evaluated the performance of PPI identification of multiple GPT and BERT models using three manually curated gold-standard corpora: Learning Language in Logic (LLL) with 164 PPIs in 77 sentences, Human Protein Reference Database with 163 PPIs in 145 sentences, and Interaction Extraction Performance Assessment with 335 PPIs in 486 sentences. BERT-based models achieved the best overall performance, with BioBERT achieving the highest recall (91.95%) and F1-score (86.84%) and PubMedBERT achieving the highest precision (85.25%). Interestingly, despite not being explicitly trained for biomedical texts, GPT-4 achieved commendable performance, comparable to the top-performing BERT models. It achieved a precision of 88.37%, a recall of 85.14%, and an F1-score of 86.49% on the LLL dataset. These results suggest that GPT models can effectively detect PPIs from text data, offering promising avenues for application in biomedical literature mining. Further research could explore how these models might be fine-tuned for even more specialized tasks within the biomedical domain.

研究动机与目标

• 评估 GPT 和基于 BERT 的模型在生物医学文本中识别 PPI 的有效性。
• 在多个经过精心整理的 PPI 语料库中比较性能。
• 确定哪些模型在 PPI 提取任务上实现最佳的精确度、召回率和 F1 分数。

提出的方法

• 使用三份人工精心整理的金标准语料库（LLL、HPRD、IPEA）来评估 PPI 识别。
• 在 PPI 任务上对多种 GPT 和 BERT 基于的模型进行基准测试。
• 报告每个模型和每个语料库的精确度、召回率和 F1 分数。
• 突出表现最好的模型（BioBERT、PubMedBERT）以及 GPT-4 的比较性能。

实验结果

研究问题

• RQ1 GPT 基于的模型（包括 GPT-4）在生物医学文本中识别 PPI 的效果，与基于 BERT 的模型相比如何？
• RQ2 在这三份金标准语料库中，哪些模型变体在精确度、召回率和 F1 上达到最佳？
• RQ3 非生物医学预训练的 GPT 模型（如 GPT-4）在 PPI 提取方面是否能够匹配专门的生物医学 BERT 模型？
• RQ4 BioBERT、PubMedBERT 与 GPT-4 在不同数据集上在 PPI 检测性能上的权衡是什么？

主要发现

• BERT-based models achieved the best overall performance across the corpora.
• BioBERT achieved the highest recall at 91.95% and F1-score at 86.84%.
• PubMedBERT achieved the highest precision at 85.25%.
• GPT-4, despite not being specifically trained on biomedical text, showed commendable performance comparable to top BERT models.
• On the LLL dataset, GPT-4 achieved precision 88.37%, recall 85.14%, and F1 86.49%.