[Paper Review] Evaluation of GPT and BERT-based models on identifying protein-protein interactions in biomedical text

Hasin Rehana, Nur Bengisu Çam+6 more|PubMed|Mar 30, 2023

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology43 references20 citations

TL;DR

The paper evaluates GPT and BERT-based models for protein–protein interaction (PPI) extraction across three gold-standard corpora, finding BERT variants perform best overall while GPT-4 shows competitive results.

ABSTRACT

Detecting protein-protein interactions (PPIs) is crucial for understanding genetic mechanisms, disease pathogenesis, and drug design. However, with the fast-paced growth of biomedical literature, there is a growing need for automated and accurate extraction of PPIs to facilitate scientific knowledge discovery. Pre-trained language models, such as generative pre-trained transformers (GPT) and bidirectional encoder representations from transformers (BERT), have shown promising results in natural language processing (NLP) tasks. We evaluated the performance of PPI identification of multiple GPT and BERT models using three manually curated gold-standard corpora: Learning Language in Logic (LLL) with 164 PPIs in 77 sentences, Human Protein Reference Database with 163 PPIs in 145 sentences, and Interaction Extraction Performance Assessment with 335 PPIs in 486 sentences. BERT-based models achieved the best overall performance, with BioBERT achieving the highest recall (91.95%) and F1-score (86.84%) and PubMedBERT achieving the highest precision (85.25%). Interestingly, despite not being explicitly trained for biomedical texts, GPT-4 achieved commendable performance, comparable to the top-performing BERT models. It achieved a precision of 88.37%, a recall of 85.14%, and an F1-score of 86.49% on the LLL dataset. These results suggest that GPT models can effectively detect PPIs from text data, offering promising avenues for application in biomedical literature mining. Further research could explore how these models might be fine-tuned for even more specialized tasks within the biomedical domain.

Motivation & Objective

• Assess the effectiveness of GPT and BERT-based models for identifying PPIs in biomedical text.
• Compare performance across multiple well-curated PPI corpora.
• Identify which models achieve the best precision, recall, and F1 on PPI extraction tasks.

Proposed method

• Use three manually curated gold-standard corpora (LLL, HPRD, IPEA) to evaluate PPI identification.
• Benchmark multiple GPT and BERT-based models on the PPI task.
• Report precision, recall, and F1-score for each model and corpus.
• Highlight top-performing models (BioBERT, PubMedBERT) and GPT-4's comparative performance.

Experimental results

Research questions

• RQ1 How well do GPT-based models (including GPT-4) identify PPIs in biomedical text compared to BERT-based models?
• RQ2 Which model variants achieve the best precision, recall, and F1 across the three gold-standard corpora?
• RQ3 Do non-biomedical-pretrained GPT models (like GPT-4) match specialized biomedical BERT models in PPI extraction?
• RQ4 What are the trade-offs in PPI detection performance among BioBERT, PubMedBERT, and GPT-4 across different datasets?

Key findings

• BERT-based models achieved the best overall performance across the corpora.
• BioBERT achieved the highest recall at 91.95% and F1-score at 86.84%.
• PubMedBERT achieved the highest precision at 85.25%.
• GPT-4, despite not being specifically trained on biomedical text, showed commendable performance comparable to top BERT models.
• On the LLL dataset, GPT-4 achieved precision 88.37%, recall 85.14%, and F1 86.49%.