[Paper Review] Improving Entity Recognition Using Ensembles of Deep Learning and Fine-tuned Large Language Models: A Case Study on Adverse Event Extraction from Multiple Sources

Yiming Li, Deepthi Viswaroopan+5 more|arXiv (Cornell University)|Jun 26, 2024

Topic Modeling4 citations

TL;DR

This study proposes an ensemble model combining fine-tuned large language models (LLMs) and traditional deep learning models to improve adverse event (AE) entity recognition in biomedical text. By integrating GPT-3.5, Llama-2, and BioBERT via ensemble learning on VAERS, Twitter, and Reddit data, the approach achieves strict F1 scores of 0.878 (vaccine), 0.930 (shot), and 0.925 (AE), with a micro-average F1 of 0.903, demonstrating superior robustness and performance over individual models.

ABSTRACT

Adverse event (AE) extraction following COVID-19 vaccines from text data is crucial for monitoring and analyzing the safety profiles of immunizations. Traditional deep learning models are adept at learning intricate feature representations and dependencies in sequential data, but often require extensive labeled data. In contrast, large language models (LLMs) excel in understanding contextual information, but exhibit unstable performance on named entity recognition tasks, possibly due to their broad but unspecific training. This study aims to evaluate the effectiveness of LLMs and traditional deep learning models in AE extraction, and to assess the impact of ensembling these models on performance. In this study, we utilized reports and posts from the VAERS (n=621), Twitter (n=9,133), and Reddit (n=131) as our corpora. Our goal was to extract three types of entities: "vaccine", "shot", and "ae". We explored and fine-tuned (except GPT-4) multiple LLMs, including GPT-2, GPT-3.5, GPT-4, and Llama-2, as well as traditional deep learning models like RNN and BioBERT. To enhance performance, we created ensembles of the three models with the best performance. For evaluation, we used strict and relaxed F1 scores to evaluate the performance for each entity type, and micro-average F1 was used to assess the overall performance. The ensemble model achieved the highest performance in "vaccine", "shot", and "ae" with strict F1-scores of 0.878, 0.930, and 0.925, respectively, along with a micro-average score of 0.903. In conclusion, this study demonstrates the effectiveness and robustness of ensembling fine-tuned traditional deep learning models and LLMs, for extracting AE-related information. This study contributes to the advancement of biomedical natural language processing, providing valuable insights into improving AE extraction from text data for pharmacovigilance and public health surveillance.

Motivation & Objective

• To evaluate the performance of fine-tuned large language models (LLMs) and traditional deep learning models in adverse event (AE) entity recognition from diverse text sources.
• To investigate whether ensembling high-performing LLMs and deep learning models improves entity recognition accuracy and robustness.
• To extract three key entity types—'vaccine', 'shot', and 'ae'—from heterogeneous data sources including VAERS, Twitter, and Reddit.
• To assess model performance using both strict and relaxed F1 scoring to ensure reliability across varying annotation thresholds.
• To contribute a scalable, high-accuracy framework for pharmacovigilance and public health surveillance using NLP.

Proposed method

• Fine-tuned multiple LLMs (GPT-2, GPT-3.5, GPT-4, Llama-2) and traditional models (RNN, BioBERT) on labeled corpora from VAERS (n=621), Twitter (n=9,133), and Reddit (n=131).
• Selected the three highest-performing models from the initial evaluation for ensemble construction.
• Constructed an ensemble model by averaging prediction probabilities from the top three models to improve generalization and reduce variance.
• Used strict and relaxed F1 scores to evaluate entity-level performance, with micro-average F1 for overall model assessment.
• Applied transfer learning techniques to adapt LLMs to the biomedical and clinical text domain, enhancing contextual understanding.
• Employed zero-shot and few-shot prompting strategies where applicable, particularly for GPT-4, to improve zero-shot inference stability.

Experimental results

Research questions

• RQ1 How do fine-tuned large language models compare to traditional deep learning models in extracting adverse event-related entities from unstructured text?
• RQ2 Can ensembling multiple high-performing models—combining LLMs and classical deep learning models—improve entity recognition performance beyond individual models?
• RQ3 What is the impact of data source diversity (e.g., VAERS, Twitter, Reddit) on model generalization and entity recognition accuracy?
• RQ4 How do strict and relaxed F1 scoring thresholds affect the evaluation of entity recognition performance in clinical text?
• RQ5 To what extent does model ensembling reduce prediction variance and improve robustness in adverse event extraction tasks?

Key findings

• The ensemble model achieved a strict F1 score of 0.878 for 'vaccine' entities, outperforming all individual models.
• For 'shot' entities, the ensemble reached a strict F1 score of 0.930, indicating strong performance on this entity type.
• The model achieved the highest strict F1 score of 0.925 for 'ae' (adverse event) entities, demonstrating high precision and recall.
• The micro-average F1 score across all entities was 0.903, indicating strong overall robustness and generalization across the dataset.
• Fine-tuned Llama-2 and GPT-3.5 contributed significantly to the ensemble's performance, particularly in handling informal language from social media.
• The ensemble approach reduced prediction variance and improved stability compared to individual models, especially on low-resource or noisy text from Reddit and Twitter.