[Paper Review] Does Synthetic Data Generation of LLMs Help Clinical Text Mining?

Ruixiang Tang, Xiaotian Han+2 more|arXiv (Cornell University)|Mar 8, 2023

Artificial Intelligence in Healthcare and Education80 citations

TL;DR

The paper shows zero-shot LLMs underperform on biomedical NER/RE, then proposes synthetic data generation via prompts to train a local model, achieving substantial gains in NER and RE tasks while addressing privacy concerns.

ABSTRACT

Recent advancements in large language models (LLMs) have led to the development of highly potent models like OpenAI's ChatGPT. These models have exhibited exceptional performance in a variety of tasks, such as question answering, essay composition, and code generation. However, their effectiveness in the healthcare sector remains uncertain. In this study, we seek to investigate the potential of ChatGPT to aid in clinical text mining by examining its ability to extract structured information from unstructured healthcare texts, with a focus on biological named entity recognition and relation extraction. However, our preliminary results indicate that employing ChatGPT directly for these tasks resulted in poor performance and raised privacy concerns associated with uploading patients' information to the ChatGPT API. To overcome these limitations, we propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data with labels utilizing ChatGPT and fine-tuning a local model for the downstream task. Our method has resulted in significant improvements in the performance of downstream tasks, improving the F1-score from 23.37% to 63.99% for the named entity recognition task and from 75.86% to 83.59% for the relation extraction task. Furthermore, generating data using ChatGPT can significantly reduce the time and effort required for data collection and labeling, as well as mitigate data privacy concerns. In summary, the proposed framework presents a promising solution to enhance the applicability of LLM models to clinical text mining.

Motivation & Objective

• Motivate and assess the feasibility of using LLMs for clinical text mining tasks like biomedical NER and RE.
• Evaluate zero-shot performance of ChatGPT on NER and RE in healthcare texts.
• Propose a synthetic data generation framework to train a local model while mitigating privacy risks.
• Demonstrate improvements in downstream task performance using synthetic data for NER and RE.

Proposed method

• Benchmark ChatGPT on NER and RE using prompts designed for biological tasks.
• Generate large-scale synthetic data with labels via LLMs using seed examples and prompts.
• Post-process synthetic data to remove low-quality or duplicate samples.
• Fine-tune local pre-trained language models (BERT, RoBERTa, BioBERT) on synthetic data.
• Evaluate performance against zero-shot ChatGPT, synthetic-data-fine-tuned models, and models fine-tuned on original data.

Experimental results

Research questions

• RQ1 Can ChatGPT effectively perform biomedical NER and RE in a zero-shot setup?
• RQ2 Does synthetic data generation guided by seeds and prompts improve downstream biomedical NER and RE when used to fine-tune local models?
• RQ3 How does performance of models trained on synthetic data compare to models trained on original data in biomedical NER and RE?
• RQ4 What are the privacy implications of using LLMs for clinical text mining, and can offline models with synthetic data mitigate them?

Key findings

• Zero-shot ChatGPT performs significantly worse than SOTA models trained on biomedical datasets for NER and RE.
• Fine-tuning on synthetic data markedly improves NER performance across datasets and models, approaching or matching performance achieved with original data in some cases.
• For relation extraction, fine-tuning on synthetic data yields substantial gains and in some cases matches original-data performance.
• Using synthetic data reduces the need to upload patient data to APIs, addressing privacy concerns.
• Increasing the number of synthetic sentences improves performance up to a point, after which gains plateau.
• Seed-example quantity and prompts impact data quality and downstream results; about 3500 synthetic sentences and ~80 seeds suffice in their experiments.