[Paper Review] PULSAR at MEDIQA-Sum 2023: Large Language Models Augmented by Synthetic Dialogue Convert Patient Dialogues to Medical Records

Viktor Schlegel, Hao Li+9 more|arXiv (Cornell University)|Jul 5, 2023

Machine Learning in HealthcareComputer Science3 citations

TL;DR

This paper presents PULSAR, a large language model framework that converts patient-doctor dialogues into clinical notes using domain-specific pre-training and synthetic data augmentation. Despite limited gains from pre-training and data augmentation, scaling the model size yielded the most significant performance improvements, achieving second and third place in the MEDIQA-Sum 2023 challenge's Task B.

ABSTRACT

This paper describes PULSAR, our system submission at the ImageClef 2023 MediQA-Sum task on summarising patient-doctor dialogues into clinical records. The proposed framework relies on domain-specific pre-training, to produce a specialised language model which is trained on task-specific natural data augmented by synthetic data generated by a black-box LLM. We find limited evidence towards the efficacy of domain-specific pre-training and data augmentation, while scaling up the language model yields the best performance gains. Our approach was ranked second and third among 13 submissions on task B of the challenge. Our code is available at https://github.com/yuping-wu/PULSAR.

Motivation & Objective

• To investigate the effectiveness of domain-specific pre-training and data augmentation in adapting large language models to clinical dialogue summarization.
• To evaluate whether a unified LLM framework can generalize to new medical NLP tasks with minimal adaptation.
• To improve the automation of clinical documentation by transforming patient-doctor dialogues into structured medical record sections.
• To assess the impact of model scale, pre-training, and data augmentation on abstractive summarization performance in healthcare settings.
• To explore the feasibility of using black-box LLMs for synthetic data generation to enhance low-resource medical NLP tasks.

Proposed method

• Fine-tuned a large language model using a pre-training objective that reconstructs pseudo-summarized medical terms extracted from real clinical notes.
• Generated synthetic training data by prompting black-box LLMs (e.g., GPT-3.5) with instruction templates to create task-specific dialogue-to-note pairs.
• Applied adapter-based fine-tuning to update only a small subset of parameters (1.1% of a 7B model), enabling efficient adaptation with reduced computational cost.
• Evaluated performance using ROUGE, ROUGE-L, and ROUGE-LSum metrics on both real and augmented data for Tasks B and C.
• Compared model variants across different sizes (e.g., 3B, 11B) and training strategies (full fine-tuning vs. adapter fine-tuning) to isolate the impact of scale and adaptation.
• Used a two-stage training process: first pre-training on medical terminology reconstruction, then fine-tuning on dialogue-to-note generation with real and synthetic data.

Figure 1: Example of the pre-train objective of PULSAR. Both Masked Language Model (MLM) and Gap Sentences Generation (GSG) have been employed in this scenario. Red and orange arrows exemplify the UMLS and N2C2 MLM masking strategy, respectively. Meanwhile, the black arrow shows the GSG masking stra

Experimental results

Research questions

• RQ1 Does domain-specific pre-training improve zero-shot or few-shot performance in clinical dialogue summarization?
• RQ2 To what extent does synthetic data augmentation enhance model generalization, especially when real training data is limited?
• RQ3 How does model scale influence performance in dialogue-to-medical-note generation compared to architectural adaptations like adapters?
• RQ4 Can a single LLM framework generalize effectively across multiple sub-tasks of the MEDIQA-Sum challenge without task-specific retraining?
• RQ5 How do decoding hyperparameters and prompt engineering affect generation quality and hallucination rates in clinical summarization?

Key findings

• Scaling the model size from 3B to 11B parameters led to the most substantial performance gains, with the 11B model outperforming smaller variants across all metrics.
• The 11B model achieved second and third place in Task B of the MEDIQA-Sum 2023 challenge, demonstrating strong generalization with minimal task-specific adaptation.
• Data augmentation improved performance by 1.77 ROUGE-1, 1.81 ROUGE-2, 2.94 ROUGE-L, and 2.45 ROUGE-LSum points on the test set, particularly benefiting smaller models.
• Despite improvements, data augmentation did not significantly boost performance on larger models (e.g., 3B models), suggesting saturation with sufficient real data.
• Adapter fine-tuning achieved reasonable performance but underperformed full fine-tuning, even on smaller models, indicating that full parameter updates are more effective.
• The model exhibited typical generation issues like hallucination and input copying, which were mitigated by data augmentation, especially in low-data regimes like Task C.