[Paper Review] Adding more data does not always help: A study in medical conversation summarization with PEGASUS

Varun Nair, Namit Katariya+3 more|arXiv (Cornell University)|Nov 15, 2021

Topic Modeling4 citations

TL;DR

This study investigates the impact of data scale and iterative labeling strategies on medical conversation summarization using the PEGASUS model. It finds that performance saturates quickly with as few as 1,000 labeled examples, and active learning or pseudo-labeling strategies offer no significant improvement over simple data size increases, suggesting model capacity and domain-specific jargon limit gains from more data.

ABSTRACT

Medical conversation summarization is integral in capturing information gathered during interactions between patients and physicians. Summarized conversations are used to facilitate patient hand-offs between physicians, and as part of providing care in the future. Summaries, however, can be time-consuming to produce and require domain expertise. Modern pre-trained NLP models such as PEGASUS have emerged as capable alternatives to human summarization, reaching state-of-the-art performance on many summarization benchmarks. However, many downstream tasks still require at least moderately sized datasets to achieve satisfactory performance. In this work we (1) explore the effect of dataset size on transfer learning medical conversation summarization using PEGASUS and (2) evaluate various iterative labeling strategies in the low-data regime, following their success in the classification setting. We find that model performance saturates with increase in dataset size and that the various active-learning strategies evaluated all show equivalent performance consistent with simple dataset size increase. We also find that naive iterative pseudo-labeling is on-par or slightly worse than no pseudo-labeling. Our work sheds light on the successes and challenges of translating low-data regime techniques in classification to medical conversation summarization and helps guides future work in this space. Relevant code available at \\url{https://github.com/curai/curai-research/tree/main/medical-summarization-ML4H-2021}.

Motivation & Objective

• To evaluate how dataset size affects fine-tuning performance of PEGASUS on medical conversation summarization.
• To assess whether active learning and pseudo-labeling strategies improve performance in low-data regimes for abstractive summarization.
• To determine if iterative labeling strategies outperform simple data scaling in medical dialogue summarization.
• To understand the limits of transfer learning with PEGASUS in domain-specific, low-resource medical NLP tasks.

Proposed method

• Fine-tuned the PEGASUS abstractive summarization model on a medical conversation dataset with varying amounts of human-annotated summaries.
• Implemented iterative self-training with two labeling strategies: expert labeling (HL) and pseudo-labeling (PL), using model confidence to guide selection.
• Explored multiple sampling strategies for HL: Top 1%, Bottom 1%, and Random 1% of unlabeled samples based on model uncertainty.
• Evaluated performance using F1 scores for medical concepts, affirmations, and ROUGE metrics across different training set sizes and labeling iterations.
• Compared results to prior work, including Chintagunta et al. (2021), to assess performance relative to larger labeled datasets.
• Conducted ablation studies to isolate the effects of pseudo-labeling and expert labeling, including cases with no additional labeling.

Experimental results

Research questions

• RQ1 Does increasing the size of the labeled dataset lead to consistent performance gains in medical conversation summarization with PEGASUS?
• RQ2 Do active learning strategies (e.g., selecting high-confidence or uncertain samples) outperform simple data scaling in low-data regimes?
• RQ3 Is pseudo-labeling effective in improving summarization performance when combined with iterative self-training?
• RQ4 How does model performance saturate with respect to dataset size, and at what point is further labeling ineffective?
• RQ5 To what extent do sampling strategies (Top 1%, Bottom 1%, Random 1%) influence final performance in the low-data regime?

Key findings

• Model performance on medical conversation summarization saturates quickly, with a 1,250-sample training set achieving 0.41 concept F1 and a 6,400-sample set achieving 0.43, indicating diminishing returns beyond small datasets.
• The model with 1,000 initial examples and one iteration of pseudo-labeling (1% of unlabeled set) achieved 0.44 concept F1, outperforming a single-iteration model with 6,400 human-labeled examples (0.43 F1).
• Pseudo-labeling was on par or slightly worse than no pseudo-labeling, with the largest performance gain occurring after the first self-training iteration.
• All human labeling strategies—Top 1%, Bottom 1%, and Random 1%—performed similarly, indicating that sampling strategy choice had minimal impact on final performance.
• Performance remained significantly below theoretical maximums (e.g., concept F1 of 0.44 vs. 0.63), suggesting inherent model or data limitations rather than optimization issues.
• The lack of improvement with additional data, despite hyperparameter tuning, implies that PEGASUS may be a high-bias model that struggles to adapt to domain-specific jargon in medical dialogues.