[Paper Review] SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarization

Yash Mathur, Sanketh Rangreji+4 more|arXiv (Cornell University)|Jun 30, 2023

Topic Modeling4 citations

TL;DR

This paper presents a GPT-4-based in-context learning approach for medical dialogue summarization in the MEDIQA-2023 shared task, using semantic similarity and Maximal Marginal Relevance (MMR) to select few-shot examples for section-wise and full-note summarization. The method achieved 2nd place among teams in Task A (section-wise summarization) and 2nd in Division-wise Summarization for Task B, demonstrating strong few-shot prompting performance despite limitations in extractive fidelity and length.

ABSTRACT

Medical dialogue summarization is challenging due to the unstructured nature of medical conversations, the use of medical terminology in gold summaries, and the need to identify key information across multiple symptom sets. We present a novel system for the Dialogue2Note Medical Summarization tasks in the MEDIQA 2023 Shared Task. Our approach for section-wise summarization (Task A) is a two-stage process of selecting semantically similar dialogues and using the top-k similar dialogues as in-context examples for GPT-4. For full-note summarization (Task B), we use a similar solution with k=1. We achieved 3rd place in Task A (2nd among all teams), 4th place in Task B Division Wise Summarization (2nd among all teams), 15th place in Task A Section Header Classification (9th among all teams), and 8th place among all teams in Task B. Our results highlight the effectiveness of few-shot prompting for this task, though we also identify several weaknesses of prompting-based approaches. We compare GPT-4 performance with several finetuned baselines. We find that GPT-4 summaries are more abstractive and shorter. We make our code publicly available.

Motivation & Objective

• To address the challenge of low-resource medical dialogue summarization due to limited annotated data and privacy constraints.
• To explore few-shot in-context learning with GPT-4 for generating accurate, structured SOAP notes from doctor-patient conversations.
• To improve summarization quality by selecting semantically relevant in-context examples using MMR and semantic similarity.
• To compare GPT-4’s zero-shot and few-shot performance against fine-tuned models like BART and T5 in a low-data medical setting.
• To analyze the limitations of prompting-based methods, including hallucination, stochasticity, and reduced extractive ability.

Proposed method

• For section-wise summarization (Task A), the system selects top-k semantically similar dialogues using sentence embeddings and MMR to form in-context demonstrations for GPT-4.
• The method uses a two-stage pipeline: (1) section header classification via fine-tuned BioBERT, and (2) in-context prompting with selected examples for summarization.
• For full-note summarization (Task B), only k=1 in-context example is used due to token length constraints, selected via semantic similarity.
• The system employs MiniLM sentence embeddings to compute semantic similarity between dialogues and candidate examples.
• In-context examples are structured with input-output templates to guide GPT-4’s generation, with temperature set to 0 to reduce stochasticity.
• The approach combines retrieval-based example selection with GPT-4’s few-shot in-context learning to generate summaries without fine-tuning.

Experimental results

Research questions

• RQ1 Can in-context learning with GPT-4 outperform fine-tuned models like BART and T5 in low-resource medical summarization?
• RQ2 How does the number of in-context examples (k) affect summarization quality in medical dialogue tasks?
• RQ3 How do GPT-4-generated summaries compare to human references in terms of length, extractive fidelity, and abstractive quality?
• RQ4 What are the key limitations of prompting-based LLMs in high-stakes medical summarization, particularly regarding hallucination and consistency?
• RQ5 Can MMR-based example selection improve summarization quality over random or non-redundant selection in few-shot settings?

Key findings

• The system achieved 3rd place overall and 2nd among teams in Task A (section-wise summarization), demonstrating strong few-shot performance.
• For Task B Division-wise Summarization, the system placed 4th overall and 2nd among teams, confirming effectiveness in full-note generation.
• Increasing the number of in-context examples (k) improved ROUGE-L from 40.3 (k=3) to 42.8 (k=7), showing a consistent performance gain.
• GPT-4 summaries were significantly shorter and more abstractive than human references and fine-tuned models, indicating reduced extractive fidelity.
• Despite high performance, the system exhibited stochasticity in outputs even with temperature=0, highlighting instability in in-context learning.
• The method revealed ethical risks, including hallucinations and privacy concerns, necessitating post-hoc fact verification and strict HIPAA compliance.