[Paper Review] CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference

Madhumita Sushil, Vanessa E. Kennedy+4 more|arXiv (Cornell University)|Aug 7, 2023

Topic Modeling30 references7 citations

TL;DR

The paper introduces a detailed oncology annotation schema and evaluates zero-shot LLMs (GPT-4, GPT-3.5-turbo, FLAN-UL2) on annotated breast and pancreatic cancer progress notes, with GPT-4 achieving the best overall performance.

ABSTRACT

Both medical care and observational studies in oncology require a thorough understanding of a patient's disease progression and treatment history, often elaborately documented in clinical notes. Despite their vital role, no current oncology information representation and annotation schema fully encapsulates the diversity of information recorded within these notes. Although large language models (LLMs) have recently exhibited impressive performance on various medical natural language processing tasks, due to the current lack of comprehensively annotated oncology datasets, an extensive evaluation of LLMs in extracting and reasoning with the complex rhetoric in oncology notes remains understudied. We developed a detailed schema for annotating textual oncology information, encompassing patient characteristics, tumor characteristics, tests, treatments, and temporality. Using a corpus of 40 de-identified breast and pancreatic cancer progress notes at University of California, San Francisco, we applied this schema to assess the zero-shot abilities of three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) to extract detailed oncological history from two narrative sections of clinical progress notes. Our team annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an exact-match F1-score of 0.51, and an average accuracy of 68% on complex tasks (expert manual evaluation on subset). Notably, it was proficient in tumor characteristic and medication extraction, and demonstrated superior performance in relational inference like adverse event detection. However, further improvements are needed before using it to reliably extract important facts from cancer progress notes needed for clinical research, complex population management, and documenting quality patient care.

Motivation & Objective

• Motivate the need for a comprehensive oncology information representation within clinical notes.
• Develop and apply a detailed schema for annotating textual oncology information (patient/tumor characteristics, tests, treatments, temporality).
• Assess zero-shot capabilities of leading LLMs in extracting and reasoning over oncology notes.
• Quantify extraction performance using automated metrics and expert evaluation on a substantively de-identified dataset.

Proposed method

• Create an annotation schema covering patient characteristics, tumor characteristics, tests, treatments, and temporality.
• Assemble a corpus of 40 de-identified breast and pancreatic cancer progress notes from UCSF.
• Annotate 9028 entities, 9986 modifiers, and 5312 relationships using the schema.
• Apply zero-shot inference of three LLMs (GPT-4, GPT-3.5-turbo, FLAN-UL2) to extract oncological history from two narrative sections.
• Evaluate against expert manual annotations using BLEU, ROUGE, exact-match F1, and overall accuracy; perform expert subset evaluation.

Experimental results

Research questions

• RQ1 How well can zero-shot LLMs extract structured oncological history from narrative progress notes using the CORAL schema?
• RQ2 Which model among GPT-4, GPT-3.5-turbo, and FLAN-UL2 performs best on entity, relationship, and relational-inference tasks in oncology notes?
• RQ3 What are the strengths and limitations of current LLMs in capturing tumor characteristics, medications, and adverse-event relationships in oncology documentation?

Key findings

• GPT-4 achieved the strongest overall performance among the evaluated models.
• Average BLEU score for GPT-4: 0.73.
• Average ROUGE score for GPT-4: 0.72.
• Exact-match F1-score for GPT-4: 0.51.
• Average accuracy on complex tasks for GPT-4: 68% (expert subset evaluation).
• GPT-4 showed proficiency in tumor characteristics and medication extraction and in relational inference such as adverse-event detection.