[Paper Review] Classifying Cancer Stage with Open-Source Clinical Large Language Models

Chia‐Hsuan Chang, Mary M. Lucas+2 more|arXiv (Cornell University)|Apr 2, 2024

Radiomics and Machine Learning in Medical Imaging5 citations

TL;DR

The paper shows that open-source clinical LLMs can extract pathologic TNM cancer stages from unstructured pathology reports without labeled training data, achieving competitive performance to a fine-tuned benchmark across T, N, and M categories using prompting strategies.

ABSTRACT

Cancer stage classification is important for making treatment and care management plans for oncology patients. Information on staging is often included in unstructured form in clinical, pathology, radiology and other free-text reports in the electronic health record system, requiring extensive work to parse and obtain. To facilitate the extraction of this information, previous NLP approaches rely on labeled training datasets, which are labor-intensive to prepare. In this study, we demonstrate that without any labeled training data, open-source clinical large language models (LLMs) can extract pathologic tumor-node-metastasis (pTNM) staging information from real-world pathology reports. Our experiments compare LLMs and a BERT-based model fine-tuned using the labeled data. Our findings suggest that while LLMs still exhibit subpar performance in Tumor (T) classification, with the appropriate adoption of prompting strategies, they can achieve comparable performance on Metastasis (M) classification and improved performance on Node (N) classification.

Motivation & Objective

• Motivate automatic extraction of cancer TNM staging from unstructured pathology reports.
• Evaluate open-source clinical LLMs for pTNM classification without labeled training data.
• Compare LLM prompting strategies and benchmark performance against a fine-tuned model.
• Assess robustness across T, N, and M categories and across cancer types within TCGA pathology reports.

Proposed method

• Use TCGA pathology reports (n=6,940 with ground truth) to evaluate TNM classification without fine-tuning.
• Compare three open-source LLMs: Llama-2-70b-chat, ClinicalCamel-70B, and Med42-70B against a fine-tuned Clinical-BigBird baseline.
• Apply three prompting strategies: Zero-shot, Zero-shot Chain-of-Thoughts (ZS-COT), and Few-shots.
• Extract TNM labels via regex patterns after model output (T: T1–T4, N: N0–N3, M: M0–M1).
• Evaluate performance with macro precision, recall, and F1, using bootstrapping (B=500) to compute 95% CIs.

Experimental results

Research questions

• RQ1 Can open-source clinical LLMs extract pTNM staging from real-world pathology reports without labeled training data?
• RQ2 How do different prompting strategies affect TNM classification performance for T, N, and M categories?
• RQ3 How do open-source clinical LLMs compare to a fine-tuned Clinical-BigBird baseline on pTNM extraction?
• RQ4 Does performance vary by cancer type (e.g., BRCA, LUAD) or by TNM category?
• RQ5 What are the strengths and limitations of deploying open-source LLMs for clinical staging in real-world heterogeneous reports?

Key findings

• Open-source LLMs can extract pTNM staging from pathology reports without training data.
• ClinicalCamel-70B and Med42-70B outperform Llama-2-70b-chat under zero-shot prompting and achieve macro F1 comparable to or better than Clinical-BigBird for N and M categories.
• Zero-shot chain-of-thought prompting improves macro F1 across T, N, and M versus zero-shot alone for several models.
• Few-shot prompting generally does not improve macro F1 and can degrade performance due to dataset variability across institutions.
• Med42-70B with ZS-COT or FS shows strong performance in N and M categories and can in some cancer-type analyses surpass Clinical-BigBird.
• M1 (distant metastasis) remains the most challenging class across models, with consistently lower macro F1 for rare classes.