[Paper Review] CancerLLM: A Large Language Model in Cancer Domain

Mingchen Li, Huang, Jiatan+6 more|arXiv (Cornell University)|Jun 15, 2024

Biomedical Text Mining and Ontologies11 citations

TL;DR

CancerLLM is a 7B cancer-domain LLM trained on large clinical and pathology data, fine-tuned for phenotype extraction, diagnosis generation, and treatment plan generation, achieving state-of-the-art results and robust performance under targeted testbeds.

ABSTRACT

Medical Large Language Models (LLMs) have demonstrated impressive performance on a wide variety of medical NLP tasks; however, there still lacks a LLM specifically designed for phenotyping identification and diagnosis in cancer domain. Moreover, these LLMs typically have several billions of parameters, making them computationally expensive for healthcare systems. Thus, in this study, we propose CancerLLM, a model with 7 billion parameters and a Mistral-style architecture, pre-trained on nearly 2.7M clinical notes and over 515K pathology reports covering 17 cancer types, followed by fine-tuning on two cancer-relevant tasks, including cancer phenotypes extraction and cancer diagnosis generation. Our evaluation demonstrated that the CancerLLM achieves state-of-the-art results with F1 score of 91.78% on phenotyping extraction and 86.81% on disganois generation. It outperformed existing LLMs, with an average F1 score improvement of 9.23%. Additionally, the CancerLLM demonstrated its efficiency on time and GPU usage, and robustness comparing with other LLMs. We demonstrated that CancerLLM can potentially provide an effective and robust solution to advance clinical research and practice in cancer domain

Motivation & Objective

• Motivate the creation of a cancer-specific LLM to improve clinical NLP tasks in oncology.
• Develop a 7B model with Mistral-style architecture tailored to cancer data.
• Create and use three fine-tuning datasets for phenotype extraction, diagnosis generation, and treatment plan generation.
• Evaluate generation quality against diverse baselines and assess robustness under counterfactuals and misspellings.

Proposed method

• Pre-train a 7B Mistral-style LLM on 2,676,642 cancer clinical notes and 515,524 pathology reports (17 cancer types).
• Apply LoRA-based continued pre-training to inject cancer knowledge with specific hyperparameters (rank 8, alpha 16, dropout 0.05, LR 2e-4).
• Perform instruction tuning on three cancer-focused tasks using LoRA with rank 64 and alpha 16.
• Construct three downstream datasets for phenotype extraction, diagnosis generation, and treatment plan generation with non-overlapping train/test splits.
• Evaluate with Exact Match, BLEU-2, and ROUGE-L metrics; include robustness testbeds (counterfactuals and misspellings).
• Compare against 14 baselines spanning 7B, 8B, 13B, and 70B models; report both generation quality and efficiency (time and GPU memory).

Figure 1: The evolution of medical LLM performance on three tasks—cancer phenotype extraction, diagnosis generation, and treatment plan generation—is measured using the average F1 score, which includes Exact Match, BLEU-2, and ROUGE-L. Our CancerLLM achieves the highest performance with an F1 score

Experimental results

Research questions

• RQ1 Can a 7B cancer-domain LLM achieve state-of-the-art generation quality on cancer phenotype extraction, diagnosis generation, and treatment plan generation?
• RQ2 Do domain-specific cancer knowledge injections via continued pre-training and instruction tuning yield better performance than larger general medical LLMs?
• RQ3 How robust is CancerLLM to counterfactual labels and misspellings in clinical text?
• RQ4 What are the trade-offs in generation time and memory when deploying a compact cancer-domain LLM in clinical settings?

Key findings

• CancerLLM achieves the best overall performance among evaluated models across the three tasks, with a notable average F1 improvement of 8.1% over baselines on diagnosis generation.
• In cancer diagnosis generation, CancerLLM attains an average F1 of 86.81 and EM 83.50, outperforming all 7B, 13B, and 70B baselines.
• In cancer treatment plan generation, CancerLLM achieves an average F1 of 91.78 and EM 89.37, again leading among the tested models.
• In cancer phenotype extraction, CancerLLM achieves an average F1 of 93.98 and EM 89.37, close to or surpassing larger models while using far fewer parameters.
• Robustness testbeds show CancerLLM maintains competitive performance under counterfactual perturbations and misspellings, with some degradation as perturbation rate increases, yet often outperforming strong baselines under higher noise (e.g., counterfactual rate at 80%).
• CancerLLM demonstrates superior efficiency, with inference time 1:14:12 and GPU memory usage 5,550 MB for phenotype extraction, significantly lower than several 70B counterparts.

Figure 2: Overview of CancerLLM