[Paper Review] Local Large Language Models for Complex Structured Medical Tasks

Cody Bumgardner, Aaron Mullen+3 more|arXiv (Cornell University)|Aug 3, 2023

Topic Modeling4 citations

TL;DR

This paper proposes training local large language models (LLMs) on medical pathology reports to extract structured condition codes, demonstrating that LLaMA-based models significantly outperform BERT-style models in multi-label, complex structured tasks—even with reduced precision. The approach enables high-accuracy, privacy-preserving, on-premise deployment using accessible hardware.

ABSTRACT

This paper introduces an approach that combines the language reasoning capabilities of large language models (LLMs) with the benefits of local training to tackle complex, domain-specific tasks. Specifically, the authors demonstrate their approach by extracting structured condition codes from pathology reports. The proposed approach utilizes local LLMs, which can be fine-tuned to respond to specific generative instructions and provide structured outputs. The authors collected a dataset of over 150k uncurated surgical pathology reports, containing gross descriptions, final diagnoses, and condition codes. They trained different model architectures, including LLaMA, BERT and LongFormer and evaluated their performance. The results show that the LLaMA-based models significantly outperform BERT-style models across all evaluated metrics, even with extremely reduced precision. The LLaMA models performed especially well with large datasets, demonstrating their ability to handle complex, multi-label tasks. Overall, this work presents an effective approach for utilizing LLMs to perform domain-specific tasks using accessible hardware, with potential applications in the medical domain, where complex data extraction and classification are required.

Motivation & Objective

• To develop a local LLM approach that enables domain-specific, structured output generation for complex medical NLP tasks without relying on third-party APIs.
• To evaluate the performance of local LLMs (e.g., LLaMA) versus BERT-style models in extracting structured condition codes from uncurated surgical pathology reports.
• To assess the impact of dataset size, model architecture, and parameter count on classification performance in multi-label, medical coding tasks.
• To demonstrate that local LLMs can achieve high accuracy on complex, real-world medical data using standard CPU/GPU hardware, avoiding privacy and regulatory risks of cloud-based models.

Proposed method

• Fine-tuned LLaMA, BERT, LongFormer, and BioClinicalBERT models on a dataset of 150k+ uncurated surgical pathology reports containing gross descriptions, final diagnoses, and condition codes.
• Used instruction tuning to guide models to generate structured outputs in response to specific generative instructions.
• Trained models on three dataset sizes (tiny, small, large) to evaluate scalability and data complexity effects.
• Evaluated performance using F1 scores across all models and dataset sizes, with a focus on multi-label classification of condition codes.
• Employed prompt engineering and input context structuring to improve model alignment with medical coding rules and terminology.
• Deployed models on standard hardware, demonstrating feasibility without requiring high-end GPUs.

Experimental results

Research questions

• RQ1 Can local LLMs trained on medical text outperform BERT-style models in extracting structured condition codes from unstructured pathology reports?
• RQ2 How does dataset size affect the performance of local LLMs versus smaller NLP models in complex, multi-label medical coding tasks?
• RQ3 To what extent does model architecture (e.g., LLaMA vs. BERT) influence performance when fine-tuned on domain-specific, high-complexity medical data?
• RQ4 Can local LLMs achieve high accuracy in structured medical data extraction without relying on cloud-based LLM services, ensuring data privacy and compliance?

Key findings

• LLaMA-based models achieved an F1 score of 0.785 on the largest dataset, significantly outperforming BERT-style models, which scored below 0.02 across all dataset sizes.
• The LLaMA 13b model achieved an F1 of 0.785 on the large dataset, while the best BERT model (UKPathBERT) scored only 0.018, indicating a substantial performance gap.
• Performance of BERT and Longformer models peaked on the smallest dataset and declined with increased data complexity, while LLaMA models improved with larger datasets, showing superior scalability.
• No significant correlation was found between input description length and prediction accuracy, suggesting that model complexity and reasoning capacity outweigh input length effects.
• More frequent condition codes were predicted more accurately, but LLaMA models maintained strong performance even on rare codes, unlike BERT models which struggled with low-frequency labels.
• The LLaMA 7b model achieved high performance (F1 = 0.783 on large dataset) on standard CPU/GPU hardware, demonstrating feasibility for on-premise deployment without specialized infrastructure.