[Paper Review] Fine-tuning foundational models to code diagnoses from veterinary health records

Mayla R. Boguslav, Adam Kiehl+6 more|arXiv (Cornell University)|Oct 19, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This study fine-tunes ten pre-trained large language models (LLMs) on 246,473 manually coded veterinary EHR notes from Colorado State University’s Veterinary Teaching Hospital to automate SNOMED-CT diagnosis coding across 7,739 distinct codes. The best performance was achieved using large, clinical-specific LLMs with expansive labeled data, demonstrating that automated coding can significantly improve interoperability and support One Health research by enabling cross-species data integration.

ABSTRACT

Veterinary medical records represent a large data resource for application to veterinary and One Health clinical research efforts. Use of the data is limited by interoperability challenges including inconsistent data formats and data siloing. Clinical coding using standardized medical terminologies enhances the quality of medical records and facilitates their interoperability with veterinary and human health records from other sites. Previous studies, such as DeepTag and VetTag, evaluated the application of Natural Language Processing (NLP) to automate veterinary diagnosis coding, employing long short-term memory (LSTM) and transformer models to infer a subset of Systemized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) diagnosis codes from free-text clinical notes. This study expands on these efforts by incorporating all 7,739 distinct SNOMED-CT diagnosis codes recognized by the Colorado State University (CSU) Veterinary Teaching Hospital (VTH) and by leveraging the increasing availability of pre-trained language models (LMs). 13 freely-available pre-trained LMs were fine-tuned on the free-text notes from 246,473 manually-coded veterinary patient visits included in the CSU VTH's electronic health records (EHRs), which resulted in superior performance relative to previous efforts. The most accurate results were obtained when expansive labeled data were used to fine-tune relatively large clinical LMs, but the study also showed that comparable results can be obtained using more limited resources and non-clinical LMs. The results of this study contribute to the improvement of the quality of veterinary EHRs by investigating accessible methods for automated coding and support both animal and human health research by paving the way for more integrated and comprehensive health databases that span species and institutions.

Motivation & Objective

• To address the challenge of inconsistent and siloed veterinary medical records by enabling automated, standardized clinical coding.
• To improve the interoperability of veterinary electronic health records (EHRs) through integration with standardized terminologies like SNOMED-CT.
• To support One Health research by enabling data linkage between human and animal health records via common coding standards.
• To evaluate the performance of diverse pre-trained LLMs in diagnosing veterinary conditions from free-text clinical notes.
• To identify optimal model and data configurations for scalable, accurate diagnosis coding in resource-limited settings.

Proposed method

• Fine-tuned ten freely available pre-trained LLMs—including GatorTron, MedicalAI ClinicalBERT, VetBERT, and GPT-2 variants—on 246,473 manually coded veterinary patient visits from CSU’s VTH EHR system.
• Used a multi-class classification framework to map free-text clinical notes to 7,739 distinct SNOMED-CT diagnosis codes.
• Leveraged the OMOP Common Data Model (CDM) to standardize coding and enable future data integration across institutions and species.
• Employed transfer learning techniques to adapt general-purpose and clinical-specific LLMs to veterinary diagnostic coding tasks.
• Evaluated model performance using standard NLP metrics such as F1-score, precision, and recall across multi-label classification tasks.
• Explored trade-offs between model size, clinical specialization, and data availability in achieving high coding accuracy.

Experimental results

Research questions

• RQ1 Can pre-trained large language models be effectively fine-tuned to automate SNOMED-CT diagnosis coding in veterinary EHRs?
• RQ2 How does model architecture and clinical specialization affect coding accuracy for a comprehensive set of 7,739 veterinary diagnosis codes?
• RQ3 To what extent can non-clinical LLMs achieve comparable performance to clinical-specific models when fine-tuned on limited veterinary data?
• RQ4 What is the impact of training data size on the performance of LLMs in veterinary diagnosis coding?
• RQ5 Can fine-tuned LLMs enable interoperability between veterinary and human health records through standardized SNOMED-CT coding?

Key findings

• The most accurate diagnosis coding was achieved by fine-tuning large, clinical-specific LLMs such as MedicalAI ClinicalBERT and VetBERT on expansive labeled data from 246,473 veterinary visits.
• Fine-tuning GatorTron and other large, general-purpose LLMs yielded high performance, demonstrating that state-of-the-art results are achievable without specialized veterinary models.
• Comparable coding accuracy was obtained using smaller, non-clinical LLMs like BERT and RoBERTa when fine-tuned on sufficient data, indicating scalability across resource-limited settings.
• The study achieved a significant improvement in multi-label F1-score over prior methods like DeepTag and VetTag, which were limited to a subset of SNOMED-CT codes.
• The use of the OMOP CDM and standardized SNOMED-CT coding enables future data linkage across human and animal health systems, supporting One Health initiatives.
• The results demonstrate that automated, accurate diagnosis coding is feasible at scale, enhancing data quality and interoperability in veterinary clinical research.