[Paper Review] Automated clinical coding using off-the-shelf large language models

Joseph S. Boyle, Antanas Kascenas+3 more|arXiv (Cornell University)|Oct 10, 2023

Biomedical Text Mining and Ontologies4 citations

TL;DR

This paper proposes a zero-shot, few-shot ICD coding method using off-the-shelf large language models (LLMs) by framing coding as a hierarchical tree-search task, where LLMs retrieve relevant ICD codes through prompt-based relevance assessment. It achieves state-of-the-art macro-F1 of 0.225 on rare codes without any task-specific fine-tuning, demonstrating the feasibility of LLMs for automated clinical coding with minimal adaptation.

ABSTRACT

The task of assigning diagnostic ICD codes to patient hospital admissions is typically performed by expert human coders. Efforts towards automated ICD coding are dominated by supervised deep learning models. However, difficulties in learning to predict the large number of rare codes remain a barrier to adoption in clinical practice. In this work, we leverage off-the-shelf pre-trained generative large language models (LLMs) to develop a practical solution that is suitable for zero-shot and few-shot code assignment, with no need for further task-specific training. Unsupervised pre-training alone does not guarantee precise knowledge of the ICD ontology and specialist clinical coding task, therefore we frame the task as information extraction, providing a description of each coded concept and asking the model to retrieve related mentions. For efficiency, rather than iterating over all codes, we leverage the hierarchical nature of the ICD ontology to sparsely search for relevant codes.

Motivation & Objective

• To develop a zero-shot, few-shot ICD coding method that requires no task-specific training or fine-tuning.
• To address the challenge of rare ICD codes in automated coding by leveraging the inductive bias of the ICD-10 ontology.
• To improve performance on low-resource and rare ICD codes by using a sparse, hierarchical search strategy over the ICD tree.
• To evaluate whether off-the-shelf LLMs can perform clinical coding with minimal prompt engineering and no retraining.
• To demonstrate that LLMs can generalize to unseen or rarely occurring ICD codes using prompt injection and structured search.

Proposed method

• The method formulates ICD coding as an information extraction task, prompting the LLM to assess the relevance of each ICD code based on its textual description.
• It uses a tree-search strategy that traverses the hierarchical ICD-10 ontology, pruning branches that are unlikely to contain relevant codes.
• The LLM is prompted to evaluate relevance at each node level (e.g., chapter, block, category) using a consistent prompt template with the code’s description.
• The search is sparse and dynamic: only branches deemed relevant by the LLM are explored further, reducing computational cost and avoiding exhaustive code enumeration.
• The method injects structured knowledge of the ICD ontology into the prompt, reducing reliance on the LLM’s pre-trained knowledge of the coding system.
• Performance is evaluated using the CodiEsp dataset, with document-level labels abstracted from span-level annotations.

Experimental results

Research questions

• RQ1 Can off-the-shelf generative LLMs perform zero-shot ICD coding without any task-specific fine-tuning?
• RQ2 Does a hierarchical tree-search strategy improve performance on rare ICD codes compared to direct prompting?
• RQ3 Can prompt-based relevance assessment over the ICD ontology structure compensate for limited in-domain knowledge in LLMs?
• RQ4 How does the performance of the tree-search method compare to supervised models on rare versus common ICD codes?
• RQ5 To what extent does the hierarchical structure of the ICD ontology enable efficient and accurate code retrieval using LLMs?

Key findings

• The tree-search method achieved a macro-F1 of 0.225 on the CodiEsp test set, outperforming all baselines on rare codes.
• GPT-4 with the tree-search strategy achieved a micro-F1 of 0.157, slightly lower than the 0.216 and 0.219 micro-F1 scores of PLM-ICD, but significantly better on rare classes.
• The method demonstrated state-of-the-art performance on rare ICD codes, highlighting its ability to generalize to low-resource or unseen codes.
• Performance degraded at lower levels of the ICD tree, with micro-recall dropping to 0.192 and macro-recall to 0.216 at the Extension II level, indicating challenges in deep tree traversal.
• A common prompt template did not work across all models, requiring separate prompt engineering for Llama-2, GPT-3.5, and GPT-4, which may explain Llama-2’s lower performance.
• A key failure mode was predicting mutually exclusive codes (e.g., B27.89 and B27.80), indicating limitations in reasoning about code constraints.