[Paper Review] Medical code prediction with multi-view convolution and description-regularized label-dependent attention

Najmeh Sadoughi, Greg P. Finley+7 more|arXiv (Cornell University)|Nov 5, 2018

Topic Modeling11 references18 citations

TL;DR

This paper proposes MVC-RLDA, a deep learning model for medical code prediction that uses multi-view convolutional networks to adaptively learn kernel sizes and label description-regularized attention to mitigate data sparsity. It achieves a new state-of-the-art micro F1-score of 55.85% on the MIMIC-III dataset, outperforming prior methods by leveraging all clinical notes and improving generalization for rare codes through attention regularization with code descriptions.

ABSTRACT

A ubiquitous task in processing electronic medical data is the assignment of standardized codes representing diagnoses and/or procedures to free-text documents such as medical reports. This is a difficult natural language processing task that requires parsing long, heterogeneous documents and selecting a set of appropriate codes from tens of thousands of possibilities---many of which have very few positive training samples. We present a deep learning system that advances the state of the art for the MIMIC-III dataset, achieving a new best micro F1-measure of 55.85\%, significantly outperforming the previous best result (Mullenbach et al. 2018). We achieve this through a number of enhancements, including two major novel contributions: multi-view convolutional channels, which effectively learn to adjust kernel sizes throughout the input; and attention regularization, mediated by natural-language code descriptions, which helps overcome sparsity for thousands of uncommon codes. These and other modifications are selected to address difficulties inherent to both automated coding specifically and deep learning generally. Finally, we investigate our accuracy results in detail to individually measure the impact of these contributions and point the way towards future algorithmic improvements.

Motivation & Objective

• To improve medical code prediction accuracy on large-scale, sparse ICD-9 code sets, particularly for low-resource (rare) codes.
• To address data sparsity in medical coding by incorporating natural language descriptions of codes into attention mechanisms.
• To enhance model robustness by leveraging all available clinical notes—not just discharge summaries—during training and inference.
• To develop a flexible deep learning architecture that dynamically captures n-gram patterns across varying receptive fields via multi-view convolution.
• To empirically validate the contribution of each architectural component to overall performance, especially in low-data regimes.

Proposed method

• The model employs a multi-view convolution (MVC) architecture with shared cross-channel max pooling, enabling adaptive kernel size learning across different n-gram levels.
• Each code has a dedicated spatial attention layer conditioned on its textual description, regularizing attention to reduce overfitting on rare codes.
• Input representations are formed by concatenating all available clinical notes (e.g., progress notes, operative reports) for each patient.
• The model uses a shared embedding layer followed by a stack of convolutional and attention layers, with label-specific attention heads tuned via code descriptions.
• A length-aware embedding (T(l)) is used to condition the output layer on input sequence length, improving generalization across variable-length reports.
• Regularization is applied by computing attention alignment scores between input tokens and code descriptions, encouraging attention to focus on semantically relevant terms.

Experimental results

Research questions

• RQ1 How does multi-view convolution improve code prediction performance compared to fixed-kernel CNNs?
• RQ2 To what extent does attention regularization using code descriptions reduce overfitting for low-resource medical codes?
• RQ3 What is the impact of using all clinical notes versus only discharge summaries on prediction accuracy?
• RQ4 How do individual components (e.g., regularization, MVC, length embedding) contribute to the final performance gain?
• RQ5 Does the model’s performance improvement vary meaningfully across codes with different training frequency distributions?

Key findings

• The proposed MVC-RLDA model achieves a new state-of-the-art micro F1-score of 55.85% on the MIMIC-III dataset, surpassing the previous best result of 53.9%.
• Removing the multi-view CNN component caused the largest drop in micro F1 (from 55.85% to 43.78% recall), indicating its critical role in capturing diverse n-gram patterns.
• Attention regularization using code descriptions consistently improved micro F1 across all frequency bins, with the largest gains observed for codes with fewer than 10 training examples.
• Using all clinical notes instead of only discharge summaries significantly improved performance, demonstrating the value of comprehensive input context.
• The length embedding component had the smallest individual impact, suggesting it contributes less than other components to overall performance.
• The model’s PR AUC and P@8 metrics were less sensitive to component removal than micro F1, indicating that F1 is most affected by recall changes, especially from the multi-view CNN.