[Paper Review] Classification of Radiology Reports Using Neural Attention Models

Bonggun Shin, Falgun H. Chokshi+2 more|arXiv (Cornell University)|Aug 22, 2017

Topic Modeling27 references3 citations

TL;DR

This paper proposes a convolutional neural network with a novel attention mechanism to classify radiology head CT reports into five clinical categories, achieving higher accuracy than traditional models and providing interpretable heatmaps that highlight key diagnostic terms. The approach improves classification performance on a large, double-annotated dataset while enhancing model transparency through attention-based rationales.

ABSTRACT

The electronic health record (EHR) contains a large amount of multi-dimensional and unstructured clinical data of significant operational and research value. Distinguished from previous studies, our approach embraces a double-annotated dataset and strays away from obscure "black-box" models to comprehensive deep learning models. In this paper, we present a novel neural attention mechanism that not only classifies clinically important findings. Specifically, convolutional neural networks (CNN) with attention analysis are used to classify radiology head computed tomography reports based on five categories that radiologists would account for in assessing acute and communicable findings in daily practice. The experiments show that our CNN attention models outperform non-neural models, especially when trained on a larger dataset. Our attention analysis demonstrates the intuition behind the classifier's decision by generating a heatmap that highlights attended terms used by the CNN model; this is valuable when potential downstream medical decisions are to be performed by human experts or the classifier information is to be used in cohort construction such as for epidemiological studies.

Motivation & Objective

• To develop a deep learning model that classifies radiology reports with higher accuracy than traditional non-neural methods.
• To enhance model interpretability by introducing an attention mechanism that highlights clinically relevant terms in reports.
• To address the limitations of rule-based and sparse vector models by using dense word embeddings and contextual understanding.
• To evaluate the model on a large, double-annotated dataset of head CT reports from ICU patients with altered mental status.
• To demonstrate that attention mechanisms can provide rationales for predictions, supporting clinical decision-making and cohort construction.

Proposed method

• A convolutional neural network (CNN) is trained on dense word embeddings instead of sparse n-gram representations to capture semantic context in radiology reports.
• An attention mechanism is integrated into the CNN to assign weights to individual words, emphasizing those most relevant to classification.
• The attention mechanism generates a heatmap that visualizes which terms the model focuses on during prediction, offering interpretability.
• The model is trained and evaluated on a large, double-annotated dataset of 500 head CT reports labeled by two radiologists and adjudicated by a third.
• Five distinct classification tasks are defined: intracranial hemorrhage, mass effect, midline shift, hydrocephalus, and skull fracture.
• Hyperparameter tuning is performed across multiple configurations to optimize performance, with the best models selected based on development set performance.

Figure 1: The overview of our CNN model for document classification.

Experimental results

Research questions

• RQ1 Can a deep learning model with attention improve classification accuracy of radiology reports compared to traditional machine learning models?
• RQ2 Does the attention mechanism provide meaningful, clinically interpretable rationales by highlighting key diagnostic terms in reports?
• RQ3 Can the model achieve performance exceeding human inter-rater agreement in certain clinical classification tasks?
• RQ4 How does model performance scale with dataset size when using deep learning techniques?
• RQ5 Can attention heatmaps effectively reflect the clinical reasoning behind model predictions in radiology reports?

Key findings

• The proposed neural attention model (NAM) achieved test set accuracies of 88.0%, 87.5%, 85.0%, 87.5%, and 87.0% across five classification tasks, outperforming the SVM baseline.
• In three of the five tasks (bleeding, mass effect, and midline shift), the model's accuracy exceeded the human inter-rater agreement score, indicating superior performance.
• The attention mechanism successfully highlighted clinically relevant terms such as 'intraparenchymal hemorrhage' and 'sulcal effacement', demonstrating its ability to generate interpretable rationales.
• The CNN with attention mechanism outperformed non-neural models by over 3% on average, especially when trained on larger datasets.
• The model's attention heatmaps confirmed that it focused on the correct diagnostic keywords depending on the task, validating its clinical relevance.
• The NAM model was preferred over the standard CNN due to its interpretability, despite similar performance, because it provides attention-based explanations.

(a) Given a document matrix, the attention matrix is first created by performing multiple convolutions. The attention vector is then created by performing max pooling on each row of the attention matrix.