[Paper Review] Development of deep learning algorithms to categorize free-text notes pertaining to diabetes: convolution neural networks achieve higher accuracy than support vector machines

Boyi Yang, Adam Wright|arXiv (Cornell University)|Sep 16, 2018

Machine Learning in Healthcare17 references3 citations

TL;DR

This study develops and evaluates deep learning models—particularly convolutional neural networks (CNNs)—for classifying diabetes-related notes in electronic health records (EHRs). Using 2,000 annotated EHR notes from Brigham and Women’s Hospital and 1,000 from the University of Texas Physicians, the separable CNN model achieved an AUC of 0.975 on the training institution’s test set and 0.875 on external validation, outperforming support vector machines (SVMs) in intra-institutional accuracy while showing trade-offs in generalization across institutions.

ABSTRACT

Health professionals can use natural language processing (NLP) technologies when reviewing electronic health records (EHR). Machine learning free-text classifiers can help them identify problems and make critical decisions. We aim to develop deep learning neural network algorithms that identify EHR progress notes pertaining to diabetes and validate the algorithms at two institutions. The data used are 2,000 EHR progress notes retrieved from patients with diabetes and all notes were annotated manually as diabetic or non-diabetic. Several deep learning classifiers were developed, and their performances were evaluated with the area under the ROC curve (AUC). The convolutional neural network (CNN) model with a separable convolution layer accurately identified diabetes-related notes in the Brigham and Womens Hospital testing set with the highest AUC of 0.975. Deep learning classifiers can be used to identify EHR progress notes pertaining to diabetes. In particular, the CNN-based classifier can achieve a higher AUC than an SVM-based classifier.

Motivation & Objective

• To develop deep learning models that accurately classify free-text EHR notes related to diabetes.
• To compare the performance of convolutional neural networks (CNNs), recurrent neural networks (RNNs), and hybrid models against traditional support vector machines (SVMs).
• To evaluate model generalizability by validating on an external dataset from a different institution.
• To determine whether deep learning models can outperform SVMs in accuracy while maintaining practical deployment feasibility.

Proposed method

• The study used 2,000 manually annotated EHR progress notes from Brigham and Women’s Hospital (1,000 for training, 1,000 for testing), with an additional 1,000 notes from the University of Texas Physicians used for external validation.
• Text was tokenized and encoded using a word-to-number index map, resulting in a vocabulary of 20,218 unique words.
• Multiple deep learning architectures were trained, including standard CNNs, separable CNNs, RNNs, and hybrid CNN-RNN models, with word embeddings as input.
• Model performance was evaluated using the area under the ROC curve (AUC), with early stopping based on validation loss.
• The separable CNN (Model g) was optimized for speed and accuracy, achieving the best balance between performance and inference time.
• Hyperparameters were tuned using fixed random seeds at both NumPy and TensorFlow levels to ensure reproducibility.

Experimental results

Research questions

• RQ1 Can deep learning models, particularly CNNs, outperform traditional SVMs in classifying diabetes-related EHR notes?
• RQ2 How does model performance vary between internal testing (Brigham and Women’s Hospital) and external validation (University of Texas Physicians)?
• RQ3 Does combining CNNs with RNNs (e.g., bidirectional LSTM) improve classification performance on specialized EHR text?
• RQ4 To what extent do deep learning models generalize across institutions with different clinical note-writing styles?
• RQ5 Can separable convolutions improve both accuracy and inference speed compared to standard convolutions?

Key findings

• The separable CNN model (Model g) achieved the highest test AUC of 0.975 on the Brigham and Women’s Hospital dataset, significantly outperforming the SVM baseline.
• On the external validation set from the University of Texas Physicians, the best CNN model achieved an AUC of 0.875, which was lower than its internal performance but still higher than the SVM’s generalization performance.
• RNN-based models, including bidirectional LSTMs, did not improve performance over pure CNNs and were less effective for binary classification of highly specialized EHR notes.
• Adding LSTM layers after CNNs did not improve performance, and hybrid CNN-RNN models did not show synergistic gains on this dataset.
• The CNN model trained on BWH data captured institution-specific writing patterns, which enhanced accuracy within that institution but reduced generalization to another institution.
• Despite lower generalization, the CNN model was faster and more accurate than SVM within a single institution, with training completed in 161 seconds on a standard laptop.