[Paper Review] Evaluating robustness of language models for chief complaint extraction from patient-generated text

Ilya Valmianski, Caleb Goodwin+3 more|arXiv (Cornell University)|Nov 15, 2019

Topic Modeling14 references4 citations

TL;DR

This study evaluates the robustness of fine-tuned BERT models and a TF-IDF baseline for extracting structured chief complaints from patient-generated text using a novel dataset of ~200,000 patient reasons-for-visit. Surprisingly, the TF-IDF baseline outperforms even the best fine-tuned BERT model (CC BERT) on the test set (PR-AUC 0.3878 vs. 0.3597, p=7×10⁻⁵), though BERT models show greater robustness to misspellings and complex queries in qualitative analysis.

ABSTRACT

Automated classification of chief complaints from patient-generated text is a critical first step in developing scalable platforms to triage patients without human intervention. In this work, we evaluate several approaches to chief complaint classification using a novel Chief Complaint (CC) Dataset that contains ~200,000 patient-generated reasons-for-visit entries mapped to a set of 795 discrete chief complaints. We examine the use of several fine-tuned bidirectional transformer (BERT) models trained on both unrelated texts as well as on the CC dataset. We contrast this performance with a TF-IDF baseline. Our evaluation has three components: (1) a random test hold-out from the original dataset; (2) a "misspelling set," consisting of a hand-selected subset of the test set, where every entry has at least one misspelling; (3) a separate experimenter-generated free-text set. We find that the TF-IDF model performs significantly better than the strongest BERT-based model on the test (best BERT PR-AUC $0.3597 \pm 0.0041$ vs TF-IDF PR-AUC $0.3878 \pm 0.0148$, $p=7\cdot 10^{-5}$), and is statistically comparable to the misspelling sets (best BERT PR-AUC $0.2579 \pm 0.0079$ vs TF-IDF PR-AUC $0.2733 \pm 0.0130$, $p=0.06$). However, when examining model predictions on experimenter-generated queries, some concerns arise about TF-IDF baseline's robustness. Our results suggest that in certain tasks, simple language embedding baselines may be very performant; however, truly understanding their robustness requires further analysis.

Motivation & Objective

• To develop a new, large-scale dataset of patient-generated reasons-for-visit mapped to structured chief complaints for clinical NLP tasks.
• To evaluate the performance and robustness of pre-trained language models (BERT variants) and a TF-IDF baseline in extracting structured chief complaints from free-text patient input.
• To assess model behavior under distributional shifts, including misspellings and non-standard patient language.
• To investigate the impact of integrating demographic features (age, sex) with text embeddings for improved classification.

Proposed method

• Constructed a feed-forward neural network that combines pre-trained text embeddings (BERT, BioBERT, Clinical BERT, KP BERT, CC BERT) with learned embeddings for patient age (discretized) and biological sex.
• Trained models using 5-fold cross-validation on a dataset of 190,243 training encounters and 31,815 test encounters, with 795 unique chief complaints.
• Evaluated models on three test sets: a standard hold-out test set, a hand-curated misspelling set (245 entries with at least one misspelling), and a set of experimenter-generated free-text queries.
• Used micro-averaged PR-AUC and ROC-AUC as primary metrics, with Welch’s t-test for statistical significance.
• Applied stop-word removal and n-gram tokenization for the TF-IDF baseline, which was trained on the patient text corpus.
• Conducted qualitative analysis on model predictions for out-of-distribution queries to assess sensitivity to context, demographics, and spelling variations.

Experimental results

Research questions

• RQ1 Does a TF-IDF baseline outperform fine-tuned BERT models in extracting structured chief complaints from patient-generated text?
• RQ2 How do BERT-based models and TF-IDF perform under distributional shifts, such as misspellings or non-standard language?
• RQ3 To what extent do demographic features (age and sex) improve the performance of text embedding models in chief complaint classification?
• RQ4 Are BERT-based models more robust than TF-IDF to complex or colloquial patient language not seen in the training data?
• RQ5 What is the relative impact of model architecture and pre-training on robustness in low-resource clinical NLP scenarios?

Key findings

• The TF-IDF baseline achieved a micro-PR-AUC of 0.3878±0.0148 on the standard test set, significantly outperforming the best BERT model (CC BERT) with a PR-AUC of 0.3597±0.0041 (p=7×10⁻⁵).
• On the misspelled test set, the TF-IDF baseline achieved a PR-AUC of 0.2733±0.0130, compared to 0.2579±0.0079 for CC BERT, with no statistically significant difference (p=0.06).
• Despite superior performance on standard benchmarks, the TF-IDF model showed reduced robustness to out-of-distribution queries, such as colloquial or highly misspelled entries, compared to BERT-based models.
• BERT-based models, particularly KP BERT and CC BERT, demonstrated greater sensitivity to contextual cues and demographic information, correctly predicting different chief complaints for the same symptom based on patient age.
• The performance gap between TF-IDF and BERT models narrowed under distributional shift, suggesting that TF-IDF may be more sensitive to linguistic variation in real-world patient input.
• Qualitative analysis revealed that BERT models could correctly classify rare or informal expressions (e.g., 'my tum-tum is upset' → 'abdominal pain'), indicating better generalization to non-standard language.