[Paper Review] Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models

Andriy Mulyar, Elliot Schumacher+2 more|arXiv (Cornell University)|Oct 30, 2019

Topic Modeling18 references21 citations

TL;DR

This paper proposes a BERT-based document classification framework for phenotyping clinical notes, using mean pooling of contextualized BERT representations across segmented note chunks to generate document-level embeddings. It achieves state-of-the-art performance on the I2B2 2006 smoking status phenotyping task, outperforming prior rule-based systems by 2.8% in micro-F1, demonstrating that simple pooling strategies suffice over complex architectures when using contextualized language models.

ABSTRACT

Clinical notes contain an extensive record of a patient's health status, such as smoking status or the presence of heart conditions. However, this detail is not replicated within the structured data of electronic health systems. Phenotyping, the extraction of patient conditions from free clinical text, is a critical task which supports avariety of downstream applications such as decision support and secondary use of medical records. Previous work has resulted in systems which are high performing but require hand engineering, often of rules. Recent work in pretrained contextualized language models have enabled advances in representing text for a variety of tasks. We therefore explore several architectures for modeling pheno-typing that rely solely on BERT representations of the clinical note, removing the need for manual engineering. We find these architectures are competitive with or outperform existing state of the art methods on two phenotyping tasks.

Motivation & Objective

• To develop a generalizable, end-to-end document classification framework for phenotyping clinical notes using contextualized neural language models.
• To eliminate reliance on manual feature engineering, such as negation detection or temporal phrase recognition, by leveraging pre-trained BERT representations.
• To evaluate whether simple aggregation methods like mean pooling can outperform complex architectures in long-document phenotyping tasks.
• To assess the performance of BERT-based models on two established phenotyping benchmarks: smoking status and obesity-related comorbidities.
• To provide a publicly available PyTorch implementation for reproducibility and broader adoption in clinical NLP.

Proposed method

• The clinical note is tokenized and split into non-overlapping segments of length 512 (BERT's maximum input length), using WordPiece tokenization.
• Each segment is encoded using a clinically fine-tuned BERT model, producing contextualized hidden states for every subword token.
• Document-level representations are constructed using four aggregation strategies: mean pooling, max pooling, LSTM-based encoding, and multi-head self-attention (Transformer-based pooling).
• The pooled representation is passed through a feedforward classifier with sigmoid activation for binary or multi-label classification of phenotypic traits.
• The model is trained end-to-end on the I2B2 2006 (smoking) and I2B2 2008 (obesity) shared task datasets using 1000 epochs, with no hyperparameter tuning due to lack of development sets.
• The approach avoids rule-based preprocessing and instead relies solely on contextual embeddings, making it adaptable to diverse phenotypic traits without task-specific engineering.

Experimental results

Research questions

• RQ1 Can a BERT-based document classification model achieve state-of-the-art performance on clinical phenotyping tasks without handcrafted features or rule-based components?
• RQ2 Does mean pooling of BERT representations across document segments yield better performance than more complex aggregation architectures like LSTM or multi-head attention?
• RQ3 How do BERT-based models compare to prior rule-based and CNN-based systems on established phenotyping benchmarks?
• RQ4 To what extent does the model’s performance degrade when relying only on the first 510 tokens, as in standard BERT applications?
• RQ5 Is the contextualized representation from BERT sufficient to capture phenotypic signals even when combined with simple, non-learned pooling strategies?

Key findings

• The mean pooling strategy ($f_{\text{mean}}$) achieved a micro-F1 of 92.8 on the I2B2 2006 smoking status phenotyping task, outperforming the previous shared task winner (90.0) by 2.8 percentage points.
• The $f_{\text{mean}}$ architecture also achieved the highest F1 on the I2B2 2008 obesity comorbidities task (86.5), outperforming the shared task first-place system (95.0) only on the majority label baseline.
• The Transformer-based pooling method ($f_{\text{Transformer}}$) underperformed all baselines, achieving only 58.4 F1 on smoking and 70.4 on obesity, suggesting that increased model capacity may not benefit low-data settings.
• The LSTM-based pooling method ($f_{I}$) achieved 92.3 F1 on smoking and 83.1 on obesity, performing competitively but not surpassing mean pooling.
• The baseline DocBert model, which uses only the first 510 tokens, achieved only 80.2 F1 on smoking and 67.6 on obesity, confirming that it fails to capture critical information in longer clinical notes.
• The study demonstrates that simple mean pooling of BERT representations is sufficient for high performance in clinical document classification, negating the need for complex, task-specific architectures.