[Paper Review] Automated Scoring of Clinical Patient Notes using Advanced NLP and Pseudo Labeling

Jingyu Xu, Yifeng Jiang+3 more|arXiv (Cornell University)|Jan 18, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes an automated scoring system for clinical patient notes using DeBERTa-v3-large fine-tuned with Masked Language Modeling (MLM) pretraining and pseudo-labeling, achieving a micro-F1 score of 0.8911. The method improves model generalization and reduces training time through data augmentation and optimized inference, offering a scalable solution for medical education and certification assessment.

ABSTRACT

Clinical patient notes are critical for documenting patient interactions, diagnoses, and treatment plans in medical practice. Ensuring accurate evaluation of these notes is essential for medical education and certification. However, manual evaluation is complex and time-consuming, often resulting in variability and resource-intensive assessments. To tackle these challenges, this research introduces an approach leveraging state-of-the-art Natural Language Processing (NLP) techniques, specifically Masked Language Modeling (MLM) pretraining, and pseudo labeling. Our methodology enhances efficiency and effectiveness, significantly reducing training time without compromising performance. Experimental results showcase improved model performance, indicating a potential transformation in clinical note assessment.

Motivation & Objective

• To address the time-consuming and variable nature of manual scoring of clinical patient notes in medical education and certification.
• To improve model performance and generalization in clinical note scoring using advanced NLP techniques.
• To reduce training and inference time through optimization strategies like dynamic batching and padding reduction.
• To evaluate the effectiveness of pseudo-labeling and MLM pretraining in enhancing model robustness on clinical text.
• To provide a scalable, efficient, and accurate automated solution for clinical note assessment in healthcare settings.

Proposed method

• Fine-tuned DeBERTa-v3-large using Masked Language Modeling (MLM) pretraining with a masking probability of 0.15 to enhance contextual representation of clinical text.
• Applied pseudo-labeling to generate additional training examples from unlabeled data, improving model generalization and performance.
• Employed dynamic batching and bucketing techniques to minimize padding and reduce computational cost during training and inference.
• Optimized inference by reducing the effective sequence length through intelligent padding strategies, cutting inference time from 97 to 56 minutes.
• Used Binary Cross-Entropy with Logits Loss (BCEWithLogitsLoss) to train the model for binary classification of clinical note spans, with sigmoid activation on logits.
• Evaluated performance using micro-averaged F1 score, balancing precision and recall across all classes in the classification task.

Experimental results

Research questions

• RQ1 Can MLM pretraining improve the performance of NLP models in the specialized domain of clinical note scoring?
• RQ2 To what extent does pseudo-labeling enhance model generalization and performance on limited annotated clinical data?
• RQ3 How effective are padding optimization techniques in reducing inference time without sacrificing model accuracy?
• RQ4 What is the impact of combining MLM pretraining and pseudo-labeling on model convergence and final performance?
• RQ5 Can training acceleration techniques make automated clinical note scoring more practical for real-world medical education applications?

Key findings

• The DeBERTa-v3-large model achieved a cross-validation F1 score of 0.8911 when using both MLM pretraining and pseudo-labeling, outperforming baseline and individual components.
• Pseudo-labeling improved model performance by 0.0032 F1 points over the baseline, demonstrating its value in data augmentation for clinical NLP.
• The combination of MLM pretraining and pseudo-labeling yielded the highest performance, indicating synergistic benefits in model learning.
• Padding optimization reduced inference time from 97 minutes to 56 minutes, a 42% improvement, enhancing real-time deployment feasibility.
• The DeBERTa-v3-large model achieved the highest F1 score (0.8911) among all tested models, including deberta_v2_xlarge (0.8886) and deberta_v2_xxlarge (0.8904).
• The use of BCEWithLogitsLoss with sigmoid activation enabled effective learning for span-based classification, contributing to stable and accurate predictions.