[Paper Review] Towards Automated Anamnesis Summarization: BERT-based Models for Symptom Extraction

Anton Maximilian Schäfer, Nils Blach+3 more|arXiv (Cornell University)|Nov 3, 2020

Topic Modeling13 references4 citations

TL;DR

This paper proposes BERT-based models for automated symptom extraction from German patient monologues, using a novel dataset of 125 forum posts. It achieves SOTA performance with a F1 score of 0.903 on symptom classification using curriculum learning and data augmentation, demonstrating strong potential for practical clinical documentation support.

ABSTRACT

Professionals in modern healthcare systems are increasingly burdened by documentation workloads. Documentation of the initial patient anamnesis is particularly relevant, forming the basis of successful further diagnostic measures. However, manually prepared notes are inherently unstructured and often incomplete. In this paper, we investigate the potential of modern NLP techniques to support doctors in this matter. We present a dataset of German patient monologues, and formulate a well-defined information extraction task under the constraints of real-world utility and practicality. In addition, we propose BERT-based models in order to solve said task. We can demonstrate promising performance of the models in both symptom identification and symptom attribute extraction, significantly outperforming simpler baselines.

Motivation & Objective

• To address the challenge of incomplete and unstructured clinical documentation in German-speaking healthcare settings.
• To develop a practical, real-world applicable information extraction task for symptom and attribute extraction from patient monologues.
• To create a new, high-quality dataset of 125 German patient-written condition descriptions with structured symptom and attribute annotations.
• To evaluate BERT-based models for symptom classification and attribute extraction under realistic constraints of low-resource medical NLP.
• To improve generalization to unseen symptoms through a symptom query model and data augmentation techniques.

Proposed method

• Fine-tune a pre-trained German BERT model (bert-base-german-cased) on a symptom classification task with multi-label output.
• Propose a BERT Symptom Query (BERT-SQ) model that conditions on symptom descriptions to predict presence in patient text, enabling zero-shot generalization to unseen symptoms.
• Apply curriculum learning by ordering negative samples based on hierarchical symptom similarity to gradually increase difficulty.
• Implement data augmentation by generating synthetic descriptions of symptoms to improve model generalization.
• Use two attribute extraction strategies: start-end token prediction and contiguous token classification with a threshold of 0.7 for probability confidence.
• Apply heuristic post-processing to resolve overlapping or mismatched token spans in start-end prediction.

Experimental results

Research questions

• RQ1 Can BERT-based models effectively extract symptoms and their attributes from unstructured German patient monologues in a real-world clinical context?
• RQ2 How does curriculum learning improve model performance on symptom classification with hierarchical symptom structures?
• RQ3 To what extent does data augmentation with synthetic symptom descriptions enhance generalization to unseen symptoms?
• RQ4 How do different attribute extraction strategies (start-end vs. contiguous token prediction) compare in performance and robustness?
• RQ5 Can a symptom query model generalize effectively to symptoms not seen during training?

Key findings

• The BERT Symptom Query model with curriculum learning and data augmentation achieved a micro-F1 score of 0.903 on symptom classification, significantly outperforming baseline models.
• The symptom query model with data augmentation generalized better to unseen symptoms, as shown in ablation studies in Appendix D.
• The contiguous token prediction method achieved the highest F1 score of 0.66 for the location attribute, outperforming the start-end method (0.56).
• The general attribute model trained on all attributes simultaneously outperformed separate models for most attributes, with the highest F1 of 0.42 for frequency and 0.32 for action.
• The model achieved a precision of 0.956 and recall of 0.855 on the best-performing configuration, indicating high reliability in symptom detection.
• Despite strong performance on symptom classification, attribute extraction remains challenging, particularly for descriptive and temporal attributes, with F1 scores below 0.3 for some categories.