[Paper Review] Validating transformers for redaction of text from electronic health records in real-world healthcare

Željko Kraljević, Anthony Shek+9 more|arXiv (Cornell University)|Oct 5, 2023

Electronic Health Records Systems4 citations

TL;DR

This paper proposes AnonCAT, a fine-tuned transformer-based model for de-identification of electronic health records (EHRs) in real-world healthcare settings. Trained on 3,116 manually annotated EHRs from three UK hospitals with diverse EHR systems, AnonCAT achieved high recall (0.99, 0.99, and 0.96) across all sites, demonstrating that localized, continual fine-tuning of foundation models enables accurate, adaptable, and maintainable redaction of protected health information.

ABSTRACT

Protecting patient privacy in healthcare records is a top priority, and redaction is a commonly used method for obscuring directly identifiable information in text. Rule-based methods have been widely used, but their precision is often low causing over-redaction of text and frequently not being adaptable enough for non-standardised or unconventional structures of personal health information. Deep learning techniques have emerged as a promising solution, but implementing them in real-world environments poses challenges due to the differences in patient record structure and language across different departments, hospitals, and countries. In this study, we present AnonCAT, a transformer-based model and a blueprint on how deidentification models can be deployed in real-world healthcare. AnonCAT was trained through a process involving manually annotated redactions of real-world documents from three UK hospitals with different electronic health record systems and 3116 documents. The model achieved high performance in all three hospitals with a Recall of 0.99, 0.99 and 0.96. Our findings demonstrate the potential of deep learning techniques for improving the efficiency and accuracy of redaction in global healthcare data and highlight the importance of building workflows which not just use these models but are also able to continually fine-tune and audit the performance of these algorithms to ensure continuing effectiveness in real-world settings. This approach provides a blueprint for the real-world use of de-identifying algorithms through fine-tuning and localisation, the code together with tutorials is available on GitHub (https://github.com/CogStack/MedCAT).

Motivation & Objective

• To address the limitations of rule-based redaction methods in handling non-standardized, diverse, and evolving EHR structures across healthcare institutions.
• To evaluate the feasibility and performance of deep learning models, specifically transformers, for de-identification in real-world clinical environments with heterogeneous EHR systems.
• To develop a practical, scalable workflow for deploying and maintaining NLP models in healthcare by enabling local fine-tuning and continuous performance auditing.
• To reduce over-redaction of clinically relevant information while maintaining high recall in de-identification tasks.
• To integrate model deployment into existing hospital information governance workflows to minimize additional burden and support continuous model improvement.

Proposed method

• AnonCAT is built on MedCAT, a medical NLP toolkit for Named Entity Recognition and Linking (NER+L), adapted for de-identification by replacing or removing detected PHI entities.
• The model was fine-tuned using 3,116 real-world EHR documents from three UK hospitals with distinct EHR systems, using manual annotations of 14 PHI types including names, addresses, and NHS numbers.
• A multi-step annotation and validation process was used, where disagreements between model predictions and human annotators were iteratively resolved, focusing on missed PHI instances.
• Performance was evaluated using standard NLP metrics: Precision, Recall, and F1-score, with entity-level and merged metrics reported across different PHI categories.
• The model was tested in zero-shot and few-shot settings—evaluating performance on a new hospital dataset without fine-tuning (KCH model on GSTT data) and after fine-tuning with 150–300 documents.
• A biasing strategy was explored to balance precision and recall, with trade-off analysis conducted on the GSTT dataset using two models: AnonCAT trained only on KCH data and Philter as a baseline.

Figure 1: Sunburst hierarchical ontology structure of terms for redaction. Five broad terms serve as the primary child nodes to which more specific terms child nodes, e.g. postcode, were connected. The terminal leaf nodes were used to annotate documents with patient data terms. Non-healthcare identi

Experimental results

Research questions

• RQ1 Can a transformer-based model achieve high recall and precision in de-identifying EHRs across diverse hospital EHR systems with minimal domain-specific data?
• RQ2 How effective is few-shot fine-tuning (150–300 documents) in adapting a pre-trained foundation model to a new healthcare institution’s EHR structure and language patterns?
• RQ3 What is the impact of data drift and structural variation across institutions on the performance of pre-trained de-identification models, and how can this be mitigated?
• RQ4 Is an iterative annotation process that corrects model errors more efficient than double annotation for creating high-quality PHI-labeled datasets in complex clinical text?
• RQ5 To what extent can existing hospital information governance workflows be adapted to support continuous model auditing and fine-tuning?

Key findings

• AnonCAT achieved a recall of 0.99, 0.99, and 0.96 on three distinct hospital datasets, demonstrating high effectiveness across diverse EHR systems and language structures.
• Fine-tuning with as few as 150–300 manually annotated documents significantly improved model performance on new datasets, with higher performance observed as the number of fine-tuning documents increased.
• The majority of annotation errors (90%) were due to human annotators missing PHI, not mislabeling, indicating that model-assisted iterative validation is more efficient than double annotation for difficult detection tasks.
• Even state-of-the-art off-the-shelf tools like Philter required fine-tuning to achieve a recall of 95% or higher on new datasets, underscoring the necessity of local adaptation.
• The model’s performance was robust across different PHI types, with high F1-scores for critical entities such as NHS numbers (F1 = 0.83) and dates (F1 = 0.83), indicating strong generalization.
• Biasing the model toward higher recall improved overall performance, but at the cost of reduced precision, highlighting the need for careful trade-off management in clinical deployment.

Figure 2: The experimental setup across sites for AnonCAT.