[Paper Review] The Impact of Automatic Pre-annotation in Clinical Note Data Element Extraction - the CLEAN Tool

Tsung-Ting Kuo, Jina Huh+9 more|arXiv (Cornell University)|Aug 11, 2018

Biomedical Text Mining and Ontologies24 references3 citations

TL;DR

This paper introduces CLEAN, a pre-annotation-based clinical note annotation system that improves accuracy in extracting data elements from clinical notes. Using an ensemble pipeline (CLEAN-EP) and a custom annotation tool (CLEAN-AT), CLEAN achieved significantly higher F1-scores (0.896 vs. 0.820) than BRAT with no significant difference in annotation time, demonstrating enhanced correctness and user satisfaction without sacrificing efficiency.

ABSTRACT

Objective. Annotation is expensive but essential for clinical note review and clinical natural language processing (cNLP). However, the extent to which computer-generated pre-annotation is beneficial to human annotation is still an open question. Our study introduces CLEAN (CLinical note rEview and ANnotation), a pre-annotation-based cNLP annotation system to improve clinical note annotation of data elements, and comprehensively compares CLEAN with the widely-used annotation system Brat Rapid Annotation Tool (BRAT). Materials and Methods. CLEAN includes an ensemble pipeline (CLEAN-EP) with a newly developed annotation tool (CLEAN-AT). A domain expert and a novice user/annotator participated in a comparative usability test by tagging 87 data elements related to Congestive Heart Failure (CHF) and Kawasaki Disease (KD) cohorts in 84 public notes. Results. CLEAN achieved higher note-level F1-score (0.896) over BRAT (0.820), with significant difference in correctness (P-value < 0.001), and the mostly related factor being system/software (P-value < 0.001). No significant difference (P-value 0.188) in annotation time was observed between CLEAN (7.262 minutes/note) and BRAT (8.286 minutes/note). The difference was mostly associated with note length (P-value < 0.001) and system/software (P-value 0.013). The expert reported CLEAN to be useful/satisfactory, while the novice reported slight improvements. Discussion. CLEAN improves the correctness of annotation and increases usefulness/satisfaction with the same level of efficiency. Limitations include untested impact of pre-annotation correctness rate, small sample size, small user size, and restrictedly validated gold standard. Conclusion. CLEAN with pre-annotation can be beneficial for an expert to deal with complex annotation tasks involving numerous and diverse target data elements.

Motivation & Objective

• To evaluate the impact of automatic pre-annotation on clinical note data element extraction accuracy and efficiency.
• To develop and test a new annotation system, CLEAN, that integrates pre-annotation to support human annotators.
• To compare CLEAN’s performance and usability against the widely used BRAT annotation tool.
• To assess whether pre-annotation improves annotation correctness and user satisfaction without increasing time burden.
• To investigate the role of system design and user expertise in annotation outcomes.

Proposed method

• CLEAN employs an ensemble pipeline (CLEAN-EP) combining multiple NLP models to generate pre-annotations for clinical notes.
• A custom annotation tool (CLEAN-AT) presents pre-annotations to human annotators, allowing them to review and correct them efficiently.
• The system was evaluated using 84 public clinical notes from CHF and KD cohorts, with 87 data elements to be extracted.
• A comparative usability test was conducted with one domain expert and one novice annotator using both CLEAN and BRAT.
• Performance was measured via note-level F1-score, annotation time, and user-reported satisfaction.
• Statistical analysis assessed the impact of system type, note length, and user expertise on correctness and time.

Experimental results

Research questions

• RQ1 Does automatic pre-annotation improve the correctness of human annotation in clinical note data element extraction?
• RQ2 How does the CLEAN system compare to BRAT in terms of annotation time and efficiency?
• RQ3 To what extent does pre-annotation enhance user satisfaction and perceived usefulness?
• RQ4 Is the performance gain from pre-annotation dependent on user expertise or note complexity?
• RQ5 What factors—system design, note length, or user type—most significantly influence annotation correctness and time?

Key findings

• CLEAN achieved a note-level F1-score of 0.896, significantly higher than BRAT’s 0.820 (p < 0.001), indicating improved annotation correctness.
• The difference in correctness was primarily attributed to the system/software used (p < 0.001), not user expertise.
• Average annotation time was 7.262 minutes per note for CLEAN and 8.286 minutes for BRAT, with no significant difference (p = 0.188).
• Note length was the most significant factor affecting annotation time (p < 0.001), with system type also contributing (p = 0.013).
• The expert annotator reported CLEAN to be useful and satisfactory, while the novice reported slight improvements in usability.
• The results suggest that pre-annotation enhances accuracy and usability without increasing time cost, especially for complex annotation tasks.