[Paper Review] Pathology Extraction from Chest X-Ray Radiology Reports: A Performance Study

Tahsin Mostafiz, Khalid Ashraf|arXiv (Cornell University)|Dec 6, 2018

Topic Modeling11 references4 citations

TL;DR

This study evaluates pathology extraction from chest X-ray radiology reports using medical-specific APIs (MTI, MOD) and a generic NLU API (IBM NLU), finding them underperforming. It proposes a custom DNN-based Named Entity Recognition (NER) model fine-tuned on a task-specific radiology corpus, which achieves significantly higher F1 scores (49.91%) for general pathology extraction and 82.53% for opacity—surpassing all other tools, demonstrating the superiority of domain-specific model training over off-the-shelf solutions.

ABSTRACT

Extraction of relevant pathological terms from radiology reports is important for correct image label generation and disease population studies. In this letter, we compare the performance of some known application program interface (APIs) for the task of thoracic abnormality extraction from radiology reports. We explored several medical domain specific annotation tools like Medical Text Indexer(MTI) with Non-MEDLINE and Mesh On Demand(MOD) options and generic Natural Language Understanding (NLU) API provided by the IBM cloud. Our results show that although MTI and MOD are intended for extracting medical terms, their performance is worst compared to generic extraction API like IBM NLU. Finally, we trained a DNN-based Named Entity Recognition (NER) model to extract the key concept words from radiology reports. Our model outperforms the medical specific and generic API performance by a large margin. Our results demonstrate the inadequacy of generic APIs for pathology extraction task and establish the importance of domain specific model training for improved results. We hope that these results motivate the research community to release larger de-identified radiology reports corpus for building high accuracy machine learning models for the important task of pathology extraction.

Motivation & Objective

• To evaluate the performance of existing public APIs—medical-specific (MTI, MOD) and generic (IBM NLU)—for extracting pathological terms from chest X-ray radiology reports.
• To identify limitations in current medical annotation tools, particularly their tendency to generate false positives and misclassify terms.
• To develop and validate a deep neural network-based Named Entity Recognition (NER) model trained specifically on chest X-ray report data for improved pathology extraction accuracy.
• To demonstrate that task-specific model training yields significantly better results than off-the-shelf APIs, even when those APIs are designed for medical text.
• To advocate for the release of larger, de-identified radiology report corpora to enable high-accuracy machine learning models for medical NLP tasks.

Proposed method

• The study uses the publicly available Indiana chest X-ray radiology report dataset as the benchmark corpus.
• It evaluates three pre-existing APIs: MTI with Non-MEDLINE and Mesh On Demand (MOD) options, and IBM's generic NLU API.
• A custom DNN-based NER model is trained end-to-end on the same dataset using word embeddings and sequence labeling techniques to identify pathological terms.
• The model is optimized for key findings in radiology reports, including handling negation (e.g., 'no opacity') and distinguishing similar medical terms.
• Evaluation metrics include precision, recall, and F1-score, calculated per pathology class (e.g., opacity, cardiomegaly) using manually annotated ground truth.
• Qualitative analysis is performed to inspect model behavior, such as detecting negation and handling ambiguous or rare terms.

Experimental results

Research questions

• RQ1 How do medical-specific annotation tools (MTI, MOD) and a generic NLU API (IBM NLU) perform in extracting pathological terms from chest X-ray radiology reports?
• RQ2 What are the failure modes of existing medical APIs when applied to radiology reports, particularly in terms of false positives and incorrect term mapping?
• RQ3 Can a DNN-based NER model fine-tuned on a task-specific radiology report corpus outperform both medical-specific and generic APIs in pathology extraction?
• RQ4 To what extent does domain-specific model training improve F1 scores compared to off-the-shelf solutions for this task?
• RQ5 What are the key qualitative advantages of a custom NER model in understanding clinical context, such as negation or semantic similarity?

Key findings

• The DNN-based NER model achieved an F1 score of 49.91% for general pathology extraction, significantly outperforming IBM NLU (26.47%) and MTI with MOD (15.42%).
• For the extraction of 'opacity', the NER model achieved 82.53% F1, compared to only 12.61% for IBM NLU and 0.00% for both MTI configurations.
• The NER model achieved 90.90% precision and 28.57% recall for 'cardiomegaly', outperforming IBM NLU (4.54% F1) and MTI with Non-MEDLINE (10.30% F1), though MTI with MOD achieved higher recall (62.85%) for this term.
• MTI tools with both Non-MEDLINE and MOD options failed entirely to extract 'opacity', yielding 0.00% precision and recall, indicating severe limitations in handling certain radiological terms.
• The NER model successfully learned to detect negated terms (e.g., 'no opacity') and distinguish semantically similar terms, which the generic and medical-specific APIs failed to do consistently.
• The study reveals that existing medical APIs like MTI and MOD often introduce terms not present in the original reports—such as 'CABG' being misclassified as 'nitro compounds hydrogen-ion'—undermining reliability.