[Paper Review] Chest ImaGenome Dataset for Clinical Reasoning

Joy T. Wu, Nkechinyere N. Agu+10 more|arXiv (Cornell University)|Jul 31, 2021

COVID-19 diagnosis using AIMedicine48 references26 citations

TL;DR

This paper introduces Chest ImaGenome, a large multi-modal chest X-ray scene-graph dataset that automatically links anatomical locations with attributes and sequential comparison relations, enabling localized explainability and longitudinal clinical reasoning research. It builds on MIMIC-CXR to deliver 242,072 scene graphs with 1,256 relation types and a gold-standard subset for benchmarking.

ABSTRACT

Despite the progress in automatic detection of radiologic findings from chest X-ray (CXR) images in recent years, a quantitative evaluation of the explainability of these models is hampered by the lack of locally labeled datasets for different findings. With the exception of a few expert-labeled small-scale datasets for specific findings, such as pneumonia and pneumothorax, most of the CXR deep learning models to date are trained on global "weak" labels extracted from text reports, or trained via a joint image and unstructured text learning strategy. Inspired by the Visual Genome effort in the computer vision community, we constructed the first Chest ImaGenome dataset with a scene graph data structure to describe $242,072$ images. Local annotations are automatically produced using a joint rule-based natural language processing (NLP) and atlas-based bounding box detection pipeline. Through a radiologist constructed CXR ontology, the annotations for each CXR are connected as an anatomy-centered scene graph, useful for image-level reasoning and multimodal fusion applications. Overall, we provide: i) $1,256$ combinations of relation annotations between $29$ CXR anatomical locations (objects with bounding box coordinates) and their attributes, structured as a scene graph per image, ii) over $670,000$ localized comparison relations (for improved, worsened, or no change) between the anatomical locations across sequential exams, as well as ii) a manually annotated gold standard scene graph dataset from $500$ unique patients.

Motivation & Objective

• Create a large, locally labeled CXR dataset that mirrors radiologists’ reasoning by linking anatomical locations to findings and other attributes.
• Automatically extract scene graphs from CXR reports using NLP and an atlas-based bounding box pipeline.
• Provide sequential comparison relations between exams to capture disease progression over time.
• Map all objects, attributes, and relations to UMLS CUIs to enable interoperable, multi-modal reasoning.
• Offer gold-standard annotations for benchmarking and support downstream clinical and research applications.

Proposed method

• Derive annotations from MIMIC-CXR by atlas-based extraction of bounding boxes for 29 anatomical locations.
• Use a rule-based NLP pipeline and radiologist-curated CXR ontology to extract 1,256 object-attribute relation types across 242,072 images.
• Construct scene graphs per image with nodes (objects), attributes, and relationships (including sequential comparison relations).
• Automatically link nodes and edges to UMLS CUIs (radiology-focused RadLex/UMSL concepts) for interoperability.
• Create a gold-standard subset of 500 patients with manual validation for object-attribute and object-object relations and bounding boxes.
• Provide post-processing utilities to render scene graphs at study level and in RDF-like formats for visualization.

Experimental results

Research questions

• RQ1 Can localized anatomical-attribute relations and sequential comparison relations be automatically extracted from CXR reports and images at scale?
• RQ2 How well do automated extractions align with radiologist annotations (gold standard) for object-attribute and object-object relations?
• RQ3 Are the extracted scene graphs usable for longitudinal clinical reasoning and multi-modal fusion tasks?
• RQ4 What is the coverage and accuracy of bounding box localization for key anatomical structures on CXRs?
• RQ5 How can the dataset be leveraged for downstream tasks such as change detection across sequential CXRs?

Key findings

• Automated construction of 242,072 scene graphs from 217,013 unique CXR studies with 1,256 relation types and 29 anatomical locations.
• Over 678,543 object-object comparison relations extracted across 128,468 sequential CXR pairs.
• Gold-standard subset (500 patients) with manual validation shows strong precision/recall metrics for object-attribute and comparison relation extraction, with noted challenges in laterality and cross-sentence information.
• Bounding box pipeline achieves high coverage with up to 29 anatomies detectable per image and low miss rates (<0.02% for most objects).
• Dataset is mapped to UMLS CUIs, enabling cross-domain integration with other clinical data sources.
• Two provided example tasks highlight clinical utility: locating attribute types per image and determining change across sequential exams.