[Paper Review] INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis

Shih-Cheng Huang, Zepeng Huo+7 more|arXiv (Cornell University)|Nov 17, 2023

Venous Thromboembolism Diagnosis and Management7 citations

TL;DR

INSPECT introduces a large multimodal dataset (CTPAs, radiology reports, and longitudinal EHRs) for pulmonary embolism diagnosis and prognosis, and provides baseline models and a benchmark for multimodal fusion.

ABSTRACT

Synthesizing information from multiple data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE), along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications). Using INSPECT, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and multimodal fusion models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best of our knowledge, INSPECT is the largest multimodal dataset integrating 3D medical imaging and EHR for reproducible methods evaluation and research.

Motivation & Objective

• Create a large-scale, publicly available multimodal PE dataset combining 3D CT images, radiology report impressions, and longitudinal EHR data.
• Define and label eight PE-related diagnostic and prognostic tasks to benchmark multimodal learning.
• Evaluate imaging-only, EHR-only, and multimodal fusion models on these tasks to establish baselines.
• Provide open-source baselines and trained models to foster reproducible, multimodal PE research.

Proposed method

• Assemble a de-identified cohort of 23,248 CTPA studies from 19,402 patients with paired DICOM CT data, radiology report impressions, and structured EHRs.
• Process CTPA images by HU rescaling, three-view windowing (lung, PE, mediastinum) into 224x224x3 slices, and use a pretrained image encoder plus a sequence model (LSTM/GRU/Transformer).
• Featurize EHR data with count-based features for gradient-boosted trees (LightGBM) and adapt MOTOR (a time-to-event pretrained EHR model) with a linear probe for prognostic tasks.
• Define eight diagnostic/prognostic tasks (PE diagnosis via NLP-impression, and prognoses including PH, in-hospital mortality, readmission, etc.) with censoring for missing future events.
• Implement baseline fusion as late fusion via a learned logistic regression combining modality-specific predictions.
• Release code and trained model weights under a data use agreement to enable reproducible benchmarking.

Experimental results

Research questions

• RQ1 Can multimodal models combining CT imagery, radiology impressions, and longitudinal EHR data outperform single-modality models on PE diagnosis and prognosis tasks?
• RQ2 Which modalities dominate for diagnostic versus prognostic tasks, and how does late fusion affect performance?
• RQ3 How well do pretrained EHR foundation models (MOTOR) transfer to PE prognostication when combined with imaging data?
• RQ4 What are the limitations and generalizability considerations when training on a single-site dataset for multimodal PE prediction?

Key findings

• Imaging-only models excel at PE diagnostic tasks, while structured EHR models perform better on prognostic tasks.
• Fusion of CT with EHR models improves diagnostic performance but does not consistently improve prognostic tasks.
• Best-performing configurations achieve higher AUROC on diagnostic tasks when combining CT with EHR, compared to single modalities.
• Model weights and trained baselines are released to support reproducibility and further multimodal PE research.
• INSPECT represents the largest publicly available multimodal 3D medical imaging and longitudinal EHR dataset to date for reproducible benchmarking.