[Paper Review] Multi-modal AI for comprehensive breast cancer prognostication

Jan Witowski, Ke‐Wu Zeng+49 more|arXiv (Cornell University)|Oct 28, 2024

AI in cancer detection4 citations

TL;DR

This study introduces a multi-modal AI model that integrates digital pathology images and clinical data to improve breast cancer recurrence prediction. Using a vision transformer-based foundation model on whole-slide images and clinical variables, the AI achieved a C-index of 0.71 in external validation, outperforming Oncotype DX (C-index 0.61) and adding independent prognostic value across all major subtypes, including TNBC.

ABSTRACT

Treatment selection in breast cancer is guided by molecular subtypes and clinical characteristics. However, current tools including genomic assays lack the accuracy required for optimal clinical decision-making. We developed a novel artificial intelligence (AI)-based approach that integrates digital pathology images with clinical data, providing a more robust and effective method for predicting the risk of cancer recurrence in breast cancer patients. Specifically, we utilized a vision transformer pan-cancer foundation model trained with self-supervised learning to extract features from digitized H&E-stained slides. These features were integrated with clinical data to form a multi-modal AI test predicting cancer recurrence and death. The test was developed and evaluated using data from a total of 8,161 female breast cancer patients across 15 cohorts originating from seven countries. Of these, 3,502 patients from five cohorts were used exclusively for evaluation, while the remaining patients were used for training. Our test accurately predicted our primary endpoint, disease-free interval, in the five evaluation cohorts (C-index: 0.71 [0.68-0.75], HR: 3.63 [3.02-4.37, p<0.001]). In a direct comparison (n=858), the AI test was more accurate than Oncotype DX, the standard-of-care 21-gene assay, achieving a C-index of 0.67 [0.61-0.74] versus 0.61 [0.49-0.73], respectively. Additionally, the AI test added independent prognostic information to Oncotype DX in a multivariate analysis (HR: 3.11 [1.91-5.09, p<0.001)]). The test demonstrated robust accuracy across major molecular breast cancer subtypes, including TNBC (C-index: 0.71 [0.62-0.81], HR: 3.81 [2.35-6.17, p=0.02]), where no diagnostic tools are currently recommended by clinical guidelines. These results suggest that our AI test improves upon the accuracy of existing prognostic tests, while being applicable to a wider range of patients.

Motivation & Objective

• To develop a comprehensive, multi-modal AI system for breast cancer prognostication that integrates digital pathology and clinical data.
• To overcome limitations of current genomic assays, which are restricted to HR+ patients and have modest accuracy.
• To improve recurrence risk prediction across all breast cancer subtypes, including TNBC, where no standard diagnostic tools exist.
• To create a scalable, accessible tool that enhances personalized treatment decisions by integrating routinely available clinical and histopathological data.
• To validate the model’s robustness and generalizability across diverse, multi-national patient cohorts with independent external validation.

Proposed method

• A vision transformer-based foundation model, Kestrel, was pre-trained via self-supervised learning on 400 million pathology image patches from a pan-cancer dataset.
• Whole-slide images were embedded using Kestrel, and patch-level features were aggregated via mean/max pooling and gated attention-based multiple instance learning (MIL) for time-to-event modeling.
• Time-to-event prediction was performed using Cox proportional hazards and discrete-time survival models, with L1/L2-regularized loss functions to prevent overfitting.
• Clinical variables (e.g., age, ER/PR/HER2 status, T/N stage) were processed using CatBoost with AFT loss and three distributional assumptions (normal, logistic, extreme value).
• Pathology and clinical embeddings were concatenated and fed into a final multi-modal risk score model trained on 4,659 patients from 10 cohorts across six countries.
• Model performance was evaluated on 3,502 patients from five independent cohorts, including both HR+ and TNBC subtypes, with external validation and direct comparison to Oncotype DX.

Experimental results

Research questions

• RQ1 Can a multi-modal AI model combining digital pathology and clinical data improve recurrence risk prediction in breast cancer compared to current standard-of-care genomic assays?
• RQ2 Does the AI model provide independent prognostic value beyond established clinical and genomic factors, including Oncotype DX?
• RQ3 Can the model generalize across diverse populations and breast cancer subtypes, including TNBC, where no standard prognostic tools exist?
• RQ4 How does the performance of the AI model compare to Oncotype DX in terms of discrimination (C-index) and hazard ratio for recurrence?
• RQ5 Can self-supervised pre-training on a large-scale pathology dataset improve feature extraction and downstream prognostic performance?

Key findings

• The multi-modal AI model achieved a C-index of 0.71 (95% CI: 0.68–0.75) for disease-free interval prediction in five external validation cohorts, demonstrating strong discriminatory performance.
• In a direct comparison with Oncotype DX (N=858), the AI model outperformed the standard-of-care assay, with a C-index of 0.67 (0.61–0.74) versus 0.61 (0.49–0.73), respectively.
• The AI model added independent prognostic information beyond Oncotype DX in multivariate analysis, with a hazard ratio of 3.11 (95% CI: 1.91–5.09, p<0.01) for high-risk vs. low-risk patients.
• The model maintained high performance across all major subtypes, including TNBC, with a C-index of 0.71 (0.62–0.81) and a hazard ratio of 3.81 (2.35–6.17, p=0.02) for recurrence.
• The model demonstrated consistent performance across multiple datasets, including those from the U.S., Europe, and Asia, confirming its robustness and generalizability.
• The integration of self-supervised pathology representation learning with clinical data significantly enhanced risk stratification, particularly in underserved subtypes like TNBC.