[Paper Review] Interpretable histopathology-based prediction of disease relevant features in Inflammatory Bowel Disease biopsies using weakly-supervised deep learning

Ricardo Mokhtari, Azam Hamidinekoo+13 more|arXiv (Cornell University)|Mar 20, 2023

Tuberculosis Research and EpidemiologyMedicine3 citations

TL;DR

This study develops interpretable, weakly-supervised deep learning models that predict disease-relevant histopathological features in Inflammatory Bowel Disease (IBD) biopsies using only endoscopic labels. By leveraging self-supervised pre-training and attention maps, the models achieve high AUCs (up to 0.87) in classifying Crohn’s disease vs. ulcerative colitis and correlate strongly with pathologist-verified inflammation, enabling automated prioritization of severe cases and enhanced clinical trial patient stratification.

ABSTRACT

Crohn's Disease (CD) and Ulcerative Colitis (UC) are the two main Inflammatory Bowel Disease (IBD) types. We developed deep learning models to identify histological disease features for both CD and UC using only endoscopic labels. We explored fine-tuning and end-to-end training of two state-of-the-art self-supervised models for predicting three different endoscopic categories (i) CD vs UC (AUC=0.87), (ii) normal vs lesional (AUC=0.81), (iii) low vs high disease severity score (AUC=0.80). We produced visual attention maps to interpret what the models learned and validated them with the support of a pathologist, where we observed a strong association between the models' predictions and histopathological inflammatory features of the disease. Additionally, we identified several cases where the model incorrectly predicted normal samples as lesional but were correct on the microscopic level when reviewed by the pathologist. This tendency of histological presentation to be more severe than endoscopic presentation was previously published in the literature. In parallel, we utilised a model trained on the Colon Nuclei Identification and Counting (CoNIC) dataset to predict and explore 6 cell populations. We observed correlation between areas enriched with the predicted immune cells in biopsies and the pathologist's feedback on the attention maps. Finally, we identified several cell level features indicative of disease severity in CD and UC. These models can enhance our understanding about the pathology behind IBD and can shape our strategies for patient stratification in clinical trials.

Motivation & Objective

• To develop interpretable deep learning models that predict histopathological features in IBD biopsies using only endoscopic labels, avoiding reliance on costly whole-slide image annotations.
• To investigate whether weakly-supervised learning can effectively link endoscopic appearance to underlying histological features in Crohn’s disease and ulcerative colitis.
• To validate model predictions through collaboration with pathologists, using attention maps to localize inflammation and correlate with expert histological assessments.
• To explore the utility of a pre-trained CoNIC model for cell-type prediction and assess consistency with attention maps and clinical relevance.
• To enable faster, more efficient pathologist workflows by ranking biopsies by predicted severity and highlighting diseased regions via attention maps.

Proposed method

• Fine-tuned and end-to-end trained two state-of-the-art self-supervised models—DSMIL and HIPT—on 1,394 whole-slide images from the SPARC IBD dataset using endoscopic labels as weak supervision.
• Employed self-supervised pre-training on the SPARC IBD dataset to learn meaningful tissue embeddings, improving downstream performance over ImageNet or TCGA pre-training.
• Utilized attention maps from the models to localize regions of interest, which were validated by pathologists for consistency with histological inflammation patterns.
• Leveraged a pre-trained CoNIC model to predict six immune cell populations in biopsies and compared these predictions with attention maps to assess spatial consistency.
• Evaluated model performance on three tasks: (i) CD vs. UC classification, (ii) normal vs. lesional tissue, and (iii) low vs. high endoscopic severity scores.
• Conducted ablation studies on subset-specific training (e.g., only colon or ileum biopsies) to assess performance variation and clinical applicability.

Experimental results

Research questions

• RQ1 Can weakly-supervised deep learning models trained on endoscopic labels alone accurately predict histopathological features in IBD biopsies?
• RQ2 Do attention maps from these models localize regions of inflammation that are consistent with pathologist-verified histological findings?
• RQ3 How do the predictions of immune cell populations from the CoNIC model compare with attention maps and pathologist descriptions of inflammation?
• RQ4 Does end-to-end fine-tuning on the SPARC IBD dataset improve performance over transfer learning from external datasets like TCGA?
• RQ5 Can these models be used to prioritize biopsies by predicted severity and guide pathologists to the most diseased regions?

Key findings

• The HIPT-E2E model achieved the highest AUC of 0.87 for distinguishing Crohn’s disease from ulcerative colitis, significantly outperforming DSMIL-E2E (p < 0.0001).
• For classifying normal vs. lesional tissue, the HIPT-E2E model achieved an AUC of 0.81, with a Dice coefficient of 0.625 between attention maps and pathologist-annotated inflammatory regions.
• For endoscopic severity scoring, the HIPT-E2E model achieved AUCs of 0.80 for Crohn’s disease and 0.80 for ulcerative colitis, indicating strong predictive power.
• The ratio of neutrophils to all cells was the most significant histological feature for ulcerative colitis (p = 0.0007), while eosinophil ratio was most indicative in Crohn’s disease (p = 0.0002).
• End-to-end training on the SPARC IBD dataset yielded better performance than fine-tuning with pre-trained weights, suggesting domain-specific self-supervision improves feature learning.
• Performance improved when training on only UC or only colon biopsies, but declined on ileum-only subsets, likely due to limited data and greater morphological complexity.