[Paper Review] Multimodal Information Fusion for Glaucoma and DR Classification

Yihao Li, Mostafa El Habib Daho+10 more|arXiv (Cornell University)|Sep 2, 2022

Retinal Imaging and Analysis4 citations

TL;DR

This paper proposes a novel hierarchical fusion method for multimodal deep learning in retinal disease classification, combining 2D fundus photographs, 3D OCT, and OCT angiography data to improve glaucoma and diabetic retinopathy diagnosis. The approach outperforms early and intermediate fusion by modeling cross-modal correlations across multiple network dimensions, achieving a Kappa of 0.8996 on the GAMMA dataset and an AUC of 0.911 on the PlexEliteDR dataset.

ABSTRACT

Multimodal information is frequently available in medical tasks. By combining information from multiple sources, clinicians are able to make more accurate judgments. In recent years, multiple imaging techniques have been used in clinical practice for retinal analysis: 2D fundus photographs, 3D optical coherence tomography (OCT) and 3D OCT angiography, etc. Our paper investigates three multimodal information fusion strategies based on deep learning to solve retinal analysis tasks: early fusion, intermediate fusion, and hierarchical fusion. The commonly used early and intermediate fusions are simple but do not fully exploit the complementary information between modalities. We developed a hierarchical fusion approach that focuses on combining features across multiple dimensions of the network, as well as exploring the correlation between modalities. These approaches were applied to glaucoma and diabetic retinopathy classification, using the public GAMMA dataset (fundus photographs and OCT) and a private dataset of PlexElite 9000 (Carl Zeis Meditec Inc.) OCT angiography acquisitions, respectively. Our hierarchical fusion method performed the best in both cases and paved the way for better clinical diagnosis.

Motivation & Objective

• To address the limitations of single-modality retinal disease classification by leveraging multimodal imaging data.
• To improve diagnostic accuracy in glaucoma and diabetic retinopathy by fusing complementary information from fundus photography, OCT, and OCT angiography.
• To develop a more effective fusion strategy than early or intermediate fusion by modeling cross-modal correlations across multiple network dimensions.
• To validate the proposed hierarchical fusion method on both public (GAMMA) and private (PlexEliteDR) datasets for robust clinical applicability.

Proposed method

• Develops a hierarchical fusion framework that combines features across multiple dimensions of a deep neural network, enabling joint learning of modality-specific and cross-modal representations.
• Applies 3D convolutional neural networks to extract spatial features from 3D OCT volumes and 2D/3D CNNs to process fundus images and OCT angiography data.
• Uses late-stage concatenation of modality-specific features followed by attention mechanisms to emphasize informative cross-modal correlations.
• Employs model ensembling of ResNet50 and ResNet101 for final prediction to boost performance on the GAMMA dataset.
• Implements data augmentation and transfer learning with ImageNet-pretrained backbones (ResNet, DenseNet) to improve generalization.
• Optimizes training with batch normalization and learning rate scheduling, adapting batch size to hardware constraints (e.g., reduced for ResNet152).

Experimental results

Research questions

• RQ1 Can hierarchical fusion outperform early and intermediate fusion in multimodal retinal disease classification?
• RQ2 How does the spatial misalignment between fundus and OCT images affect fusion performance, and can hierarchical fusion mitigate this?
• RQ3 Does the inclusion of OCT angiography data improve classification beyond structural OCT and fundus images?
• RQ4 How do different backbone architectures (ResNet, DenseNet) influence the performance of hierarchical fusion?
• RQ5 Can hierarchical fusion generalize across datasets with varying degrees of modality alignment?

Key findings

• The hierarchical fusion method achieved a Kappa score of 0.8996 on the GAMMA dataset, significantly outperforming intermediate fusion (Kappa 0.7340) and single-modality approaches (Kappa 0.6382).
• On the PlexEliteDR dataset, hierarchical fusion reached an AUC of 0.911 using DenseNet121, surpassing the single-modality baseline (AUC 0.859) and early fusion (AUC 0.865).
• Intermediate fusion underperformed on the GAMMA dataset due to spatial misalignment between fundus and OCT images, highlighting the limitations of feature-level fusion without alignment.
• Early fusion performed well on the PlexEliteDR dataset due to spatial alignment of modalities, demonstrating that alignment is critical for input-level fusion.
• The hierarchical fusion model showed strong generalization, achieving the highest performance across multiple backbones and datasets.
• Ensembling ResNet50 and ResNet101 models in the hierarchical framework further improved the GAMMA dataset Kappa to 0.8996, indicating the benefit of model diversity.