[Paper Review] Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorization

Weisen Wang, Zhiyan Xu+9 more|arXiv (Cornell University)|Jul 28, 2019

Retinal Imaging and Analysis17 references4 citations

TL;DR

This paper proposes a two-stream convolutional neural network (CNN) with loose pair training for end-to-end multi-modal age-related macular degeneration (AMD) categorization using color fundus and optical coherence tomography (OCT) images. By fusing features from both modalities and increasing training data via label-based loose pairing, the method achieves 97.1% accuracy—significantly outperforming prior state-of-the-art methods—while enabling visual interpretation through multi-modal class activation mapping.

ABSTRACT

This paper studies automated categorization of age-related macular degeneration (AMD) given a multi-modal input, which consists of a color fundus image and an optical coherence tomography (OCT) image from a specific eye. Previous work uses a traditional method, comprised of feature extraction and classifier training that cannot be optimized jointly. By contrast, we propose a two-stream convolutional neural network (CNN) that is end-to-end. The CNN's fusion layer is tailored to the need of fusing information from the fundus and OCT streams. For generating more multi-modal training instances, we introduce Loose Pair training, where a fundus image and an OCT image are paired based on class labels rather than eyes. Moreover, for a visual interpretation of how the individual modalities make contributions, we extend the class activation mapping technique to the multi-modal scenario. Experiments on a real-world dataset collected from an outpatient clinic justify the viability of our proposal for multi-modal AMD categorization.

Motivation & Objective

• To address the lack of end-to-end trainable multi-modal deep learning models for AMD categorization using fundus and OCT images.
• To overcome the limited number of paired training instances in clinical multi-modal datasets.
• To improve classification performance beyond single-modal and existing multi-modal baselines.
• To enable visual interpretation of modality contributions using an extended class activation mapping technique.
• To validate the method on a real-world outpatient clinic dataset for clinical relevance.

Proposed method

• Designs a two-stream CNN with symmetric branches for processing fundus and OCT images independently before feature fusion.
• Introduces a custom fusion layer that combines features from both streams to support end-to-end training and interpretability.
• Proposes loose pair training, where fundus and OCT images are paired based on class labels rather than anatomical eye pairs to increase training data diversity.
• Adapts class activation mapping (CAM) to the multi-modal setting to visualize which image regions contribute to predictions.
• Trains the model using an end-to-end backpropagation framework with cross-entropy loss for three-class classification (normal, dryAMD, wetAMD).
• Uses ResNet-18 as the backbone network for both streams, fine-tuned on the multi-modal dataset.

Experimental results

Research questions

• RQ1 Can an end-to-end two-stream CNN outperform single-modal models in multi-modal AMD categorization?
• RQ2 Does loose pair training effectively increase the number of useful training instances when paired data is scarce?
• RQ3 Can the proposed method achieve higher accuracy than the state-of-the-art multi-modal approach by Yoo et al. [16]?
• RQ4 How do the fundus and OCT modalities contribute to the final prediction, and can this be visually interpreted?
• RQ5 Is the fusion strategy effective in combining complementary information from fundus and OCT images?

Key findings

• The proposed multi-modal CNN with loose pair training (MM-CNN-L) achieves 97.1% overall accuracy, significantly outperforming the prior state-of-the-art method (82.6% accuracy).
• The OCT-CNN single-modal model achieves 94.2% F1 score, establishing a strong baseline that the multi-modal model must surpass.
• MM-CNN-L improves the F1 score for dryAMD from 82.8% (Yoo et al. -L) to 95.8%, indicating better detection of intermediate AMD stages.
• The method reduces misclassification of dryAMD as wetAMD, a common error in single-modal models, demonstrating improved diagnostic precision.
• Loose pair training boosts performance on the baseline method (Yoo et al. [16]) from 83.5% to 89.0% overall F1, confirming its general utility.
• Visual interpretation via extended CAM reveals that both fundus and OCT contribute meaningfully to predictions, with distinct attention patterns across modalities.