[Paper Review] Multimodal Densenet

Faisal Mahmood, Ziyun Yang+2 more|arXiv (Cornell University)|Nov 18, 2018

AI in cancer detection4 citations

TL;DR

This paper proposes Multimodal Densenet, a novel deep learning architecture that fuses multimodal medical imaging data—such as white light, narrow-band imaging, and depth maps—through dense, multi-layer feature fusion in the first dense block, significantly improving polyp classification and anatomical landmark detection accuracy over monomodal and existing multimodal fusion methods.

ABSTRACT

Humans make accurate decisions by interpreting complex data from multiple sources. Medical diagnostics, in particular, often hinge on human interpretation of multi-modal information. In order for artificial intelligence to make progress in automated, objective, and accurate diagnosis and prognosis, methods to fuse information from multiple medical imaging modalities are required. However, combining information from multiple data sources has several challenges, as current deep learning architectures lack the ability to extract useful representations from multimodal information, and often simple concatenation is used to fuse such information. In this work, we propose Multimodal DenseNet, a novel architecture for fusing multimodal data. Instead of focusing on concatenation or early and late fusion, our proposed architectures fuses information over several layers and gives the model flexibility in how it combines information from multiple sources. We apply this architecture to the challenge of polyp characterization and landmark identification in endoscopy. Features from white light images are fused with features from narrow band imaging or depth maps. This study demonstrates that Multimodal DenseNet outperforms monomodal classification as well as other multimodal fusion techniques by a significant margin on two different datasets.

Motivation & Objective

• To address the challenge of fusing multimodal medical imaging data—such as endoscopic images from different contrast settings—into a unified, robust representation for improved diagnostic accuracy.
• To overcome limitations of traditional fusion methods like early/late fusion or simple concatenation, which often fail to capture complex inter-modal dependencies.
• To enable effective fusion of unpaired and unregistered multimodal data (e.g., white light and narrow-band imaging) through adversarial domain alignment with cycle-consistency loss.
• To develop a deep learning architecture that leverages the benefits of dense connectivity and multi-layer fusion to mitigate vanishing gradient issues and improve feature learning across modalities.
• To demonstrate state-of-the-art performance in multimodal scene classification tasks in endoscopy, including polyp characterization and anatomical landmark detection.

Proposed method

• Proposes a Multimodal Densenet architecture that integrates features from multiple imaging modalities within the first dense block, using dense skip connections to enhance feature flow and gradient propagation.
• Employs multi-layer fusion by fusing features from different modalities across multiple layers (specifically the last eight layers of the first dense block), enabling hierarchical and dynamic fusion of cross-modal representations.
• Utilizes a dual-GAN framework with cycle-consistency loss to align unpaired multimodal data (e.g., white light and NBI endoscopy) from the same anatomical region, enabling effective fusion without requiring paired training data.
• Adapts the DenseNet architecture by modifying residual connections to support dual-stream input (e.g., RGB and depth or white light and NBI), allowing shared feature learning across modalities.
• Applies gradient class activation mapping (Grad-CAM) to visualize and validate the model’s attention mechanisms, confirming that the network focuses on diagnostically relevant regions such as polyp surfaces.
• Uses standard training protocols with cross-entropy loss and Adam optimization, with training conducted on two distinct endoscopic datasets for polyp classification and landmark detection.

Experimental results

Research questions

• RQ1 Can a deep learning architecture that fuses multimodal medical imaging data across multiple layers outperform monomodal and traditional multimodal fusion methods in endoscopic image classification?
• RQ2 How effective is dense, multi-layer fusion in capturing inter-modal dependencies and improving diagnostic accuracy compared to early, late, or concatenation-based fusion strategies?
• RQ3 Can unpaired and unregistered multimodal endoscopic images (e.g., white light and narrow-band imaging) be effectively aligned and fused using adversarial training with cycle-consistency loss?
• RQ4 To what extent does the proposed Multimodal Densenet improve performance in real-world clinical tasks such as polyp characterization and anatomical landmark detection?
• RQ5 Does the model’s attention, as visualized via Grad-CAM, align with clinically relevant features such as polyp surface texture and color, indicating robust and interpretable feature learning?

Key findings

• Multimodal Densenet achieved a 24.72% higher F1 score on RGB-D polyp classification compared to standard DenseNet, demonstrating significant performance gains from multi-modal fusion.
• On the white light and NBI polyp classification task, the proposed model improved F1 score by 8.03% over monomodal DenseNet, outperforming both VGG16-Fuse (+4.28%) and ResNet-50-Fuse (+8.03%) baselines.
• The model achieved state-of-the-art results in 7-class anatomical landmark and pathological classification using RGB and depth images, with improved accuracy over existing fusion architectures.
• Grad-CAM visualization confirmed that Multimodal Densenet focuses on diagnostically relevant regions—such as the polyp surface—using combined color, texture, and depth cues, unlike monomodal Densenet, which often relies on surrounding tissue.
• Adversarial domain alignment with cycle-consistency loss successfully enabled fusion of unpaired white light and NBI images, enabling effective use of unregistered data without requiring manual registration.
• Despite requiring longer training times, the model’s performance gains justify the computational cost, particularly in clinical settings where diagnostic accuracy is critical.