[Paper Review] MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network

Zizhao Zhang, Yuanpu Xie+3 more|arXiv (Cornell University)|Jul 8, 2017

AI in cancer detection62 citations

TL;DR

MDNet builds a unified multimodal network that reads medical images, generates diagnostic reports, retrieves images by symptom descriptions, and visualizes attention for interpretable diagnosis. It advances an ensemble-connection ResNet for multi-scale features and an attention-enhanced LSTM with an auxiliary attention sharpening module.

ABSTRACT

The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.

Motivation & Objective

• Motivate interpretable medical image diagnosis by aligning image features with diagnostic reports.
• Develop a unified model that can read images, generate reports, retrieve images by symptom descriptions, and visualize attention.
• Enhance multi-scale image representation through a novel ensemble-connection-based network.
• Learn semantic mappings from report words to image pixels to provide justifications for predictions.
• Train end-to-end to jointly optimize image and language models with auxiliary supervision to improve attention.

Proposed method

• Introduce EcNet with ensemble-connection to integrate multi-scale features and improve feature utilization.
• Adopt an LSTM-based language model with soft attention to map sentence words to image regions.
• Incorporate an auxiliary attention sharpening (AAS) module to produce sharper, class-specific attention maps.
• Use a joint training objective that optimizes both diagnostic conclusion prediction and report generation, with a backpropagation scheme that blends gradients from both tasks.
• Implement a training scheme that duplicates samples to create K task-specific inputs for the language model to guide effective gradient flow.
• Provide end-to-end optimization of image model, language model, and AAS under a unified objective.

Experimental results

Research questions

• RQ1 Can a unified multimodal network generate diagnostic reports from medical images while providing semantically and visually interpretable explanations?
• RQ2 Does the proposed MDNet outperform baselines in bladder cancer image diagnosis and in image-language tasks on the BCIDR dataset?
• RQ3 Do ensemble-connection based networks improve multi-scale feature utilization for medical imaging tasks?
• RQ4 Does attention sharpening improve the localization and usefulness of attention maps in clinical interpretations?

Key findings

• MDNet achieves superior diagnostic conclusion accuracy and report-generation metrics on BCIDR compared with baselines.
• MDNet’s sentence-guided attention maps focus on medically informative urothelial regions, improving interpretability.
• The model’s DCA (diagnostic conclusion accuracy) scores show improvements over baselines and demonstrate stable performance across folds.
• Ensemble-connection in EcNet improves image representation efficiency, achieving competitive or state-of-the-art results on CIFAR-10 and CIFAR-100 with fewer parameters than some baselines.
• Joint optimization with effective gradient flow (balancing image-model and language-model gradients) enhances multimodal mapping quality and reduces overfitting on small medical datasets.
• MDNet demonstrates strong sentence-to-image mapping accuracy for symptom-based image retrieval, outperforming baselines in Cr@k metrics.