[Paper Review] Cross-modal Memory Networks for Radiology Report Generation

Zhihong Chen, Yaling Shen+2 more|arXiv (Cornell University)|Apr 28, 2022

Multimodal Machine Learning ApplicationsComputer Science26 citations

TL;DR

The paper introduces Cross-modal Memory Networks (CMN) to enhance radiology report generation by storing cross-modal image-text alignments in a shared memory, achieving state-of-the-art results on IU X-Ray and MIMIC-CXR datasets.

ABSTRACT

Medical imaging plays a significant role in clinical practice of medical diagnosis, where the text reports of the images are essential in understanding them and facilitating later treatments. By generating the reports automatically, it is beneficial to help lighten the burden of radiologists and significantly promote clinical automation, which already attracts much attention in applying artificial intelligence to medical domain. Previous studies mainly follow the encoder-decoder paradigm and focus on the aspect of text generation, with few studies considering the importance of cross-modal mappings and explicitly exploit such mappings to facilitate radiology report generation. In this paper, we propose a cross-modal memory networks (CMN) to enhance the encoder-decoder framework for radiology report generation, where a shared memory is designed to record the alignment between images and texts so as to facilitate the interaction and generation across modalities. Experimental results illustrate the effectiveness of our proposed model, where state-of-the-art performance is achieved on two widely used benchmark datasets, i.e., IU X-Ray and MIMIC-CXR. Further analyses also prove that our model is able to better align information from radiology images and texts so as to help generating more accurate reports in terms of clinical indicators.

Motivation & Objective

• Motivate automatic radiology report generation to alleviate radiologists’ workload.
• Explicitly model and leverage cross-modal alignments between chest X-ray images and reports.
• Introduce a memory-based medium that stores shared cross-modal information.
• Enhance an encoder-decoder (Transformer) with memory-driven cross-modal interactions.
• Demonstrate state-of-the-art performance on two benchmark datasets.

Proposed method

• Use a CNN-based visual extractor to obtain region-based image features.
• Introduce a cross-modal memory network (CMN) with a shared memory matrix to encode image-text alignments.
• Perform memory querying by mapping input features and memory vectors into a common space and selecting top-K memories.
• Generate responses by weighted summation of transformed memory vectors for both visual and textual inputs.
• Feed memory responses into a Transformer-based encoder-decoder to produce radiology reports.

Experimental results

Research questions

• RQ1 Can a shared cross-modal memory improve alignment and generation quality in radiology report generation?
• RQ2 Does CMN outperform single-modality memory or no-memory baselines on standard radiology benchmarks?
• RQ3 How do memory size and querying parameters affect generation quality and alignment?
• RQ4 Does joint memory usage in encoding and decoding yield larger gains than memory in decoding alone?

Key findings

• CMN-based models outperform the baselines on both NLG and, when applicable, CE metrics.
• Base+cmn achieves the best BLEU/METEOR/ROUGE-L scores among evaluated models on IU X-Ray and MIMIC-CXR.
• Memory-augmented encoding plus decoding yields larger gains than memory in decoding alone (Base+cmn > Base+mem).
• Larger memory size helps up to a point (optimal around moderate N); beyond that, gains saturate or degrade due to memory not being sufficiently updated.
• The model demonstrates meaningful image-text mappings and alignment through qualitative analyses and case studies.