[Paper Review] A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning

Jing Wang, Junyan Fan+3 more|arXiv (Cornell University)|Jul 26, 2024

Digital Imaging in MedicineMedicine3 citations

TL;DR

This paper introduces a novel, large-scale ophthalmic ultrasound dataset comprising 4,858 de-identified eye ultrasound images, corresponding Chinese diagnostic reports, and blood flow parameters from 2,417 patients. It enables cross-modal deep learning for automated medical report generation, demonstrating strong performance with the CMN model in aligning image regions with relevant report phrases, particularly for common ocular pathologies.

ABSTRACT

Ultrasound imaging reveals eye morphology and aids in diagnosing and treating eye diseases. However, interpreting diagnostic reports requires specialized physicians. We present a labeled ophthalmic dataset for the precise analysis and the automated exploration of medical images along with their associated reports. It collects three modal data, including the ultrasound images, blood flow information and examination reports from 2,417 patients at an ophthalmology hospital in Shenyang, China, during the year 2018, in which the patient information is de-identified for privacy protection. To the best of our knowledge, it is the only ophthalmic dataset that contains the three modal information simultaneously. It incrementally consists of 4,858 images with the corresponding free-text reports, which describe 15 typical imaging findings of intraocular diseases and the corresponding anatomical locations. Each image shows three kinds of blood flow indices at three specific arteries, i.e., nine parameter values to describe the spectral characteristics of blood flow distribution. The reports were written by ophthalmologists during the clinical care. The proposed dataset is applied to generate medical report based on the cross-modal deep learning model. The experimental results demonstrate that our dataset is suitable for training supervised models concerning cross-modal medical data.

Motivation & Objective

• To address the lack of multimodal ophthalmic ultrasound datasets by creating a comprehensive, real-world clinical dataset with images, reports, and hemodynamic parameters.
• To support the development of cross-modal medical report generation models that align visual features with clinical language patterns.
• To reduce the diagnostic burden on ophthalmologists by enabling AI-assisted report generation based on ultrasound imaging.
• To provide a benchmark dataset for training and evaluating supervised models in ophthalmic AI, especially for non-English (Chinese) clinical report generation.
• To improve diagnostic accuracy and consistency by minimizing inter-physician variability through AI-driven report standardization.

Proposed method

• The dataset was constructed from real clinical data collected at a single ophthalmology hospital in Shenyang, China, in 2018, with patient information de-identified for privacy.
• Each case includes three ultrasound image types (B-mode, Doppler, spectral Doppler), nine blood flow indices from three ocular arteries, and a free-text diagnostic report written by ophthalmologists.
• A cross-modal memory network (CMN) was trained to generate medical reports by attending to relevant image regions and integrating visual and hemodynamic features.
• The model uses a dual-branch architecture: a convolutional neural network (CNN) for image feature extraction and a recurrent neural network (RNN) for text generation.
• Attention mechanisms were applied to visualize the alignment between image regions and report phrases, enabling interpretability of model predictions.
• Evaluation was conducted using metrics such as BLEU, ROUGE, and CIDEr, with qualitative analysis of attention maps and report length distribution.

Experimental results

Research questions

• RQ1 Can a multimodal ophthalmic ultrasound dataset including images, reports, and hemodynamic parameters support effective cross-modal medical report generation?
• RQ2 How well can a cross-modal deep learning model like CMN align image features with clinically relevant report descriptions?
• RQ3 Does the inclusion of blood flow parameters improve the accuracy and clinical relevance of generated reports compared to image-only models?
• RQ4 To what extent does the model’s attention mechanism focus on actual pathological regions in ultrasound images as verified by expert interpretation?
• RQ5 How does the generated report length and content distribution compare to real physician-written reports?

Key findings

• The proposed dataset contains 4,858 ultrasound images, 4,858 corresponding Chinese diagnostic reports, and 2,417 patient records with nine blood flow parameters, making it the only ophthalmic dataset with all three modalities simultaneously.
• The CMN model generated reports with length distributions closely matching real reports, particularly in the 50–60 word range, where ~350 ground-truth reports were concentrated.
• Visualization of attention maps confirmed that the CMN model successfully focused on lesion regions such as vitreous hemorrhage, vitreous opacities, and post-vitrectomy changes.
• The model demonstrated strong generalization, correctly identifying complex pathologies like 'vitreous mechanized accumulation of blood' and 'abnormal intraocular echoes' in some cases.
• Despite strong performance, challenges remained in detecting lesions at image edges (e.g., posterior scleral staphyloma) and distinguishing visually similar pathologies.
• The dataset enables effective training of supervised cross-modal models and shows promise for reducing clinical workload through AI-assisted report generation.