[Paper Review] Multimodal classification of Radiation-Induced Contrast Enhancements and tumor recurrence using deep learning

Robin Peretzke, Marlin Hanstein+16 more|arXiv (Cornell University)|Mar 12, 2026

Glioma Diagnosis and Treatment0 citations

TL;DR

RICE-Net is a multimodal 3D deep learning model that differentiates radiation-induced contrast enhancements from glioblastoma recurrence by integrating longitudinal MRI data with radiotherapy dose maps, achieving 0.92 macro F1 on an independent test set.

ABSTRACT

The differentiation between tumor recurrence and radiation-induced contrast enhancements in post-treatment glioblastoma patients remains a major clinical challenge. Existing approaches rely on clinically sparsely available diffusion MRI or do not consider radiation maps, which are gaining increasing interest in the tumor board for this differentiation. We introduce RICE-NET, a multimodal 3D deep learning model that integrates longitudinal MRI data with radiotherapy dose distributions for automated lesion classification using conventional T1-weighted MRI data. Using a cohort of 92 patients, the model achieved an F1 score of 0.92 on an independent test set. During extensive ablation experiments, we quantified the contribution of each timepoint and modality and showed that reliable classification largely depends on the radiation map. Occlusion-based interpretability analyses further confirmed the model's focus on clinically relevant regions. These findings highlight the potential of multimodal deep learning to enhance diagnostic accuracy and support clinical decision-making in neuro-oncology.

Motivation & Objective

• Differentiate radiation-induced contrast enhancements (RICE) from glioblastoma recurrence in post-treatment GBM patients.
• Incorporate longitudinal MRI data and radiotherapy dose distributions to improve diagnostic accuracy.
• Quantify the contribution of each modality and identify the most informative inputs.
• Provide interpretable insights into model focus via occlusion analysis.
• Highlight the potential for early, automated decision support in neuro-oncology.

Proposed method

• Use a 3D ResNet-18 architecture adapted for volumetric medical images.
• Integrate three input modalities: post-operative MRI (MRI post-OP), contrast-event MRI (MRI event), and radiation dose (RD) map by channel-wise fusion.
• Preprocess data with ANTS registration, HD-BET skull-stripping, z-score normalization, and cropping to 224x224x224 voxels.
• Train with 800 epochs, five-fold cross-validation, Adam optimizer, and class-balanced sampling.
• Evaluate primarily with macro F1-score and perform occlusion-based interpretability analyses.
• Conduct ablation studies across input modalities and combinations to quantify contribution of each modality.

Figure 1 : Axial slice of sample subject with tumor recurrence. On the left, the post-operative MRI with the resection area highlighted with a red circle. The middle image shows the new progression, which is to be classified as recurrence or RICE. On the right, the radiation treatment plan is displa

Experimental results

Research questions

• RQ1 Can a multimodal 3D deep learning model distinguish RICE from tumor recurrence using conventional T1-weighted MRI and radiation dose maps?
• RQ2 What is the relative diagnostic value of each input modality (MRI post-OP, MRI event, RD) for this task?
• RQ3 Does integrating radiation dose with MRI data improve classification performance compared to MRI data alone?
• RQ4 Where does the model focus when differentiating RICE from recurrence, and are the regions clinically meaningful?

Key findings

• The RD map alone yielded the highest single-modality validation macro F1 of 0.78.
• MRI post-OP and MRI event alone yielded lower F1 scores of 0.70 and 0.58, respectively.
• RD+MRI post-OP achieved 0.828, RD+MRI event achieved 0.83, with all three inputs giving 0.804 validation F1.
• On the independent test set, an ensemble of cross-validated models achieved 0.916 macro F1.
• Ablation results show radiation dosage information is highly informative and that combining modalities provides complementary gains.
• Occlusion analysis highlighted high-dose regions and clinically relevant areas, supporting model focus on meaningful regions.

Figure 2 : F1 Macro after 800 training epochs on validation data (striped) aggregated across all folds and majority vote on the test data (dotted) by input volume combinations with cross validation standard deviation as error bars.