[Paper Review] A framework for dynamically training and adapting deep reinforcement learning models to different, low-compute, and continuously changing radiology deployment environments

Guangyao Zheng, Shuhao Lai+3 more|arXiv (Cornell University)|Jun 8, 2023

Radiomics and Machine Learning in Medical Imaging4 citations

TL;DR

This paper proposes a dynamic, lifelong reinforcement learning framework using image coreset compression to enable efficient, low-compute deployment of deep reinforcement learning (DRL) models in evolving radiology environments. By applying neighborhood averaging, sensitivity-based sampling, and maximum entropy coreset methods for 27× image compression, the framework reduces computational load while preserving performance, achieving a 11.97±12.02 average distance error in anatomical landmark localization—significantly better than conventional lifelong learning (19.24±50.77).

ABSTRACT

While Deep Reinforcement Learning has been widely researched in medical imaging, the training and deployment of these models usually require powerful GPUs. Since imaging environments evolve rapidly and can be generated by edge devices, the algorithm is required to continually learn and adapt to changing environments, and adjust to low-compute devices. To this end, we developed three image coreset algorithms to compress and denoise medical images for selective experience replayed-based lifelong reinforcement learning. We implemented neighborhood averaging coreset, neighborhood sensitivity-based sampling coreset, and maximum entropy coreset on full-body DIXON water and DIXON fat MRI images. All three coresets produced 27x compression with excellent performance in localizing five anatomical landmarks: left knee, right trochanter, left kidney, spleen, and lung across both imaging environments. Maximum entropy coreset obtained the best performance of $11.97\pm 12.02$ average distance error, compared to the conventional lifelong learning framework's $19.24\pm 50.77$.

Motivation & Objective

• Address the challenge of deploying large-scale DRL models in low-compute, edge-based radiology environments due to high computational demands.
• Overcome catastrophic forgetting in lifelong learning scenarios where models must adapt to new imaging sequences (e.g., DIXON water vs. fat MRI) without losing prior knowledge.
• Enable efficient, real-time adaptation of DRL models to continuously changing medical imaging environments, including diverse MRI sequences and anatomical orientations.
• Develop a scalable, computationally lightweight framework that supports continual learning across multiple imaging modalities and anatomical tasks without retraining from scratch.
• Improve model generalization and robustness across diverse clinical imaging conditions using selective experience replay with compressed, representative image coresets.

Proposed method

• Propose a selective experience replay-based lifelong reinforcement learning framework (ICSERIL) that uses image coreset techniques to compress input MRI data before training.
• Implement three coreset methods: neighborhood averaging (mean of 3D voxel cubes), neighborhood sensitivity-based sampling (selecting center voxel of each cube), and maximum entropy coreset (selecting highest-entropy voxel in 3D cube).
• Apply a 3× scaling ratio across all three dimensions (x, y, z), achieving 27× compression of full-body DIXON MRI images while preserving structural and intensity information.
• Use a 3D bounding box as the agent’s state representation (15×15×9) for efficient inference on coreset images, with predictions scaled back to original resolution.
• Train agents on alternating sequences (DIXON water then DIXON fat) using experience replay buffers (ERB) storing state-action-reward-resulting state tuples to enable continual learning.
• Integrate entropy-based sampling to prioritize information-rich regions in the image, enhancing feature representation in compressed data for improved localization performance.

Figure 1 : Illustration of catastrophic forgetting. (A) Model 1 is trained on DIXON fat data, and it is able to localize the right trochanter. (B) When Model 1 encounters DIXON water data, it cannot localize the right trochanter. (C) When we finetune Model 1 on DIXON water data to produce Model 2, i

Experimental results

Research questions

• RQ1 Can image coreset compression techniques effectively reduce the computational burden of DRL models in low-compute radiology deployment environments without sacrificing localization accuracy?
• RQ2 How do different coreset selection strategies (averaging, sensitivity-based, maximum entropy) compare in preserving critical anatomical information for DRL-based landmark localization?
• RQ3 To what extent can the proposed ICSERIL framework mitigate catastrophic forgetting when adapting to new imaging sequences (e.g., DIXON water vs. fat MRI) across multiple anatomical landmarks?
• RQ4 Does the use of compressed coresets in experience replay improve the generalization and robustness of DRL agents across diverse imaging environments and tasks?
• RQ5 What is the trade-off between computational efficiency (per-epoch training time) and performance (localization error) when using coreset-based compression in lifelong DRL for medical imaging?

Key findings

• The maximum entropy coreset method achieved the best landmark localization performance with an average distance error of 11.97±12.02 mm, significantly outperforming the conventional lifelong learning baseline (19.24±50.77 mm).
• All three coreset methods achieved 27× compression of full-body DIXON MRI images while maintaining high localization accuracy across five anatomical landmarks: left knee, right trochanter, left kidney, spleen, and lung.
• The ICSERIL framework with maximum entropy coreset reduced the average localization error by 37.8% compared to the conventional lifelong learning approach, demonstrating improved robustness and generalization.
• The proposed framework achieved a 2.5× reduction in per-epoch training time compared to the conventional LL baseline, indicating improved computational efficiency despite lower-resolution input.
• The neighborhood sensitivity-based sampling coreset showed moderate performance, with a mean error of 13.85±14.11 mm, indicating that representative sampling is effective but less optimal than entropy-based selection.
• Statistical analysis confirmed that the performance improvement of the maximum entropy coreset over the conventional method was significant (p < 0.05), validating its effectiveness in lifelong learning under low-compute constraints.

Figure 2 : Illustration of the two different full-body MRI imaging environments (DIXON water denoted ”W” and DIXON fat denoted ”F”), and the five different anatomical landmark localization tasks (left knee, right trochanter, left kidney, spleen, and lung). The performance of the ICSERIL models with