[Paper Review] Effective Medical Test Suggestions Using Deep Reinforcement Learning

Yang-En Chen, Kai-Fu Tang+2 more|arXiv (Cornell University)|May 30, 2019

Reinforcement Learning in RoboticsComputer Science12 references3 citations

TL;DR

This paper proposes a deep reinforcement learning agent that learns to suggest multiple medical tests simultaneously to improve disease diagnosis accuracy. Using a stage-wise Markov decision process with a novel multiple-action policy representation and label-guided exploration, the agent reduces redundant testing while achieving 89.75% top-5 diagnosis accuracy across 200 diseases with an average of 4.77 tests per patient.

ABSTRACT

Effective medical test suggestions benefit both patients and physicians to conserve time and improve diagnosis accuracy. In this work, we show that an agent can learn to suggest effective medical tests. We formulate the problem as a stage-wise Markov decision process and propose a reinforcement learning method to train the agent. We introduce a new representation of multiple action policy along with the training method of the proposed representation. Furthermore, a new exploration scheme is proposed to accelerate the learning of disease distributions. Our experimental results demonstrate that the accuracy of disease diagnosis can be significantly improved with good medical test suggestions.

Motivation & Objective

• To improve medical diagnosis accuracy by automating effective medical test suggestions.
• To address the challenge of suggesting multiple medical tests at once, reducing patient visits and improving efficiency.
• To reduce the exploration burden in large action spaces during reinforcement learning training.
• To accelerate learning by leveraging disease label information during training.
• To develop a stage-wise MDP framework that models symptom queries, test suggestions, and disease prediction as sequential decision stages.

Proposed method

• Formulates medical test suggestion as a stage-wise Markov decision process with three distinct stages: symptom query, medical test suggestion, and disease prediction.
• Introduces a novel multiple-action policy representation that allows the agent to select multiple medical tests simultaneously in a single decision step.
• Develops a specialized training method for the multiple-action policy using a hierarchical action space and policy gradient techniques.
• Proposes a label-guided exploration strategy that uses ground-truth disease labels during training to prioritize informative actions and accelerate convergence.
• Employs a state representation that includes demographic data, symptoms, and medical test results, with a stage flag to guide sequential decision-making.
• Uses a discounted cumulative reward function to optimize long-term diagnosis accuracy, with rewards shaped by improvement in top-k prediction performance.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent learn to suggest multiple medical tests simultaneously to improve diagnosis accuracy?
• RQ2 How does a stage-wise MDP framework enhance learning efficiency and reduce action space complexity in medical test suggestion?
• RQ3 To what extent does label-guided exploration accelerate policy training in disease distribution learning?
• RQ4 What is the optimal balance between test suggestion frequency and diagnostic accuracy in real-world medical scenarios?
• RQ5 How effective is the proposed method in discovering abnormal test results compared to baseline approaches?

Key findings

• The proposed method achieved 89.75% top-5 diagnosis accuracy on a 200-disease dataset, representing a 7%–14% improvement over the baseline REFUEL model.
• The agent suggested an average of 4.77 medical tests per patient, with a suggestion probability of 47.59% across the 200-disease setting.
• The agent discovered 92.81% of abnormal test results in the 200-disease case, demonstrating high sensitivity to clinically relevant abnormalities.
• The average number of suggested tests decreased over training time, indicating effective learning of minimal yet informative test sets.
• The label-guided exploration strategy significantly accelerated learning, as evidenced by faster convergence and improved early training accuracy.
• In the 400-disease setting, the agent maintained strong performance with 79.73% top-5 accuracy and discovered 73.70% of abnormalities, showing robustness to increased disease complexity.