[Paper Review] Reinforcement Learning in Healthcare: A Survey

Chao Yu, Jiming Liu+1 more|arXiv (Cornell University)|Aug 22, 2019

Machine Learning in Healthcare336 references116 citations

TL;DR

A comprehensive survey of reinforcement learning (RL) in healthcare, detailing theoretical foundations, key techniques, diverse clinical applications, and open challenges.

ABSTRACT

As a subfield of machine learning, reinforcement learning (RL) aims at empowering one's capabilities in behavioural decision making by using interaction experience with the world and an evaluative feedback. Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure. This survey discusses the broad applications of RL techniques in healthcare domains, in order to provide the research community with systematic understanding of theoretical foundations, enabling methods and techniques, existing challenges, and new insights of this emerging paradigm. By first briefly examining theoretical foundations and key techniques in RL research from efficient and representational directions, we then provide an overview of RL applications in healthcare domains ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis from both unstructured and structured clinical data, as well as many other control or scheduling domains that have infiltrated many aspects of a healthcare system. Finally, we summarize the challenges and open issues in current research, and point out some potential solutions and directions for future research.

Motivation & Objective

• Provide systematic understanding of RL foundations and techniques relevant to healthcare.
• Summarize RL applications in dynamic treatment regimes, critical care, automated diagnosis, and health system management.
• Identify challenges, open issues, and potential directions for future RL research in healthcare.
• Contrast efficient and representational RL techniques and their suitability for healthcare problems.

Proposed method

• Review theoretical foundations of RL (MDP, value functions, Q-learning, DP, policy vs value-based methods).
• Discuss efficient (BRL, model-based, transfer) and representational (HRL, RRL, POMDP/PORL, IRL, MORL) techniques.
• Explain exploration-exploitation strategies and key challenges (safety, robustness, interpretability).
• Summarize core RL representations for states, actions, rewards, and tasks (factored MDPs, HRL, RRL, POMDP).
• Organize healthcare RL applications into dynamic treatment regimes, automated diagnosis, and other healthcare domains.

Experimental results

Research questions

• RQ1 What RL foundations and techniques are most applicable to healthcare sequential decision problems?
• RQ2 How are RL methods applied to dynamic treatment regimes and automated medical diagnosis in healthcare?
• RQ3 What are the main challenges and open issues hindering broader adoption of RL in healthcare, and what future directions are suggested?

Key findings

• RL is well-suited for sequential, delayed-feedback healthcare problems and can tailor treatments to individuals without explicit system models.
• BRL, MRl, HRL, RRL, IRL, MORL and POMDP/PORL representations enhance learning efficiency and scalability in healthcare contexts.
• Applications span dynamic treatment regimes in chronic diseases and critical care, automated diagnosis from structured/unstructured data, and health system management.
• The survey highlights challenges such as safety, robustness, data scarcity, interpretability, and the need for principled evaluation in healthcare RL.
• Future directions emphasize integrating domain knowledge, improving sample efficiency, and developing reliable, explainable RL methods for clinical deployment.