[论文解读] Reinforcement Learning in Healthcare: A Survey

Chao Yu, Jiming Liu+1 more|arXiv (Cornell University)|Aug 22, 2019

Machine Learning in Healthcare参考文献 336被引用 116

一句话总结

对医疗保健中强化学习（RL）的全面综述，详细介绍理论基础、关键技术、多样化的临床应用以及未解决的挑战。

ABSTRACT

As a subfield of machine learning, reinforcement learning (RL) aims at empowering one's capabilities in behavioural decision making by using interaction experience with the world and an evaluative feedback. Unlike traditional supervised learning methods that usually rely on one-shot, exhaustive and supervised reward signals, RL tackles with sequential decision making problems with sampled, evaluative and delayed feedback simultaneously. Such distinctive features make RL technique a suitable candidate for developing powerful solutions in a variety of healthcare domains, where diagnosing decisions or treatment regimes are usually characterized by a prolonged and sequential procedure. This survey discusses the broad applications of RL techniques in healthcare domains, in order to provide the research community with systematic understanding of theoretical foundations, enabling methods and techniques, existing challenges, and new insights of this emerging paradigm. By first briefly examining theoretical foundations and key techniques in RL research from efficient and representational directions, we then provide an overview of RL applications in healthcare domains ranging from dynamic treatment regimes in chronic diseases and critical care, automated medical diagnosis from both unstructured and structured clinical data, as well as many other control or scheduling domains that have infiltrated many aspects of a healthcare system. Finally, we summarize the challenges and open issues in current research, and point out some potential solutions and directions for future research.

研究动机与目标

• 为医疗保健相关的RL基础与技术提供系统性理解。
• 总结RL在动态治疗方案、重症护理、自动诊断和卫生系统管理中的应用。
• 识别医疗保健中RL研究的挑战、待解决的问题以及未来潜在方向。
• 对比高效与表征型RL技术及其对医疗问题的适用性。

提出的方法

• 回顾RL的理论基础（MDP、价值函数、Q-learning、DP、策略与基于价值的方法）。
• 讨论高效（BRL、基于模型、迁移）与表征型（HRL、RRL、POMDP/PORL、IRL、MORL）技术。
• 解释探索-开发策略及关键挑战（安全性、鲁棒性、可解释性）。
• 总结状态、动作、奖励与任务的核心RL表征（因子化MDP、HRL、RRL、POMDP）。
• 将医疗保健RL应用划分为动态治疗方案、自动诊断以及其他医疗领域。

实验结果

研究问题

• RQ1 哪些RL基础与技术最适用于医疗保健中的序贯决策问题？
• RQ2 RL方法如何在医疗保健中的动态治疗方案和自动医疗诊断中应用？
• RQ3 阻碍RL在医疗保健中更广泛应用的主要挑战与待解决的问题有哪些，以及有哪些未来方向？

主要发现

• RL适用于序贯、延迟反馈的医疗保健问题，能够在没有显式系统模型的情况下实现个体化治疗。
• BRL、MRl、HRL、RRL、IRL、MORL 与 POMDP/PORL 表征在医疗保健情境中提升学习效率与可扩展性。
• 应用涵盖慢性病和重症护理中的动态治疗方案、来自结构化/非结构化数据的自动诊断，以及卫生系统管理。
• 该综述强调了在医疗保健RL中存在的安全性、鲁棒性、数据稀缺、可解释性以及需要基于原理的评估等挑战。
• 未来方向强调整合领域知识、提升样本效率、以及开发可可靠、可解释的RL方法以用于临床部署。