[论文解读] Bandit Algorithms for Precision Medicine

Yangyi Lu, Ziping Xu+1 more|arXiv (Cornell University)|Aug 10, 2021

Advanced Bandit Algorithms Research参考文献 123被引用 6

一句话总结

本文提出专为精准医疗设计的上下文bandit与多臂bandit算法，利用患者特异性数据优化个性化治疗决策。引入了如核化多任务学习和Thompson采样中的自适应聚合等先进方法，分别实现了$\tilde{O}(\sqrt{Tr_zr_c})$和$\tilde{O}(dn\sqrt{T})$的遗憾边界，展示了在异质患者群体中提升学习效率的潜力。

ABSTRACT

The Oxford English Dictionary defines precision medicine as "medical care designed to optimize efficiency or therapeutic benefit for particular groups of patients, especially by using genetic or molecular profiling." It is not an entirely new idea: physicians from ancient times have recognized that medical treatment needs to consider individual variations in patient characteristics. However, the modern precision medicine movement has been enabled by a confluence of events: scientific advances in fields such as genetics and pharmacology, technological advances in mobile devices and wearable sensors, and methodological advances in computing and data sciences. This chapter is about bandit algorithms: an area of data science of special relevance to precision medicine. With their roots in the seminal work of Bellman, Robbins, Lai and others, bandit algorithms have come to occupy a central place in modern data science ( Lattimore and Szepesvari, 2020). Bandit algorithms can be used in any situation where treatment decisions need to be made to optimize some health outcome. Since precision medicine focuses on the use of patient characteristics to guide treatment, contextual bandit algorithms are especially useful since they are designed to take such information into account. The role of bandit algorithms in areas of precision medicine such as mobile health and digital phenotyping has been reviewed before (Tewari and Murphy, 2017; Rabbi et al., 2019). Since these reviews were published, bandit algorithms have continued to find uses in mobile health and several new topics have emerged in the research on bandit algorithms. This chapter is written for quantitative researchers in fields such as statistics, machine learning, and operations research who might be interested in knowing more about the algorithmic and mathematical details of bandit algorithms that have been used in mobile health.

研究动机与目标

• 为精准医疗与移动健康应用相关的bandit算法提供全面概述。
• 强调bandit算法中的基础与高级主题，包括非平稳性、鲁棒性、公平性与因果推断。
• 弥合bandit算法的理论进展与医疗决策中的实际挑战之间的鸿沟。
• 提出如核化多任务学习与自适应随机效应聚合等先进方法，以实现个性化治疗策略。
• 识别奖励设计、延迟结果与临床目标对齐等方面的开放研究挑战。

提出的方法

• 形式化多臂bandit（MAB）与上下文bandit框架，建模随时间推移的治疗选择与健康结果反馈。
• 引入独立同分布奖励的随机MAB设定，并将遗憾定义为最优奖励与累积奖励之间的差值。
• 应用上置信度界（UCB）与Thompson采样算法，以在治疗选择中平衡探索与利用。
• 提出KMTL-UCB，一种使用半正定核$\tilde{k}$建模任务与上下文相似性的核化多任务bandit算法。
• 开发IntelligentPooling，一种基于随机效应自适应聚合患者间数据的Thompson采样变体，依据与总体治疗效应的偏离程度进行调整。
• 推导理论遗憾边界：KMTL-UCB为$\tilde{\mathcal{O}}(\sqrt{Tr_zr_c})$，IntelligentPooling为$\tilde{\mathcal{O}}(dn\sqrt{T})$，其中$r_z$反映任务相似性，$d$为特征维度。

实验结果

研究问题

• RQ1 如何将bandit算法调整以利用患者特异性上下文，在精准医疗中优化个性化治疗决策？
• RQ2 在异质患者群体中，多任务与上下文bandit算法可提供哪些理论保证？
• RQ3 通过随机效应实现的患者数据自适应聚合，如何提升个性化治疗策略的学习效率与遗憾边界？
• RQ4 当前bandit框架在处理延迟治疗效应与奖励函数错位方面存在哪些局限？
• RQ5 如何使bandit算法在临床环境中对被污染奖励与决策约束更具鲁棒性？

主要发现

• KMTL-UCB算法实现$\tilde{\mathcal{O}}(\sqrt{Tr_zr_c})$的遗憾边界，其中$r_z$为任务相似性核的秩，表明当任务相似时样本效率更高。
• 当忽略任务相似性（$r_z = n$）时，边界退化为$\tilde{\mathcal{O}}(\sqrt{Tr_c})$，与标准上下文bandit性能一致。
• 当所有任务被聚合（$r_z = 1$）时，边界变为$\tilde{\mathcal{O}}(\sqrt{Tr_c}}$，表明共享学习具有优势。
• IntelligentPooling实现$\tilde{\mathcal{O}}(dn\sqrt{T})$的遗憾边界，表明其在患者数$n$与特征维度$d$上的可扩展性。
• 使用随机效应可实现自适应聚合，使反应异质的患者减少被聚合，从而提升个性化程度。
• 本文识别出奖励设计中的“对齐问题”为关键挑战，即即使遗憾较低，奖励错位仍可能削弱临床实用性。