[论文解读] Variational inference for the multi-armed contextual bandit

Iñigo Urteaga, Chris H. Wiggins|arXiv (Cornell University)|Sep 10, 2017

Advanced Bandit Algorithms Research参考文献 24被引用 7

一句话总结

本文提出变分Thompson采样（VTS）用于上下文多臂老虎机问题，利用变分推断通过高斯混合模型近似复杂且未知的奖励分布。通过学习最小化真实奖励分布KL散度的变分参数，VTS即使在模型误设情况下也能显著降低累积遗憾，优于具有严格参数假设的标准Thompson采样。

ABSTRACT

In many biomedical, science, and engineering problems, one must sequentially decide which action to take next so as to maximize rewards. One general class of algorithms for optimizing interactions with the world, while simultaneously learning how the world operates, is the multi-armed bandit setting and, in particular, the contextual bandit case. In this setting, for each executed action, one observes rewards that are dependent on a given 'context', available at each interaction with the world. The Thompson sampling algorithm has recently been shown to enjoy provable optimality properties for this set of problems, and to perform well in real-world settings. It facilitates generative and interpretable modeling of the problem at hand. Nevertheless, the design and complexity of the model limit its application, since one must both sample from the distributions modeled and calculate their expected rewards. We here show how these limitations can be overcome using variational inference to approximate complex models, applying to the reinforcement learning case advances developed for the inference case in the machine learning community over the past two decades. We consider contextual multi-armed bandit applications where the true reward distribution is unknown and complex, which we approximate with a mixture model whose parameters are inferred via variational inference. We show how the proposed variational Thompson sampling approach is accurate in approximating the true distribution, and attains reduced regrets even with complex reward distributions. The proposed algorithm is valuable for practical scenarios where restrictive modeling assumptions are undesirable.

研究动机与目标

• 将Thompson采样从上下文老虎机设置中受限制的参数模型中拓展出来。
• 解决当真实分布复杂或未知时奖励建模中的模型误设问题。
• 通过混合模型实现对奖励分布的准确、可解释且灵活的建模。
• 通过变分推断学习复杂奖励结构，在不确定性下的序列决策中减少累积遗憾。
• 为上下文老虎机中的在线学习提供一种可扩展且原理严谨的方法，无需强参数假设。

提出的方法

• 使用层次贝叶斯混合模型表示上下文老虎机中复杂且未知的奖励分布。
• 应用具有平均场近似的变分推断，以学习高斯混合模型的参数。
• 通过最小化真实后验分布与变分近似之间的Kullback-Leibler散度，确保分布估计的准确性。
• 采用变分Thompson采样：从变分后验中采样，以平衡探索与利用。
• 通过证据下界（ELBO）优化推导出变分参数的闭式更新方程。
• 通过在新上下文-奖励对到达时逐步更新变分参数，实现在线学习。

实验结果

研究问题

• RQ1 变分推断能否有效用于近似上下文老虎机中复杂的非指数族奖励分布？
• RQ2 与模型误设下的标准Thompson采样相比，变分Thompson采样是否能降低累积遗憾？
• RQ3 在复杂奖励场景中，模型灵活性（如混合成分数量）如何影响遗憾与估计精度？
• RQ4 当真实奖励分布不平衡或重叠时，所提方法能否学习到准确的期望奖励估计？
• RQ5 在在线、序列决策中，模型复杂性与遗憾性能之间存在何种权衡？

主要发现

• 在奖励混合分布不平衡且重叠的场景B中，K=3成分的VTS在t=500时相比K=1的VTS将累积遗憾降低了40%。
• K=3的VTS方法在性能上与真实模型（K=2）相当，表明其对模型复杂性的鲁棒性。
• 在复杂场景中，K=2和K=3的VTS相比K=1的VTS，期望奖励估计的均方误差显著更低，表明学习精度更高。
• 即使在简单场景（场景A）中，K=1的VTS表现良好，但更高K的模型在复杂性下展现出更优性能。
• 变分近似成功最小化了KL散度，实现了准确的后验采样并提升了决策质量。
• 该方法在多种奖励结构下均保持低遗憾，展现出强大的泛化能力，且无需强参数假设。