[论文解读] The Statistical Complexity of Interactive Decision Making

Dylan J. Foster, Sham M. Kakade+2 more|arXiv (Cornell University)|Dec 27, 2021

Advanced Bandit Algorithms Research被引用 9

一句话总结

本文引入了决策-估计系数（DEC）作为样本高效交互式决策学习的基本复杂度度量，用于刻画此类学习的统计极限。本文提出了估计到决策（E2D）元算法，可将任意监督估计算法转化为在线决策策略，其遗憾上界与DEC的下界完全匹配，从而统一并优化了上下文Bandits、强化学习及结构化决策问题中的学习。

ABSTRACT

A fundamental challenge in interactive learning and decision making, ranging from bandit problems to reinforcement learning, is to provide sample-efficient, adaptive learning algorithms that achieve near-optimal regret. This question is analogous to the classical problem of optimal (supervised) statistical learning, where there are well-known complexity measures (e.g., VC dimension and Rademacher complexity) that govern the statistical complexity of learning. However, characterizing the statistical complexity of interactive learning is substantially more challenging due to the adaptive nature of the problem. The main result of this work provides a complexity measure, the Decision-Estimation Coefficient, that is proven to be both necessary and sufficient for sample-efficient interactive learning. In particular, we provide: 1. a lower bound on the optimal regret for any interactive decision making problem, establishing the Decision-Estimation Coefficient as a fundamental limit. 2. a unified algorithm design principle, Estimation-to-Decisions (E2D), which transforms any algorithm for supervised estimation into an online algorithm for decision making. E2D attains a regret bound that matches our lower bound up to dependence on a notion of estimation performance, thereby achieving optimal sample-efficient learning as characterized by the Decision-Estimation Coefficient. Taken together, these results constitute a theory of learnability for interactive decision making. When applied to reinforcement learning settings, the Decision-Estimation Coefficient recovers essentially all existing hardness results and lower bounds. More broadly, the approach can be viewed as a decision-theoretic analogue of the classical Le Cam theory of statistical estimation; it also unifies a number of existing approaches -- both Bayesian and frequentist.

研究动机与目标

• 建立自适应、序列反馈下交互式决策学习可学习性的统一理论。
• 识别出在交互式设置中实现样本高效学习所必需且充分的复杂度度量。
• 弥合统计估计理论与交互式决策学习之间的鸿沟，类似于监督学习中Le Cam理论的作用。
• 通过单一复杂度框架统一贝叶斯与频率学派方法在在线决策学习中的应用。

提出的方法

• 提出决策-估计系数（DEC）作为新的复杂度度量，用于量化交互式决策问题的内在难度。
• 引入估计到决策（E2D）元算法，将任意在线估计预言机映射为决策策略。
• 推导出E2D的遗憾上界，其与DEC刻画的下界完全匹配，证明了在估计误差范围内的最优性。
• 采用对偶视角与信息论工具，将E2D与后验抽样及乐观估计相联系。
• 将该框架应用于Bandits与强化学习，恢复已知的困难性结果，并将其推广至结构化函数类。
• 将该方法推广至上下文与模型无关设置，实现更广泛的应用。

实验结果

研究问题

• RQ1 交互式决策学习的基本统计复杂度是什么？如何对其进行形式化刻画？
• RQ2 是否存在一种单一的算法原则，可统一最优学习于各类交互式决策问题之中？
• RQ3 DEC与现有复杂度度量（如VC维、Bellman秩或可消除维数）之间有何关系？
• RQ4 E2D框架在强化学习与Bandit问题中能在多大程度上实现最优遗憾？
• RQ5 DEC能否作为一般交互学习设置下最优遗憾的下界？

主要发现

• 决策-估计系数（DEC）被证明是实现样本高效交互学习的必要且充分条件。
• E2D元算法实现的遗憾上界与DEC的下界完全匹配（仅存在估计误差），从而确立了最优性。
• 在表格型强化学习中，DEC恢复了已知的下界与困难性结果，包括基于Bellman秩与可消除维数的结果。
• 该框架统一了贝叶斯与频率学派方法，与后验抽样及乐观估计存在紧密联系。
• DEC推广了现有复杂度度量，并为统计估计理论提供了决策理论的类比，类似于Le Cam理论。
• 后续工作证实了DEC的普适性，已将其扩展至PAC决策学习、对抗性结果及多智能体系统。