[论文解读] The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond

Jiin Woo, Gauri Joshi+1 more|arXiv (Cornell University)|May 18, 2023

Reinforcement Learning in Robotics被引用 4

一句话总结

该论文提出了一种新颖的联邦Q-learning算法，采用重要性平均机制，在样本复杂度方面实现了稳健的线性加速，即使在本地行为策略高度异质的情况下亦如此。通过根据状态-动作访问频率对本地Q估计进行加权，该方法消除了对各智能体间均匀覆盖的要求，揭示了‘异质性之福’，并实现了异步、去中心化强化学习设置下的高效协作学习。

ABSTRACT

When the data used for reinforcement learning (RL) are collected by multiple agents in a distributed manner, federated versions of RL algorithms allow collaborative learning without the need for agents to share their local data. In this paper, we consider federated Q-learning, which aims to learn an optimal Q-function by periodically aggregating local Q-estimates trained on local data alone. Focusing on infinite-horizon tabular Markov decision processes, we provide sample complexity guarantees for both the synchronous and asynchronous variants of federated Q-learning. In both cases, our bounds exhibit a linear speedup with respect to the number of agents and near-optimal dependencies on other salient problem parameters. In the asynchronous setting, existing analyses of federated Q-learning, which adopt an equally weighted averaging of local Q-estimates, require that every agent covers the entire state-action space. In contrast, our improved sample complexity scales inverse proportionally to the minimum entry of the average stationary state-action occupancy distribution of all agents, thus only requiring the agents to collectively cover the entire state-action space, unveiling the blessing of heterogeneity in enabling collaborative learning by relaxing the coverage requirement of the single-agent case. However, its sample complexity still suffers when the local trajectories are highly heterogeneous. In response, we propose a novel federated Q-learning algorithm with importance averaging, giving larger weights to more frequently visited state-action pairs, which achieves a robust linear speedup as if all trajectories are centrally processed, regardless of the heterogeneity of local behavior policies.

研究动机与目标

• 解决现有联邦Q-learning方法在所有智能体间对状态-动作空间实现均匀覆盖的局限性。
• 分析在异质数据分布下，同步与异步联邦Q-learning的样本复杂度。
• 开发一种鲁棒算法，无论本地策略异质性如何，均能保持线性加速。
• 揭示‘异质性之福’——即智能体数据的多样性可提升学习效率而非造成阻碍。

提出的方法

• 提出一种联邦Q-learning框架，通过周期性地在K个智能体之间平均本地Q估计，并由中央服务器聚合更新。
• 引入重要性平均，即根据各智能体间状态-动作对的访问频率对本地Q估计进行加权。
• 推导出样本复杂度界，其与平均平稳状态-动作占据分布的最小元素成反比。
• 利用集中不等式和鞅论证，为同步与异步设置建立理论保证。
• 采用一种新颖的误差项分解方法，结合值函数差异与转移动态，以界定收敛性。
• 采用两阶段分析：一阶段用于早期收敛，另一阶段用于误差衰减，利用几何级数与指数衰减界。

实验结果

研究问题

• RQ1 当智能体具有异质数据分布时，联邦Q-learning能否在样本复杂度上实现线性加速？
• RQ2 先前方法对完整状态-动作空间覆盖的要求是否限制了可扩展性，且能否被放宽？
• RQ3 重要性平均能否提升异步联邦Q-learning中对本地策略异质性的鲁棒性？
• RQ4 在异步采样且智能体异质的条件下，联邦Q-learning的理论样本复杂度是多少？
• RQ5 ‘异质性之福’如何体现为整体样本复杂度的降低？

主要发现

• 所提出的 importance averaging 方法在样本复杂度上实现了线性加速，其效果等同于所有轨迹被集中处理，且不受本地策略异质性影响。
• 样本复杂度与平均平稳状态-动作占据分布的最小元素成反比，从而无需每个智能体都覆盖整个状态-动作空间。
• 在异步设置中，该方法实现了近乎最优的样本复杂度，其对问题参数的依赖关系在对数因子范围内达到最优。
• 理论分析表明，误差随交互次数呈指数衰减，其界取决于智能体间平均访问频率。
• 在本地策略异质性较高的场景中，该方法优于等权重平均基线，后者无法实现线性加速。
• 实验结果证实，即使本地轨迹在各智能体间高度非均匀，重要性平均仍能保持线性加速。