[论文解读] Reinforcement Learning in the Wild: Scalable RL Dispatching Algorithm Deployed in Ridehailing Marketplace

Soheil Sadeghi Eshkevari, Xiaocheng Tang+5 more|arXiv (Cornell University)|Feb 10, 2022

Transportation and Mobility Innovations被引用 5

一句话总结

本文提出了一种可扩展的、实时的强化学习（RL）派单算法，用于网约车平台，该算法采用时序差分（TD）值更新和自适应机制，以优化司机-订单匹配。在A/B测试中，该方法使司机收入提升了1.3%，在因果分析中最高实现5.3%的性能提升，证明了其在真实市场中的稳健性和大规模部署能力。

ABSTRACT

In this study, a real-time dispatching algorithm based on reinforcement learning is proposed and for the first time, is deployed in large scale. Current dispatching methods in ridehailing platforms are dominantly based on myopic or rule-based non-myopic approaches. Reinforcement learning enables dispatching policies that are informed of historical data and able to employ the learned information to optimize returns of expected future trajectories. Previous studies in this field yielded promising results, yet have left room for further improvements in terms of performance gain, self-dependency, transferability, and scalable deployment mechanisms. The present study proposes a standalone RL-based dispatching solution that is equipped with multiple mechanisms to ensure robust and efficient on-policy learning and inference while being adaptable for full-scale deployment. A new form of value updating based on temporal difference is proposed that is more adapted to the inherent uncertainty of the problem. For the driver-order assignment, a customized utility function is proposed that when tuned based on the statistics of the market, results in remarkable performance improvement and interpretability. In addition, for reducing the risk of cancellation after drivers' assignment, an adaptive graph pruning strategy based on the multi-arm bandit problem is introduced. The method is evaluated using offline simulation with real data and yields notable performance improvement. In addition, the algorithm is deployed online in multiple cities under DiDi's operation for A/B testing and is launched in one of the major international markets as the primary mode of dispatch. The deployed algorithm shows over 1.3% improvement in total driver income from A/B testing. In addition, by causal inference analysis, as much as 5.3% improvement in major performance metrics is detected after full-scale deployment.

研究动机与目标

• 开发一种可扩展的、实时的强化学习（RL）派单系统，用于网约车平台，且无需离线预训练。
• 通过在线TD学习引入长期价值估计和历史数据，以克服短视和基于规则的派单方法的局限性。
• 通过设计定制化的效用函数和自适应图剪枝，降低取消风险，从而提升司机收入和服务完成率。
• 通过新颖机制（如自适应学习率和奖励平滑）确保高效且稳健的策略内学习与推理。
• 通过离线仿真、A/B测试和在主要国际市场的全规模部署，验证该算法的有效性。

提出的方法

• 采用在线时序差分（TD）学习，实时更新状态值，用确定性的TD更新替代期望回报，以考虑行程完成结果的影响。
• 引入自适应学习率，根据状态访问频率和更新幅度缩放更新量，降低值估计的方差。
• 应用奖励平滑机制，缓解即时强化学习信号中的高方差，提升训练稳定性。
• 设计定制化的效用函数，对独立分量（如距离、价格）进行标准化并动态加权，用于司机-订单匹配的二分图匹配。
• 实施基于反馈控制的多臂老虎机策略，动态设定图剪枝阈值，根据全局性能指标调整派单激进程度，从而减少派单后取消。
• 集成两阶段派单流水线：首先使用效用函数进行最大二分图匹配，随后通过TD方法进行值更新，并通过反馈回路调节派单失败率。

实验结果

研究问题

• RQ1 一种实时的、在线的基于强化学习的派单系统是否能在大规模网约车运营中超越传统的基于规则和短视的方法？
• RQ2 时序差分学习如何适应现实派单中行程完成的不确定性和延迟奖励？
• RQ3 与静态边权重相比，定制化的效用函数和自适应剪枝在多大程度上能提升司机收入和完成率？
• RQ4 在真实市场中，部署基于强化学习的派单系统对关键性能指标的因果影响是什么？
• RQ5 该强化学习系统的性能在需求密度和城市结构各异的城市中如何扩展？

主要发现

• 在多个大型城市的A/B测试中，部署的基于RL的派单算法使司机总收入提升了1.3%。
• 使用合成控制和双重差分法的因果推断分析显示，在主要国际市场的全规模部署后，关键性能指标平均提升了5.3%。
• 离线仿真表明，所提出的RLW算法在大多数测试城市中优于当前最先进的基线方法。
• 在较小城市中，该算法表现出边际或轻微的性能下降，归因于需求稀疏性和信号-噪声比不稳定的因素。
• 基于反馈控制的多臂老虎机机制有效减少了派单后的取消率，通过根据实时性能动态调整图剪枝阈值。
• 自适应学习率与奖励平滑机制的集成显著提升了高不确定性环境下的训练稳定性和值函数收敛性。