[论文解读] Deep Reinforcement Learning for Crowdsourced Urban Delivery: System States Characterization, Heuristics-guided Action Choice, and Rule-Interposing Integration

Tanvir Ahamed, Bo Zou+2 more|arXiv (Cornell University)|Nov 29, 2020

Transportation and Mobility Innovations参考文献 39被引用 4

一句话总结

本文提出了一种深度强化学习（DRL）框架，用于优化众包城市配送，通过时空和容量数据对系统状态进行建模，结合启发式引导的动作选择以提高训练效率，并采用规则插入机制防止冗余路径探索。该方法在解决方案质量、收敛速度和可扩展性方面均优于现有启发式方法。

ABSTRACT

This paper investigates the problem of assigning shipping requests to ad hoc couriers in the context of crowdsourced urban delivery. The shipping requests are spatially distributed each with a limited time window between the earliest time for pickup and latest time for delivery. The ad hoc couriers, termed crowdsourcees, also have limited time availability and carrying capacity. We propose a new deep reinforcement learning (DRL)-based approach to tackling this assignment problem. A deep Q network (DQN) algorithm is trained which entails two salient features of experience replay and target network that enhance the efficiency, convergence, and stability of DRL training. More importantly, this paper makes three methodological contributions: 1) presenting a comprehensive and novel characterization of crowdshipping system states that encompasses spatial-temporal and capacity information of crowdsourcees and requests; 2) embedding heuristics that leverage the information offered by the state representation and are based on intuitive reasoning to guide specific actions to take, to preserve tractability and enhance efficiency of training; and 3) integrating rule-interposing to prevent repeated visiting of the same routes and node sequences during routing improvement, thereby further enhancing the training efficiency by accelerating learning. The effectiveness of the proposed approach is demonstrated through extensive numerical analysis. The results show the benefits brought by the heuristics-guided action choice and rule-interposing in DRL training, and the superiority of the proposed approach over existing heuristics in both solution quality, time, and scalability. Besides the potential to improve the efficiency of crowdshipping operation planning, the proposed approach also provides a new avenue and generic framework for other problems in the vehicle routing context.

研究动机与目标

• 解决在具有时间窗和容量限制的城市环境中，动态分配配送请求给临时骑手的挑战。
• 开发一种可扩展且高效的深度强化学习方法，用于众包物流中的实时配送分配。
• 通过结构化的状态表征和领域特定启发式方法，提升DRL训练的稳定性和收敛性。
• 通过规则插入机制防止学习过程中的冗余路径探索。
• 证明所提出方法在解决方案质量、计算时间与可扩展性方面优于传统启发式方法。

提出的方法

• 该方法采用带有经验回放和目标网络的深度Q网络（DQN），以稳定并加速DRL训练。
• 设计了全面的状态表征，用于编码骑手和配送请求的时空及容量信息。
• 基于直观的路径规划逻辑，将启发式方法嵌入动作选择过程，提升训练效率与可处理性。
• 集成规则插入机制，以检测并阻止路径优化过程中的重复节点序列和路径，减少探索冗余。
• 在具有动态请求到达和骑手可用性的模拟城市配送场景中，端到端训练DQN。
• 通过大量数值实验评估该框架，对比基线启发式方法在解决方案质量、训练速度和可扩展性方面的表现。

实验结果

研究问题

• RQ1 如何有效表征众包城市配送中的系统状态，以支持基于DRL的决策？
• RQ2 在复杂路径规划环境中，启发式引导的动作选择在多大程度上提升了DRL训练的效率与稳定性？
• RQ3 规则插入机制是否能减少DRL在车辆路径问题中的冗余探索并加速收敛？
• RQ4 所提出的DRL框架在解决方案质量、运行时间和可扩展性方面与传统启发式方法相比如何？
• RQ5 所提出方法是否可推广至众包配送以外的其他车辆路径问题？

主要发现

• 将启发式方法融入动作选择显著提升了DRL模型的训练效率与收敛速度。
• 规则插入机制减少了重复的路径探索，加速了学习过程并提高了样本效率。
• 所提出的DRL框架在解决方案质量方面优于现有启发式方法，尤其在复杂且动态的环境中表现更优。
• 该方法在配送请求和骑手数量增加时表现出更优的可扩展性。
• 状态表征、启发式引导与规则插入的结合，相比标准DQN，实现了更快的收敛速度与更稳定的训练过程。
• 实证结果证实了该框架在具有时间窗和容量约束的真实城市配送场景中的有效性。