[Paper Review] Deep Reinforcement Learning for Crowdsourced Urban Delivery: System States Characterization, Heuristics-guided Action Choice, and Rule-Interposing Integration

Tanvir Ahamed, Bo Zou+2 more|arXiv (Cornell University)|Nov 29, 2020

Transportation and Mobility Innovations39 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL) framework for optimizing crowdsourced urban delivery by modeling system states with spatial-temporal and capacity data, integrating heuristic-guided action selection for training efficiency, and applying rule-interposing to prevent redundant route exploration. The approach achieves superior solution quality, faster convergence, and better scalability than existing heuristics.

ABSTRACT

This paper investigates the problem of assigning shipping requests to ad hoc couriers in the context of crowdsourced urban delivery. The shipping requests are spatially distributed each with a limited time window between the earliest time for pickup and latest time for delivery. The ad hoc couriers, termed crowdsourcees, also have limited time availability and carrying capacity. We propose a new deep reinforcement learning (DRL)-based approach to tackling this assignment problem. A deep Q network (DQN) algorithm is trained which entails two salient features of experience replay and target network that enhance the efficiency, convergence, and stability of DRL training. More importantly, this paper makes three methodological contributions: 1) presenting a comprehensive and novel characterization of crowdshipping system states that encompasses spatial-temporal and capacity information of crowdsourcees and requests; 2) embedding heuristics that leverage the information offered by the state representation and are based on intuitive reasoning to guide specific actions to take, to preserve tractability and enhance efficiency of training; and 3) integrating rule-interposing to prevent repeated visiting of the same routes and node sequences during routing improvement, thereby further enhancing the training efficiency by accelerating learning. The effectiveness of the proposed approach is demonstrated through extensive numerical analysis. The results show the benefits brought by the heuristics-guided action choice and rule-interposing in DRL training, and the superiority of the proposed approach over existing heuristics in both solution quality, time, and scalability. Besides the potential to improve the efficiency of crowdshipping operation planning, the proposed approach also provides a new avenue and generic framework for other problems in the vehicle routing context.

Motivation & Objective

• To address the challenge of dynamically assigning shipping requests to ad hoc couriers in urban environments with time windows and capacity limits.
• To develop a scalable and efficient deep reinforcement learning approach for real-time delivery assignment in crowdsourced logistics.
• To enhance DRL training stability and convergence through structured state representation and domain-specific heuristics.
• To prevent redundant route exploration during learning via rule-interposing mechanisms.
• To demonstrate the superiority of the proposed method over traditional heuristics in solution quality, computation time, and scalability.

Proposed method

• The method employs a Deep Q-Network (DQN) with experience replay and a target network to stabilize and accelerate DRL training.
• A comprehensive state representation is designed to encode spatial, temporal, and capacity information of both couriers and delivery requests.
• Heuristics are embedded into the action selection process based on intuitive routing logic, improving training efficiency and tractability.
• Rule-interposing is integrated to detect and block repeated node sequences and routes during routing improvement, reducing exploration redundancy.
• The DQN is trained end-to-end on simulated urban delivery scenarios with dynamic request arrivals and courier availability.
• The framework is evaluated using extensive numerical experiments comparing solution quality, training speed, and scalability against baseline heuristics.

Experimental results

Research questions

• RQ1 How can system states in crowdsourced urban delivery be effectively characterized to support DRL-based decision-making?
• RQ2 To what extent do heuristic-guided action choices improve the efficiency and stability of DRL training in complex routing environments?
• RQ3 Can rule-interposing mechanisms reduce redundant exploration and accelerate convergence in DRL for vehicle routing?
• RQ4 How does the proposed DRL framework compare to traditional heuristics in terms of solution quality, runtime, and scalability?
• RQ5 Can the proposed method be generalized to other vehicle routing problems beyond crowdsourced delivery?

Key findings

• The integration of heuristics into action selection significantly improves training efficiency and convergence speed of the DRL model.
• Rule-interposing reduces repeated route exploration, accelerating learning and enhancing sample efficiency.
• The proposed DRL framework achieves higher solution quality than existing heuristics, particularly in complex, dynamic environments.
• The method demonstrates superior scalability with increasing numbers of delivery requests and couriers.
• The combination of state characterization, heuristic guidance, and rule-interposing leads to faster convergence and more stable training compared to standard DQN.
• Empirical results confirm the framework's effectiveness in real-world-like urban delivery scenarios with time windows and capacity constraints.