[Paper Review] Learning to Optimize Industry-Scale Dynamic Pickup and Delivery Problems

Li Xijun, Weilin Luo+6 more|arXiv (Cornell University)|May 27, 2021

Transportation and Mobility Innovations24 references4 citations

TL;DR

This paper proposes ST-DDGN, a data-driven deep reinforcement learning framework that combines spatial-temporal demand forecasting with graph-based value function learning using attention-based graph neural networks and Double DQN to solve large-scale dynamic pickup and delivery problems. The method reduces vehicle usage by 11.27% and total transportation cost by 13.12% compared to strong baselines on real-world industrial data.

ABSTRACT

The Dynamic Pickup and Delivery Problem (DPDP) is aimed at dynamically scheduling vehicles among multiple sites in order to minimize the cost when delivery orders are not known a priori. Although DPDP plays an important role in modern logistics and supply chain management, state-of-the-art DPDP algorithms are still limited on their solution quality and efficiency. In practice, they fail to provide a scalable solution as the numbers of vehicles and sites become large. In this paper, we propose a data-driven approach, Spatial-Temporal Aided Double Deep Graph Network (ST-DDGN), to solve industry-scale DPDP. In our method, the delivery demands are first forecast using spatial-temporal prediction method, which guides the neural network to perceive spatial-temporal distribution of delivery demand when dispatching vehicles. Besides, the relationships of individuals such as vehicles are modelled by establishing a graph-based value function. ST-DDGN incorporates attention-based graph embedding with Double DQN (DDQN). As such, it can make the inference across vehicles more efficiently compared with traditional methods. Our method is entirely data driven and thus adaptive, i.e., the relational representation of adjacent vehicles can be learned and corrected by ST-DDGN from data periodically. We have conducted extensive experiments over real-world data to evaluate our solution. The results show that ST-DDGN reduces 11.27% number of the used vehicles and decreases 13.12% total transportation cost on average over the strong baselines, including the heuristic algorithm deployed in our UAT (User Acceptance Test) environment and a variety of vanilla DRL methods. We are due to fully deploy our solution into our online logistics system and it is estimated that millions of USD logistics cost can be saved per year.

Motivation & Objective

• To address the limitations of traditional heuristic and optimization-based methods in solving large-scale, dynamic pickup and delivery problems under real-world uncertainty.
• To develop a scalable, data-driven reinforcement learning framework that captures spatial-temporal demand patterns and vehicle interactions.
• To improve solution quality and efficiency by integrating spatial-temporal forecasting with relational modeling of vehicles via graph networks.
• To enable end-to-end learning of dispatching policies that adapt to dynamic, real-time order arrivals and complex enterprise constraints.
• To achieve superior performance over strong baselines, including heuristic methods and vanilla DRL approaches, in real industrial deployment settings.

Proposed method

• The method formulates DPDP as a route-centric Markov Decision Process (MDP) for deep reinforcement learning.
• It employs a spatial-temporal prediction model to forecast delivery demand distribution across 27 factories and 144 time intervals, guiding policy learning.
• A graph-based value function models relationships among vehicles using attention-based graph neural networks to capture cooperation and competition.
• The framework combines Double DQN (DDQN) with graph embedding to stabilize training and improve policy generalization.
• The ST Score, derived from spatial-temporal demand and capacity distributions, guides the agent toward balancing supply and demand across space and time.
• The model is trained end-to-end using real-world logistics data, with policy updates guided by both immediate rewards and long-term spatial-temporal balance.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning framework effectively learn to dispatch vehicles in large-scale, dynamic pickup and delivery scenarios with uncertain future orders?
• RQ2 How does integrating spatial-temporal demand forecasting improve the performance and convergence speed of a graph-based DRL policy in vehicle dispatching?
• RQ3 To what extent can relational modeling of vehicles via graph networks enhance coordination and reduce sub-optimality in multi-vehicle routing?
• RQ4 Does the introduction of an ST Score metric that aligns supply and demand distributions lead to faster and more effective policy learning?
• RQ5 How does the proposed ST-DDGN framework compare in performance and scalability to heuristic and vanilla DRL baselines in real industrial settings?

Key findings

• ST-DDGN reduces the number of vehicles used by 11.27% on average compared to strong baselines, including the heuristic algorithm used in the UAT environment.
• The method decreases the total transportation cost by 13.12% compared to the same baselines, demonstrating significant cost savings.
• ST-DDGN converges 50 episodes faster than DDGN and 100 episodes faster than DDQN, showing accelerated learning due to the ST Score guidance.
• The Frobenius norm between predicted demand and actual vehicle capacity distributions decreases fastest and reaches the lowest value (236.8) with ST-DDGN, indicating superior spatial-temporal alignment.
• The model learns to dispatch vehicles to high-demand grids and enables 'hitchhiking' by postponing some orders to serve them with future ones, suggesting foresight in decision-making.
• The framework is estimated to save millions of USD in logistics costs annually upon full deployment in Huawei’s online logistics system.