[Paper Review] Multi-Vehicle Routing Problems with Soft Time Windows: A Multi-Agent Reinforcement Learning Approach

Ke Zhang, Meng Li+3 more|arXiv (Cornell University)|Feb 13, 2020

Vehicle Routing Optimization Methods28 references17 citations

TL;DR

This paper proposes a multi-agent reinforcement learning approach, Multi-Agent Attention Model (MAAM), for solving Multi-Vehicle Routing Problems with Soft Time Windows (MVRPSTW). By framing routing as an iterative tour generation process using an encoder-decoder architecture with attention mechanisms and training via multi-agent reinforcement learning with an unsupervised auxiliary network, the method achieves high solution quality with near-instant inference after offline training, outperforming Google OR-Tools and traditional heuristics on synthetic benchmarks with minimal computation time.

ABSTRACT

Multi-vehicle routing problem with soft time windows (MVRPSTW) is an indispensable constituent in urban logistics distribution systems. Over the past decade, numerous methods for MVRPSTW have been proposed, but most are based on heuristic rules that require a large amount of computation time. With the current rapid increase of logistics demands, traditional methods incur the dilemma between computational efficiency and solution quality. To efficiently solve the problem, we propose a novel reinforcement learning algorithm called the Multi-Agent Attention Model that can solve routing problem instantly benefit from lengthy offline training. Specifically, the vehicle routing problem is regarded as a vehicle tour generation process, and an encoder-decoder framework with attention layers is proposed to generate tours of multiple vehicles iteratively. Furthermore, a multi-agent reinforcement learning method with an unsupervised auxiliary network is developed for the model training. By evaluated on four synthetic networks with different scales, the results demonstrate that the proposed method consistently outperforms Google OR-Tools and traditional methods with little computation time. In addition, we validate the robustness of the well-trained model by varying the number of customers and the capacities of vehicles.

Motivation & Objective

• To address the growing computational challenge in urban logistics distribution systems where traditional heuristic methods for MVRPSTW are too slow under rising logistics demand.
• To develop a scalable, efficient routing solution that maintains high solution quality while drastically reducing computation time.
• To explore the feasibility of using deep reinforcement learning with attention mechanisms for complex, multi-agent routing problems with soft time windows.
• To validate the robustness of the trained model across varying customer counts and vehicle capacities.

Proposed method

• The vehicle routing problem is modeled as an iterative tour generation process using an encoder-decoder framework with multi-head attention layers to capture long-range dependencies in route planning.
• A multi-agent reinforcement learning framework is employed, where each vehicle acts as an independent agent, enabling decentralized decision-making and scalable training.
• An unsupervised auxiliary network is introduced to provide additional training signals, improving policy learning stability and convergence.
• The model is trained offline on synthetic networks of varying scales, enabling real-time inference during deployment.
• The attention mechanism allows the model to dynamically attend to relevant customers and time window constraints during route construction.
• The encoder processes the full state of the problem, while the decoder generates routes step-by-step with attention over unvisited customers.

Experimental results

Research questions

• RQ1 Can a multi-agent reinforcement learning approach with attention mechanisms effectively solve MVRPSTW with soft time windows while maintaining high solution quality?
• RQ2 How does the proposed method compare in performance and inference speed to traditional heuristic solvers like Google OR-Tools?
• RQ3 To what extent is the trained model robust to changes in the number of customers and vehicle capacities?
• RQ4 Can the unsupervised auxiliary network significantly improve training efficiency and policy performance in multi-agent routing?

Key findings

• The proposed MAAM method consistently outperformed Google OR-Tools and traditional heuristic methods across all four synthetic network benchmarks in terms of solution quality.
• The model achieved near-instant inference with minimal computation time after extensive offline training, demonstrating strong scalability and efficiency.
• The model maintained high performance and robustness when tested with varying numbers of customers and different vehicle capacities, indicating strong generalization.
• The inclusion of the unsupervised auxiliary network led to faster convergence and improved policy stability during training.
• The attention mechanism enabled effective long-range dependency modeling, resulting in more coherent and efficient route generation.