[Paper Review] Solving the single-track train scheduling problem via Deep Reinforcement Learning

Valerio Agasucci, Giorgio Grani+1 more|arXiv (Cornell University)|Sep 1, 2020

Railway Systems and Energy Efficiency16 references4 citations

TL;DR

This paper proposes a Deep Q-Network (DQN)-based reinforcement learning approach to solve the single-track train dispatching problem, comparing decentralized (per-train agents) and centralized (global observation with graph neural networks) methods. The centralized DQN outperforms traditional linear Q-learning and a mixed-integer programming (MILP) formulation, especially on larger, complex instances, reducing average delay by up to 50% in high-complexity test sets.

ABSTRACT

Every day, railways experience disturbances and disruptions, both on the network and the fleet side, that affect the stability of rail traffic. Induced delays propagate through the network, which leads to a mismatch in demand and offer for goods and passengers, and, in turn, to a loss in service quality. In these cases, it is the duty of human traffic controllers, the so-called dispatchers, to do their best to minimize the impact on traffic. However, dispatchers inevitably have a limited depth of perception of the knock-on effect of their decisions, particularly how they affect areas of the network that are outside their direct control. In recent years, much work in Decision Science has been devoted to developing methods to solve the problem automatically and support the dispatchers in this challenging task. This paper investigates Machine Learning-based methods for tackling this problem, proposing two different Deep Q-Learning methods(Decentralized and Centralized). Numerical results show the superiority of these techniques with respect to the classical linear Q-Learning based on matrices. Moreover, the Centralized approach is compared with a MILP formulation showing interesting results. The experiments are inspired by data provided by a U.S. Class 1 railroad.

Motivation & Objective

• To address the real-time, online train dispatching problem under network disruptions, where human dispatchers face limited perception of knock-on effects.
• To investigate whether Deep Reinforcement Learning can outperform classical optimization and linear Q-learning methods in solving the single-track train scheduling problem.
• To evaluate the scalability and performance of decentralized vs. centralized DQN approaches on realistic, large-scale railway instances inspired by U.S. Class 1 railroads.
• To compare the deep RL approach with a standard MILP formulation in terms of solution quality and computational efficiency under time constraints.

Proposed method

• The paper formulates the train dispatching problem as a Markov Decision Process (MDP), where actions correspond to rescheduling or rerouting decisions.
• Two DQN variants are proposed: a decentralized approach where each train acts as an independent agent with local observation of nearby tracks, and a centralized approach with global network visibility.
• The centralized method employs a Graph Neural Network (GNN) to encode the rail network state and estimate Q-values, enabling generalization across different network sizes.
• The DQN agents are trained using experience replay and target networks to stabilize learning, with sparse, delay-based shaping rewards.
• The approach is evaluated on five test sets with varying numbers of trains and resources (tracks/stations), using a time limit of 10 minutes per instance.
• Performance is benchmarked against a MILP formulation using metrics including solved instances, average delay, and solution quality on overlapping problem sets.

Experimental results

Research questions

• RQ1 Can Deep Q-Learning outperform classical linear Q-learning in solving the single-track train dispatching problem?
• RQ2 How does the centralized DQN with GNN-based state representation compare to a MILP formulation in terms of solution quality and scalability?
• RQ3 Does the centralized DQN generalize effectively to larger railway networks without retraining?
• RQ4 How do the decentralized and centralized DQN approaches differ in performance and robustness across increasing problem complexity?
• RQ5 To what extent does the RL approach reduce average delay compared to MILP on large, complex instances?

Key findings

• The centralized DQN approach reduced average delay by up to 50% compared to the MILP formulation on the most complex test set (50 trains, 196 resources), where MILP's average delay was 32,504 and RL's was 16,379.
• On the 50 T 185 R test set, the MILP average delay was 150% higher than the RL approach, demonstrating a significant performance gap under high complexity.
• The centralized DQN solved 43 out of 50 instances within the 10-minute time limit, while the MILP solved only 41, indicating comparable or better feasibility under time constraints.
• The centralized method generalized well to larger networks, maintaining low delay even when the number of resources increased from 137 to 196, unlike the MILP which saw a sharp rise in average delay.
• The decentralized DQN showed strong performance on smaller instances but struggled with scalability, particularly on larger networks, due to limited perception of global network state.
• The RL-based methods consistently found lower average delays than MILP on overlapping solved instances, especially as problem size increased, highlighting the advantage of learning-based global planning.