[Paper Review] Smart Train Operation Algorithms based on Expert Knowledge and Reinforcement Learning

Kaichen Zhou, Shiji Song+3 more|arXiv (Cornell University)|Mar 6, 2020

Railway Systems and Energy Efficiency37 references4 citations

TL;DR

This paper proposes two smart train operation algorithms—STOD (based on Deep Deterministic Policy Gradient) and STON (based on Normalized Advantage Function)—that integrate expert driving knowledge with reinforcement learning to optimize energy efficiency, punctuality, and riding comfort in subway systems. The algorithms achieve superior performance over manual driving and existing ATO systems, demonstrating flexibility and robustness under varying trip times and resistance conditions, with STON showing slightly better overall results in simulations using real Beijing Yizhuang Line data.

ABSTRACT

During recent decades, the automatic train operation (ATO) system has been gradually adopted in many subway systems for its low-cost and intelligence. This paper proposes two smart train operation algorithms by integrating the expert knowledge with reinforcement learning algorithms. Compared with previous works, the proposed algorithms can realize the control of continuous action for the subway system and optimize multiple critical objectives without using an offline speed profile. Firstly, through learning historical data of experienced subway drivers, we extract the expert knowledge rules and build inference methods to guarantee the riding comfort, the punctuality, and the safety of the subway system. Then we develop two algorithms for optimizing the energy efficiency of train operation. One is the smart train operation (STO) algorithm based on deep deterministic policy gradient named (STOD) and the other is the smart train operation algorithm based on normalized advantage function (STON). Finally, we verify the performance of proposed algorithms via some numerical simulations with the real field data from the Yizhuang Line of the Beijing Subway and illustrate that the developed smart train operation algorithm are better than expert manual driving and existing ATO algorithms in terms of energy efficiency. Moreover, STOD and STON can adapt to different trip times and different resistance conditions.

Motivation & Objective

• To develop intelligent train operation algorithms that optimize energy efficiency, punctuality, and riding comfort in subway systems.
• To address the limitations of existing ATO systems by enabling continuous action control without relying on precomputed offline speed profiles.
• To integrate expert knowledge from experienced drivers into reinforcement learning frameworks to improve stability and real-world applicability.
• To evaluate the flexibility and robustness of the proposed algorithms under varying trip times and dynamic resistance conditions.
• To demonstrate that model-free reinforcement learning enhanced with expert inference can outperform both manual driving and conventional ATO methods.

Proposed method

• Extracted expert knowledge rules from historical driving data of experienced subway drivers to model comfort, safety, and punctuality.
• Developed a knowledge-informed inference mechanism to embed driver expertise into the reinforcement learning policy network.
• Proposed STOD using Deep Deterministic Policy Gradient (DDPG) for continuous action control in energy-efficient train operation.
• Proposed STON using the Normalized Advantage Function (NAF) to improve sample efficiency and policy stability in continuous control.
• Integrated expert rules into the reward function and policy network to ensure safety and comfort constraints are met during training.
• Trained and validated the algorithms using real field data from the Yizhuang Line of the Beijing Subway under multiple operational scenarios.

Experimental results

Research questions

• RQ1 Can reinforcement learning algorithms enhanced with expert knowledge achieve better energy efficiency than manual driving and existing ATO systems?
• RQ2 How do STOD and STON perform in terms of comfort, punctuality, and safety when operating under varying trip times?
• RQ3 To what extent can the proposed algorithms adapt to changes in track resistance, such as those caused by weather or aging infrastructure?
• RQ4 How does the integration of expert knowledge improve the stability and convergence of deep reinforcement learning in continuous control tasks for train operation?
• RQ5 Which of the two proposed algorithms—STOD or STON—demonstrates superior performance and robustness across diverse operational conditions?

Key findings

• STOD and STON significantly outperformed expert manual driving in energy efficiency, with STON achieving the lowest energy consumption in all test cases.
• Both algorithms maintained punctuality and safety under all tested conditions, with time errors within the allowable 3-second threshold.
• STON demonstrated better overall performance than STOD, particularly in energy efficiency and comfort, despite a 2-second longer trip time in one scenario.
• The algorithms exhibited strong robustness under changed gradient conditions, maintaining stable performance even when resistance varied due to environmental or infrastructure changes.
• The integration of expert knowledge improved policy stability and reduced the risk of unsafe actions, enabling reliable deployment in real-world settings.
• The proposed models showed flexibility in handling different trip times, with STOD and STON generating rational control strategies across a range of planning durations.