[Paper Review] Progress and summary of reinforcement learning on energy management of MPS-EV

Jincheng Hu, Lin Yang+5 more|arXiv (Cornell University)|Nov 8, 2022

Electric Vehicles and Infrastructure4 citations

TL;DR

This paper provides a comprehensive review and systematic analysis of reinforcement learning (RL)-based energy management strategies (RL-EMS) in multi-power-source electric vehicles (MPS-EVs). It examines key design elements—algorithms, perception and decision schemes, reward functions, and training methods—identifying gaps between advanced RL techniques and current RL-EMS applications, and proposes future directions for integrating advanced AI into EMS.

ABSTRACT

The high emission and low energy efficiency caused by internal combustion engines (ICE) have become unacceptable under environmental regulations and the energy crisis. As a promising alternative solution, multi-power source electric vehicles (MPS-EVs) introduce different clean energy systems to improve powertrain efficiency. The energy management strategy (EMS) is a critical technology for MPS-EVs to maximize efficiency, fuel economy, and range. Reinforcement learning (RL) has become an effective methodology for the development of EMS. RL has received continuous attention and research, but there is still a lack of systematic analysis of the design elements of RL-based EMS. To this end, this paper presents an in-depth analysis of the current research on RL-based EMS (RL-EMS) and summarizes the design elements of RL-based EMS. This paper first summarizes the previous applications of RL in EMS from five aspects: algorithm, perception scheme, decision scheme, reward function, and innovative training method. The contribution of advanced algorithms to the training effect is shown, the perception and control schemes in the literature are analyzed in detail, different reward function settings are classified, and innovative training methods with their roles are elaborated. Finally, by comparing the development routes of RL and RL-EMS, this paper identifies the gap between advanced RL solutions and existing RL-EMS. Finally, this paper suggests potential development directions for implementing advanced artificial intelligence (AI) solutions in EMS.

Motivation & Objective

• To systematically analyze the design elements of reinforcement learning-based energy management strategies (RL-EMS) in multi-power-source electric vehicles (MPS-EVs).
• To evaluate the role of advanced RL algorithms, perception schemes, decision mechanisms, reward functions, and innovative training methods in improving EMS performance.
• To identify the gap between state-of-the-art RL solutions and existing RL-EMS implementations in MPS-EVs.
• To propose actionable development directions for integrating advanced artificial intelligence into energy management systems for enhanced efficiency and range.

Proposed method

• Conducts a structured review of 142 RL-EMS studies published between 2010 and 2022, focusing on algorithmic, perception, decision, reward, and training method components.
• Classifies RL algorithms used in EMS into deep Q-networks (DQN), proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and other deep RL methods.
• Analyzes perception schemes based on state observation methods, including full-state, partial-state, and sensor-based inputs.
• Categorizes decision schemes into online and offline control strategies, with emphasis on real-time applicability.
• Classifies reward functions into fuel economy, energy efficiency, and combined objectives, highlighting trade-offs in design.
• Reviews innovative training methods such as curriculum learning, transfer learning, and multi-agent training, assessing their impact on convergence and performance.

Experimental results

Research questions

• RQ1 How do different RL algorithms influence the training stability and performance of energy management strategies in MPS-EVs?
• RQ2 What are the most effective perception and decision schemes for real-time energy management in RL-EMS?
• RQ3 How do varying reward function designs affect fuel economy and energy efficiency in RL-EMS?
• RQ4 What role do innovative training methods play in improving sample efficiency and generalization in RL-EMS?
• RQ5 What are the key gaps between advanced RL techniques and current RL-EMS implementations in MPS-EVs?

Key findings

• Deep reinforcement learning algorithms such as PPO and DQN show strong performance in energy management, with PPO demonstrating better sample efficiency and stability.
• Reward functions that combine fuel economy and battery degradation show improved long-term efficiency compared to single-objective rewards.
• Partial-state perception schemes are widely used due to practical sensor limitations, but full-state observation leads to better performance in simulation.
• Innovative training methods like curriculum learning and transfer learning significantly reduce training time and improve convergence speed.
• Despite advances in RL, most RL-EMS implementations still rely on simplified environments and do not fully leverage modern RL techniques.
• A clear gap exists between state-of-the-art RL solutions and their application in real-world MPS-EV energy management, particularly in generalization and robustness.