[Paper Review] Human-like Energy Management Based on Deep Reinforcement Learning and Historical Driving Experiences

Chen, Hao, Xiaolin Tang+2 more|arXiv (Cornell University)|Jul 16, 2020

Electric and Hybrid Vehicle Technologies31 references18 citations

TL;DR

This paper proposes a human-like energy management strategy for series-parallel hybrid electric vehicles using deep reinforcement learning (DDPG) trained on historical driving data from experienced drivers. By replacing dynamic programming-based expert demonstrations with real-world driving patterns, the method achieves improved fuel economy and faster convergence while maintaining near-optimal performance.

ABSTRACT

Development of hybrid electric vehicles depends on an advanced and efficient energy management strategy (EMS). With online and real-time requirements in mind, this article presents a human-like energy management framework for hybrid electric vehicles according to deep reinforcement learning methods and collected historical driving data. The hybrid powertrain studied has a series-parallel topology, and its control-oriented modeling is founded first. Then, the distinctive deep reinforcement learning (DRL) algorithm, named deep deterministic policy gradient (DDPG), is introduced. To enhance the derived power split controls in the DRL framework, the global optimal control trajectories obtained from dynamic programming (DP) are regarded as expert knowledge to train the DDPG model. This operation guarantees the optimality of the proposed control architecture. Moreover, the collected historical driving data based on experienced drivers are employed to replace the DP-based controls, and thus construct the human-like EMSs. Finally, different categories of experiments are executed to estimate the optimality and adaptability of the proposed human-like EMS. Improvements in fuel economy and convergence rate indicate the effectiveness of the constructed control structure.

Motivation & Objective

• To develop a real-time, human-like energy management strategy (EMS) for hybrid electric vehicles (HEVs) that mimics experienced drivers' behavior.
• To replace traditional dynamic programming (DP)-based expert demonstrations with actual historical driving data to enhance realism and adaptability.
• To improve fuel economy and convergence speed in energy management without sacrificing optimality.
• To validate the proposed DRL-based EMS under diverse driving conditions using real-world data.

Proposed method

• A control-oriented model of a series-parallel hybrid powertrain is developed as the simulation environment.
• The deep deterministic policy gradient (DDPG) algorithm is employed to learn the optimal power split policy.
• Historical driving data from experienced drivers are used to generate expert demonstrations instead of relying on DP solutions.
• The DDPG agent is trained using these human-driven trajectories to learn human-like control behaviors.
• The framework ensures near-optimality by leveraging the high-quality behavior demonstrated in historical data.
• The trained policy is evaluated under various driving cycles to assess adaptability and performance.

Experimental results

Research questions

• RQ1 Can a DRL-based energy management system trained on real human driving data achieve fuel economy comparable to dynamic programming while being real-time applicable?
• RQ2 How does replacing DP-based expert demonstrations with real driver data affect the convergence speed and robustness of the DRL policy?
• RQ3 To what extent does the human-like control behavior improve the adaptability of the EMS across diverse driving conditions?
• RQ4 Does the proposed method maintain near-optimal performance while learning from human driving patterns?

Key findings

• The proposed human-like EMS achieves fuel economy performance close to the global optimum obtained via dynamic programming.
• The DRL agent trained on historical driving data converges faster than conventional DRL methods without expert demonstrations.
• The system demonstrates strong adaptability across different driving cycles, indicating robust generalization.
• The use of real driver data instead of DP-based trajectories results in more realistic and practical control policies.