[Paper Review] Autonomous Charging of Electric Vehicle Fleets to Enhance Renewable Generation Dispatchability

Reza Bayani, Saeed D. Manshadi+3 more|arXiv (Cornell University)|Dec 22, 2020

Electric Vehicles and Infrastructure34 references4 citations

TL;DR

This paper proposes an autonomous charging framework for electric vehicle (EV) fleets using fitted Q-iteration reinforcement learning to reduce renewable energy curtailment by coordinating EV charging with stochastic solar generation. The method enables EVs to learn optimal charging schedules that follow grid demand, reduce ramping stress, and maximize PV utilization—achieving 90.1% PV utilization in simulations with near-optimal performance after 60 days of training.

ABSTRACT

A total 19% of generation capacity in California is offered by PV units and over some months, more than 10% of this energy is curtailed. In this research, a novel approach to reduce renewable generation curtailments and increasing system flexibility by means of electric vehicles' charging coordination is represented. The presented problem is a sequential decision making process, and is solved by fitted Q-iteration algorithm which unlike other reinforcement learning methods, needs fewer episodes of learning. Three case studies are presented to validate the effectiveness of the proposed approach. These cases include aggregator load following, ramp service and utilization of non-deterministic PV generation. The results suggest that through this framework, EVs successfully learn how to adjust their charging schedule in stochastic scenarios where their trip times, as well as solar power generation are unknown beforehand.

Motivation & Objective

• To address the growing challenge of solar power curtailment in renewable-heavy grids, particularly in California where up to 18% of PV generation is curtailed.
• To enhance grid flexibility and dispatchability by leveraging EV fleets as responsive energy storage through intelligent charging coordination.
• To develop a scalable, autonomous charging strategy that operates under uncertainty in EV trip times and variable solar generation.
• To evaluate the performance of reinforcement learning in comparison to traditional optimization and deep RL methods for EV scheduling.
• To demonstrate that EVs can learn to delay charging and align with renewable generation peaks without prior knowledge of future conditions.

Proposed method

• The problem is modeled as a sequential decision-making process where EVs choose charging actions at each time step to maximize long-term rewards.
• A fitted Q-iteration algorithm with decision trees as function approximators is used to learn the optimal Q-function, enabling efficient action selection without solving optimization problems at each step.
• The reward function is designed to penalize energy purchases from the grid and reward utilization of locally generated PV power, encouraging curtailment reduction.
• The framework is tested across three case studies: aggregator load following, ramp service provision, and stochastic PV integration with unknown generation and trip times.
• A benchmark 'all-knowing' optimizer based on linear programming provides a global optimum for performance comparison.
• The method is compared against a deep Q-network (DRL) approach to evaluate sample efficiency and learning speed.

Experimental results

Research questions

• RQ1 Can EV fleets autonomously learn to coordinate charging to reduce renewable energy curtailment under uncertain solar generation and EV arrival/departure times?
• RQ2 How effective is fitted Q-iteration in learning optimal charging policies compared to deep reinforcement learning in this EV scheduling context?
• RQ3 To what extent can EVs learn to follow a reference power trajectory or provide ramping services without prior knowledge of future conditions?
• RQ4 How does the performance of the autonomous EV charging system compare to an optimal offline solution?
• RQ5 What is the impact of training duration on the convergence and performance of the reinforcement learning policy?

Key findings

• After 60 days of training, the EV fleet achieved 90.1% utilization of photovoltaic (PV) generation in the stochastic PV integration case, significantly reducing curtailment.
• The fitted Q-iteration algorithm achieved near-optimal performance (reward of -350) after 60 days, approaching the benchmark optimizer’s reward of -105 in Case 0.
• The DRL approach required substantially more data and showed slower convergence, with rewards still far from optimal after 75 days (reward of -5732 vs. -350 for fitted Q-iteration).
• In the ramp service case, the EV fleet successfully learned to reduce consumption during peak periods, demonstrating its ability to provide grid regulation services.
• The system learned to delay charging when PV output was high and grid demand was low, effectively using EVs as flexible storage to absorb surplus renewable energy.
• The results confirm that EV fleets can autonomously learn to enhance grid flexibility and reduce curtailment even with incomplete information about future conditions.