[Paper Review] CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Management

José R. Vázquez-Canteli, Sourav Dey+2 more|arXiv (Cornell University)|Dec 18, 2020

Smart Grid Energy Management16 references44 citations

TL;DR

The paper introduces CityLearn, an OpenAI Gym environment to standardize multi-agent reinforcement learning research for demand response and urban energy management, plus the CityLearn Challenge to spur progress.

ABSTRACT

Rapid urbanization, increasing integration of distributed renewable energy resources, energy storage, and electric vehicles introduce new challenges for the power grid. In the US, buildings represent about 70% of the total electricity demand and demand response has the potential for reducing peaks of electricity by about 20%. Unlocking this potential requires control systems that operate on distributed systems, ideally data-driven and model-free. For this, reinforcement learning (RL) algorithms have gained increased interest in the past years. However, research in RL for demand response has been lacking the level of standardization that propelled the enormous progress in RL research in the computer science community. To remedy this, we created CityLearn, an OpenAI Gym Environment which allows researchers to implement, share, replicate, and compare their implementations of RL for demand response. Here, we discuss this environment and The CityLearn Challenge, a RL competition we organized to propel further progress in this field.

Motivation & Objective

• Motivate standardized, data-driven, model-free RL research for city-scale energy systems.
• Provide a reusable, open environment to implement, share, replicate, and compare RL approaches for demand response.
• Promote reproducibility and fair benchmarking across studies in urban energy management.
• Demonstrate the potential of RL to reduce peak electricity demand through distributed control of buildings.

Proposed method

• Introduce CityLearn as an OpenAI Gym environment for multi-agent RL in energy management.
• Describe the environment setup and interfaces that enable implementation, sharing, and replication of RL algorithms.
• Present a framework for benchmarking and comparing RL methods in demand response scenarios.
• Organize the CityLearn Challenge to accelerate progress and foster competition among researchers.

Experimental results

Research questions

• RQ1 How can a standardized RL environment facilitate reproducibility and fair benchmarking in demand response and urban energy management?
• RQ2 Can multi-agent RL achieve meaningful demand response and peak suppression in city-scale energy systems using CityLearn?
• RQ3 What are the key design considerations for environments that realistically model buildings, loads, and energy resources in urban grids?

Key findings

• CityLearn enables implementation, sharing, replication, and comparison of RL approaches for demand response.
• The CityLearn Challenge is proposed to propel progress in RL for urban energy management (organizational detail referenced in abstract).
• The environment supports distributed control and data-driven, model-free RL methods in building-level energy management.
• The approach targets reducing peaks in electricity demand through coordinated control of distributed resources.