[Paper Review] Quantifying the Carbon Emissions of Machine Learning

Alexandre Lacoste, Alexandra Sasha Luccioni+2 more|arXiv (Cornell University)|Oct 21, 2019

Explainable Artificial Intelligence (XAI)18 references130 citations

TL;DR

The paper presents the Machine Learning Emissions Calculator to estimate CO2 equivalents for ML training, highlighting how location, hardware, and training time drive emissions and offering actionable mitigation strategies.

ABSTRACT

From an environmental standpoint, there are a few crucial aspects of training a neural network that have a major impact on the quantity of carbon that it emits. These factors include: the location of the server used for training and the energy grid that it uses, the length of the training procedure, and even the make and model of hardware on which the training takes place. In order to approximate these emissions, we present our Machine Learning Emissions Calculator, a tool for our community to better understand the environmental impact of training ML models. We accompany this tool with an explanation of the factors cited above, as well as concrete actions that individual practitioners and organizations can take to mitigate their carbon emissions.

Motivation & Objective

• Motivate the need to quantify ML training emissions and standardize reporting.
• Present the Machine Learning Emissions Calculator as a tool for estimating CO2e from ML training.
• Explain the key factors that influence ML emissions (location, hardware, training time) and how to mitigate them.
• Provide best practices and actionable guidance for practitioners and organizations to reduce carbon impact.

Proposed method

• Define CO2 equivalents (CO2e) as the normalization metric for emissions.
• Gather public data on grid emissions factors and map them to GPU server locations from major cloud providers.
• Develop an emissions calculator that inputs geographic zone, GPU type, and training duration to estimate CO2e.
• Discuss factors such as energy grid mix, data center PUE, and provider offsets (RECs) in emissions estimates.
• Advocate for openness and community updates to the data via public repositories and issues.

Experimental results

Research questions

• RQ1 How much CO2e is emitted when training ML models across different geographic regions?
• RQ2 How do hardware choices (CPU/GPU/TPU) and training duration influence total emissions?
• RQ3 To what extent can choosing data center locations or cloud providers reduce emissions?
• RQ4 What practical actions and best practices most effectively lower ML training carbon footprints?
• RQ5 What are the limitations and uncertainties in current emissions estimation for ML training?

Key findings

• Emissions vary significantly by region, with examples showing large inter-region differences in CO2e per kWh.
• Training multi-GPU models for weeks can lead to substantial emissions, underscoring the impact of infrastructure and duration.
• Selecting data center locations powered by lower-carbon grids can markedly reduce emissions (up to large regional differences mentioned).
• Choosing more efficient hardware (e.g., TPUs) and adopting random hyperparameter search can lower energy usage compared to exhaustive searches.
• Using pre-trained models with fine-tuning can achieve comparable performance with potentially lower emissions than full training.
• Acknowledges limitations and uncertainties in emissions estimates and the need to consider inference/ deployment emissions as well.