[Paper Review] Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model

Alexandra Sasha Luccioni, Sylvain Viguier+1 more|arXiv (Cornell University)|Nov 3, 2022

Green IT and SustainabilityEngineering188 citations

TL;DR

The paper estimates BLOOM's lifecycle carbon emissions, including embodied, dynamic, idle, and deployment emissions, and compares BLOOM with other LLMs.

ABSTRACT

Progress in machine learning (ML) comes with a cost to the environment, given that training ML models requires significant computational resources, energy and materials. In the present article, we aim to quantify the carbon footprint of BLOOM, a 176-billion parameter language model, across its life cycle. We estimate that BLOOM's final training emitted approximately 24.7 tonnes of~\carboneq~if we consider only the dynamic power consumption, and 50.5 tonnes if we account for all processes ranging from equipment manufacturing to energy-based operational consumption. We also study the energy requirements and carbon emissions of its deployment for inference via an API endpoint receiving user queries in real-time. We conclude with a discussion regarding the difficulty of precisely estimating the carbon footprint of ML models and future research directions that can contribute towards improving carbon emissions reporting.

Motivation & Objective

• Quantify BLOOM's total carbon footprint across its life cycle.
• Disaggregate emissions into embodied, dynamic, idle, and deployment components.
• Compare BLOOM's emissions with other large language models to contextualize environmental impact.
• Discuss limitations, methodological choices, and directions for transparent reporting.

Proposed method

• Adopts a life cycle assessment framework focusing on manufacturing to deployment stages.
• Calculates embodied emissions for servers and GPUs using external footprint estimates and a 6-year replacement assumption.
• Uses GPU-hour counts, GPU TDP, and grid carbon intensity to estimate dynamic emissions.
• Measures idle consumption via cluster power usage experiments to allocate non-training energy.
• Analyzes deployment/inference emissions using a real-time API scenario and CodeCarbon tool.
• Provides cross-model comparisons while noting variability in methodologies across studies.

Experimental results

Research questions

• RQ1 What are BLOOM's total greenhouse gas emissions across its life cycle (embodied, dynamic, idle, deployment)?
• RQ2 How do BLOOM's emissions compare to other 176B+ parameter LLMs under similar life-cycle accounting?
• RQ3 What are the main contributors to BLOOM's carbon footprint, and how sensitive are estimates to method choices?
• RQ4 What are the implications for reporting practices and future work in ML carbon accounting?

Key findings

• Total BLOOM emissions: about 50.5 tonnes CO2eq across life cycle (embodied + dynamic + idle).
• Dynamic energy contributes 24.69 tonnes CO2eq (about 48.9% of total), idle contributes 14.6 tonnes (28.9%), embodied contributes 11.2 tonnes (22.2%).
• BLOOM training used 1,082,990 GPU hours on Nvidia A100 GPUs with 57 gCO2eq/kWh grid intensity, yielding 433 MWh energy use.
• Compared to GPT-3 and Gopher, BLOOM shows lower emissions largely due to lower grid carbon intensity (57 gCO2eq/kWh) and similar energy use; OPT shows higher reported emissions due to different accounting (with PUE considerations discussed).
• Deployment/inference emissions were estimated from a real-time API deployment on GCP, highlighting energy use when idle between requests and region-specific grid intensity.