[Paper Review] Exploring the Carbon Footprint of Hugging Face's ML Models: A Repository Mining Study

Joel Castaño, Silverio Martínez‐Fernández+2 more|arXiv (Cornell University)|May 18, 2023

Green IT and Sustainability32 references4 citations

TL;DR

This study analyzes the carbon footprint of 1,417 machine learning models on Hugging Face through repository mining, revealing stagnant reporting rates and strong correlations between emissions and model size, dataset size, and NLP focus. It proposes two classification systems—carbon reporting practices and carbon efficiency—to promote transparency and sustainable model development in the ML community.

ABSTRACT

The rise of machine learning (ML) systems has exacerbated their carbon footprint due to increased capabilities and model sizes. However, there is scarce knowledge on how the carbon footprint of ML models is actually measured, reported, and evaluated. In light of this, the paper aims to analyze the measurement of the carbon footprint of 1,417 ML models and associated datasets on Hugging Face, which is the most popular repository for pretrained ML models. The goal is to provide insights and recommendations on how to report and optimize the carbon efficiency of ML models. The study includes the first repository mining study on the Hugging Face Hub API on carbon emissions. This study seeks to answer two research questions: (1) how do ML model creators measure and report carbon emissions on Hugging Face Hub?, and (2) what aspects impact the carbon emissions of training ML models? The study yielded several key findings. These include a stalled proportion of carbon emissions-reporting models, a slight decrease in reported carbon footprint on Hugging Face over the past 2 years, and a continued dominance of NLP as the main application domain. Furthermore, the study uncovers correlations between carbon emissions and various attributes such as model size, dataset size, and ML application domains. These results highlight the need for software measurements to improve energy reporting practices and promote carbon-efficient model development within the Hugging Face community. In response to this issue, two classifications are proposed: one for categorizing models based on their carbon emission reporting practices and another for their carbon efficiency. The aim of these classification proposals is to foster transparency and sustainable model development within the ML community.

Motivation & Objective

• To investigate how machine learning model creators on Hugging Face report carbon emissions, addressing a critical gap in sustainability transparency.
• To identify key factors influencing the carbon footprint of ML models, particularly model size, dataset size, and application domain.
• To address the lack of standardized carbon reporting practices in the ML community by proposing actionable classification systems.
• To promote carbon-efficient model development through empirical insights and recommendations for practitioners and researchers.

Proposed method

• Conducted a large-scale repository mining study using the Hugging Face Hub API to extract metadata and self-reported carbon emissions from 1,417 models.
• Applied data preprocessing and harmonization techniques to standardize inconsistent self-reported carbon emission values across models.
• Performed statistical analysis to identify correlations between carbon emissions and model attributes such as model size, dataset size, and performance metrics.
• Developed and proposed two classification systems: one for categorizing models based on their carbon reporting practices, and another for assessing carbon efficiency.
• Provided a comprehensive replication package on Zenodo, including code, datasets, and Jupyter notebooks for reproducibility and future extension.

Experimental results

Research questions

• RQ1 How do ML model creators on Hugging Face measure and report carbon emissions for their models?
• RQ2 What model and dataset attributes correlate most strongly with carbon emissions in Hugging Face models?
• RQ3 How has the proportion of models reporting carbon emissions changed over the past two years on Hugging Face?
• RQ4 To what extent do application domains like NLP, computer vision, or audio processing differ in their carbon footprint profiles?
• RQ5 What systemic barriers hinder carbon reporting and efficiency in the Hugging Face ML ecosystem?

Key findings

• The proportion of Hugging Face models reporting carbon emissions has remained stagnant over the past two years, indicating little progress in sustainability transparency.
• There is a significant correlation between model size and carbon emissions, with larger models consistently showing higher emissions.
• Dataset size also shows a strong positive correlation with carbon emissions, highlighting data-intensive training as a major contributor.
• NLP remains the dominant application domain for emissions reporting, accounting for the majority of models with self-reported carbon footprints.
• Despite growing awareness, no clear trade-off between model performance and emissions was consistently observed, suggesting complex interdependencies.
• The study identifies a lack of standardized reporting practices, with many models lacking informative model cards, limiting the reliability of emissions assessments.