[Paper Review] LLMCO2: Advancing Accurate Carbon Footprint Prediction for LLM Inferences

Zhenxiao Fu, Fan Chen+3 more|arXiv (Cornell University)|Oct 3, 2024

Data Quality and Management4 citations

TL;DR

LLMCO2 introduces a graph neural network (GNN)-based model that accurately predicts the carbon footprint of large language model (LLM) inferences by modeling autoregressive prefill and decode phases separately, incorporating hardware-specific Roofline performance, and using focused sampling of common inference configurations. It improves prediction accuracy by 51%–123% over prior methods, achieving a 123% higher Error Bound Accuracy (EBA) at 10% error than state-of-the-art baselines.

ABSTRACT

Throughout its lifecycle, a large language model (LLM) generates a substantially larger carbon footprint during inference than training. LLM inference requests vary in batch size, prompt length, and token generation number, while cloud providers employ different GPU types and quantities to meet diverse service-level objectives for accuracy and latency. It is crucial for both users and cloud providers to have a tool that quickly and accurately estimates the carbon impact of LLM inferences based on a combination of inference request and hardware configurations before execution. Estimating the carbon footprint of LLM inferences is more complex than training due to lower and highly variable model FLOPS utilization, rendering previous equation-based models inaccurate. Additionally, existing machine learning (ML) prediction methods either lack accuracy or demand extensive training data, as they inadequately handle the distinct prefill and decode phases, overlook hardware-specific features, and inefficiently sample uncommon inference configurations. We introduce \coo, a graph neural network (GNN)-based model that greatly improves the accuracy of LLM inference carbon footprint predictions compared to previous methods.

Motivation & Objective

• To address the lack of accurate, real-time carbon footprint estimation tools for LLM inference, which is more complex than training due to variable FLOPS utilization and autoregressive behavior.
• To overcome limitations of existing equation-based and ML-based models that fail to capture distinct prefill/decode phases, hardware-specific features, and real-world configuration distributions.
• To develop a scalable, accurate prediction model that supports users and cloud providers in evaluating trade-offs between latency, accuracy, and carbon emissions before inference execution.
• To improve prediction reliability across diverse LLM architectures, GPU types, and common inference configurations (e.g., small batch sizes, short prompts) through targeted data sampling and graph-based modeling.

Proposed method

• LLMCO2 uses a novel graph embedding technique where each transformer layer’s kernels are represented as a graph, with nodes for kernels and edges for data dependencies.
• It applies separate node feature encoding for the prefill and decode phases to model their distinct computational characteristics.
• Hardware-specific features such as GPU peak throughput, memory bandwidth, and network bandwidth are incorporated via Roofline performance metrics to enhance generalization across GPU platforms.
• A focused data sampling strategy prioritizes common real-world configurations—small batch sizes, short prompts, and limited token generation—to improve model performance on typical inference workloads.
• The GNN-based regression model learns the relationship between input configurations (prompt length, batch size, generated tokens) and carbon emissions, enabling fast inference-time predictions.
• The model is trained on a curated dataset of energy measurements across multiple LLMs (e.g., Bloom, Gemma, Mixtral) and GPUs (e.g., A100, L4, T4), with emphasis on operational energy and carbon overhead.

Figure 1: A decoder-only LLM’s autoregressive inference consists of prefill and decode phases.

Experimental results

Research questions

• RQ1 How can carbon footprint prediction for LLM inference be improved by modeling the distinct prefill and decode phases separately?
• RQ2 To what extent do hardware-specific features such as GPU throughput and memory bandwidth improve prediction accuracy across diverse GPU platforms?
• RQ3 Can focused sampling of common inference configurations enhance model generalization and accuracy on real-world workloads?
• RQ4 How does a GNN-based approach compare to existing ML and equation-based models in predicting LLM inference carbon emissions?
• RQ5 What is the impact of architectural and configuration diversity (e.g., batch size, prompt length, model size) on carbon footprint prediction accuracy?

Key findings

• LLMCO2 improves prediction accuracy by 51%–123% over existing ML-based energy predictors across diverse LLMs and GPU configurations.
• It achieves a 123% higher Error Bound Accuracy (EBA) at 10% error compared to NNLQP, the next-best baseline, across all tested models.
• The model’s EBA(10%) reaches 45.7% on average, with individual scores up to 60% for Bloom and 53.1% for Qwen2, demonstrating strong generalization.
• Separate modeling of prefill and decode phases improves EBA(10%) by 67% compared to a unified-phase approach, highlighting the importance of phase-aware design.
• Incorporating Roofline performance as a node feature boosts EBA(10%) by 13.1%, enabling better knowledge transfer across GPU types like L4 and T4.
• The focused sampling strategy contributes most significantly, increasing EBA(10%) by 123% over NNLQP, proving that data distribution alignment with real-world usage enhances model performance.

Figure 2: Comparing the energy of 2 phases in a Bloom-3B inference on an Nvidia L4 GPU.