[Paper Review] EcoServe: Designing Carbon-Aware AI Inference Systems

Yueying, Zhanqiu Hu+4 more|ArXiv.org|Feb 7, 2025

Big Data and Business Intelligence3 citations

TL;DR

EcoServe presents a four-pillar, carbon-aware framework (Reuse, Rightsize, Reduce, Recycle) to optimize both operational and embodied carbon in LLM inference, achieving up to 47% carbon reduction with small performance trade-offs.

ABSTRACT

The rapid increase in LLM ubiquity and scale levies unprecedented demands on computing infrastructure. These demands not only incur large compute and memory resources but also significant energy, yielding large operational and embodied carbon emissions. In this work, we present three main observations based on modeling and traces from the production deployment of two Generative AI services in a major cloud service provider. First, while GPUs dominate operational carbon, host processing systems (e.g., CPUs, memory, storage) dominate embodied carbon. Second, offline, batch inference accounts for a significant portion (up to 55\%) of serving capacity. Third, there are different levels of heterogeneity across hardware and workloads for LLM inference. Based on these observations, we design EcoServe, a carbon-aware resource provision and scheduling framework for LLM serving systems. It is based on four principles - Reduce, Reuse, Rightsize, and Recycle (4R). With a cross-stack ILP formulation and design, we demonstrate that EcoServe can lower carbon emissions by up to 47\%, compared to performance, energy, and cost-optimized design points, while maintaining performance targets and SLOs.

Motivation & Objective

• Motivate the need to address both operational and embodied carbon in AI inference systems.
• Develop a holistic model for carbon emissions that includes host systems and GPUs.
• Propose EcoServe’s four design pillars to optimize carbon while maintaining SLOs and performance.
• Demonstrate how co-design across capacity planning, resource allocation, and scheduling reduces carbon.
• Quantify carbon reductions across open-source and production traces.

Proposed method

• Develop an embodied carbon model for AI systems that includes host CPU, memory, storage, PCB, and GPU components.
• Characterize total carbon as a function of power, time, and carbon intensity, incorporating both operational and embodied emissions.
• Propose EcoServe’s four design pillars (Reuse, Rightsize, Reduce, Recycle) and formulate an ILP to co-optimize performance, efficiency, and carbon.
• Showcase CPU reuse for offline inference to increase capacity and reduce embodied carbon.
• Right-size heterogeneous GPUs for online/offline phases based on workload characteristics and carbon efficiency.
• Present strategies to reduce wasted hardware and asymmetrically recycle host and accelerator components.

Figure 1 : (Left) Breakdown of power (TDP) and embodied carbon between host systems (CPU) and GPU. (Right) EcoServe optimizes operational and embodied carbon based on input hardware, model, and workload traces by reusing CPU for offline inference, rightsizing GPUs, reducing underutilized hardware, a

Experimental results

Research questions

• RQ1 How do operational and embodied carbon contribute to the total carbon footprint of AI inference systems?
• RQ2 Can a cross-layer, carbon-aware design (Reuse, Rightsize, Reduce, Recycle) reduce emissions without sacrificing SLOs?
• RQ3 What is the impact of reusing CPUs for offline inference on capacity and embodied carbon?
• RQ4 How should heterogeneous GPUs be provisioned to optimize energy, carbon, and performance for different LLM phases (prompt vs decode)?
• RQ5 What strategies balance hardware recycling and lifetime extension to minimize total carbon?

Key findings

• EcoServe can lower carbon emissions by up to 47% relative to optimized baselines while preserving performance targets and SLOs.
• Reusing idle CPU resources for offline inference increases cluster capacity and reduces embodied carbon by leveraging underutilized host components.
• Right-sizing GPUs across heterogeneous hardware and considering offline/online phases yields notable gains in energy, carbon, and performance efficiency.
• Reducing wasted embodied carbon by leaner hardware and efficiently recycling host systems can achieve 29% to 41% carbon savings per pillar (Reuse, Rightsize, Reduce, Recycle).
• An ILP-based co-optimization across workload slices (prompt vs decode) and hardware leads to 1.4x–2.2x carbon benefits with minimal performance degradation.

Figure 2 : Trends in bit density (left) and embodied carbon footprint (right) across varoius DRAM memory technologies for 3 different manufacturers.