[Paper Review] Power Hungry Processing: Watts Driving the Cost of AI Deployment?

Alexandra Sasha Luccioni, Yacine Jernite+1 more|arXiv (Cornell University)|Nov 28, 2023

Machine Learning and Data Classification45 references27 citations

TL;DR

The paper systematically compares inference energy and carbon across task-specific and multi-purpose ML models, showing multi-purpose generative systems are far more energy-intensive per 1,000 inferences than task-specific ones, across tasks and modalities.

ABSTRACT

Recent years have seen a surge in the popularity of commercial AI products based on generative, multi-purpose AI systems promising a unified approach to building machine learning (ML) models into technology. However, this ambition of ``generality'' comes at a steep cost to the environment, given the amount of energy these systems require and the amount of carbon that they emit. In this work, we propose the first systematic comparison of the ongoing inference cost of various categories of ML systems, covering both task-specific (i.e. finetuned models that carry out a single task) and `general-purpose' models, (i.e. those trained for multiple tasks). We measure deployment cost as the amount of energy and carbon required to perform 1,000 inferences on representative benchmark dataset using these models. We find that multi-purpose, generative architectures are orders of magnitude more expensive than task-specific systems for a variety of tasks, even when controlling for the number of model parameters. We conclude with a discussion around the current trend of deploying multi-purpose generative ML systems, and caution that their utility should be more intentionally weighed against increased costs in terms of energy and emissions. All the data from our study can be accessed via an interactive demo to carry out further exploration and analysis.

Motivation & Objective

• Assess energy and carbon emissions of ML model inferences across diverse tasks and modalities.
• Compare task-specific versus multi-purpose models in deployment environments.
• Quantify how model size, architecture, and task type influence inference energy and emissions.
• Provide guidance for practitioners on accuracy-efficiency trade-offs and policy implications during deployment.

Proposed method

• Select 10 tasks across 5 modalities and 30 datasets; use 88 models (task-specific and multi-purpose) representative of deployment use cases.
• Measure energy and CO2 emissions per 1,000 inferences using Code Carbon in a fixed AWS region (us-west-2) on an NVIDIA A100-SXM4-80GB GPU.
• Run each inference set 10 times without batching to reflect deployment variability.
• Compare task-specific models to multi-purpose zero-shot models (Flan-T5 and BLOOMz families) on selected tasks.
• Analyze relationship between model size, task type, and emissions; examine training vs. inference cost parity using available training/fine-tuning data.

Figure 1. The tasks examined in our study and the average quantity of carbon emissions they produced (in g of $CO_{2}$ ) for 1,000 queries. N.B. The y axis is in logarithmic scale.

Experimental results

Research questions

• RQ1 What are the energy and carbon costs of deploying different ML models for common NLP and CV tasks per 1,000 inferences?
• RQ2 How do task-specific models compare to multi-purpose models in terms of emissions across tasks and modalities?
• RQ3 How do model size and architecture influence energy use and emissions within task-specific vs. multi-purpose categories?
• RQ4 When do inference costs reach parity with training costs for representative models?

Key findings

• Inference energy and CO2 emissions vary by up to three orders of magnitude across tasks and modalities.
• Task-specific models generally emit far less CO2 per 1,000 inferences than multi-purpose models, especially for discriminative vs. generative tasks.
• Text-to-image and image generation tasks are the most energy- and carbon-intensive; text-based tasks are comparatively efficient.
• Within multi-purpose models, sequence-to-sequence architectures tend to be more energy-efficient than decoder-only models of similar size for comparable tasks.
• Larger model size generally increases emissions, but architecture (seq2seq vs. decoder) and task output length substantially influence total emissions.

Figure 2. The 5 modalities examined in our study, with the number of parameters of each model on the x axis and the average amount of carbon emitted for 1000 inferences on the y axis. NB: Both axes are in logarithmic scale.