[Paper Review] Watt For What: Rethinking Deep Learning's Energy-Performance Relationship

Shreyank N Gowda, Xinyue Hao+4 more|arXiv (Cornell University)|Oct 10, 2023

Green IT and Sustainability4 citations

TL;DR

This paper proposes a new metric—accuracy per unit of electricity consumed—to reframe the deep learning efficiency-accuracy trade-off, demonstrating that smaller, energy-efficient models can match or exceed large models in performance per watt. The study reveals that pre-training and self-supervised learning, while costly, yield high transferability, and advocates for sustainable, equitable AI development through energy-aware model design.

ABSTRACT

Deep learning models have revolutionized various fields, from image recognition to natural language processing, by achieving unprecedented levels of accuracy. However, their increasing energy consumption has raised concerns about their environmental impact, disadvantaging smaller entities in research and exacerbating global energy consumption. In this paper, we explore the trade-off between model accuracy and electricity consumption, proposing a metric that penalizes large consumption of electricity. We conduct a comprehensive study on the electricity consumption of various deep learning models across different GPUs, presenting a detailed analysis of their accuracy-efficiency trade-offs. By evaluating accuracy per unit of electricity consumed, we demonstrate how smaller, more energy-efficient models can significantly expedite research while mitigating environmental concerns. Our results highlight the potential for a more sustainable approach to deep learning, emphasizing the importance of optimizing models for efficiency. This research also contributes to a more equitable research landscape, where smaller entities can compete effectively with larger counterparts. This advocates for the adoption of efficient deep learning practices to reduce electricity consumption, safeguarding the environment for future generations whilst also helping ensure a fairer competitive landscape.

Motivation & Objective

• To address the growing environmental and economic burden of energy-intensive deep learning models.
• To investigate whether large-scale pre-training justifies its massive electricity consumption relative to performance gains.
• To enable smaller research entities to compete with large institutions by promoting energy-efficient model design.
• To quantify the energy cost of pre-training and fine-tuning across diverse architectures and tasks.
• To advocate for a shift in evaluation metrics from accuracy alone to accuracy per unit of electricity consumed.

Proposed method

• Measures electricity consumption of deep learning models across multiple GPUs and hardware configurations.
• Introduces a novel metric: accuracy per kilowatt-hour (kWh), to evaluate model efficiency.
• Evaluates a wide range of models, including ResNets, Vision Transformers, Swin, ViT, BEiT, and MAE, across image classification, segmentation, and video action recognition.
• Analyzes pre-training and fine-tuning costs separately, highlighting that pre-training dominates total energy use.
• Compares self-supervised models (BEiT, MAE) with supervised baselines to assess transferability and cost efficiency.
• Uses real-world electricity benchmarks (e.g., household, hypermarket, aircraft) to contextualize model energy use.

Experimental results

Research questions

• RQ1 How does the energy consumption of state-of-the-art deep learning models compare to everyday electricity users?
• RQ2 To what extent does pre-training improve model accuracy relative to its energy cost?
• RQ3 Can self-supervised learning methods deliver high performance with lower energy overhead compared to supervised pre-training?
• RQ4 How does the accuracy-per-watt metric reveal a more equitable landscape for smaller research institutions?
• RQ5 Is the performance gain from large-scale pre-training justified by its massive electricity consumption?

Key findings

• The GPT-3-175B model consumed 1,287,000 kWh of electricity during training—equivalent to the annual electricity use of a UK hypermarket.
• The OPT-175B model consumed 356,000 kWh during training, highlighting the extreme energy cost of large models.
• Self-supervised pre-training accounts for over 90% of total training energy cost, with fine-tuning and inference being negligible by comparison.
• ViT-B achieved a 6% accuracy improvement over training from scratch, but required ten times more electricity than training from scratch.
• Smaller, energy-efficient models can achieve comparable or better performance per watt, enabling faster and more sustainable research.
• Pre-training on large datasets like JFT and ImageNet-21k shows a linear correlation between data volume and projected electricity consumption.