[Paper Review] Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling

Paul Joe Maliakel, Shashikant Ilager+1 more|ArXiv.org|Jan 14, 2025

Business Process Modeling and Analysis5 citations

TL;DR

This paper benchmarks a range of LLMs on diverse NLP tasks to characterize energy, latency, throughput, and memory trade-offs during inference, and analyzes hardware DVFS effects and input characteristics to guide energy-efficient deployments.

ABSTRACT

LLM inference exhibits substantial variability across queries and execution phases, yet inference configurations are often applied uniformly. We present a measurement-driven characterization of workload heterogeneity and energy-performance behavior of LLM inference under GPU dynamic voltage and frequency scaling (DVFS). We evaluate five decoder-only LLMs (1B-32B parameters) across four NLP benchmarks using a controlled offline setup. We show that lightweight semantic features predict inference difficulty better than input length, with 44.5% of queries achieving comparable quality across model sizes. At the hardware level, the decode phase dominates inference time (77-91%) and is largely insensitive to GPU frequency. Consequently, reducing GPU frequency from 2842 MHz to 180 MHz achieves an average of 42% energy savings with only a 1-6% latency increase. We further provide a use case with an upper-bound analysis of the potential benefits of combining workload-aware model selection with phase-aware DVFS, motivating future energy-efficient LLM inference systems.

Motivation & Objective

• Motivate sustainable AI by understanding how model size, architecture, task, and hardware settings affect energy and latency during inference.
• Evaluate a diverse set of models (GPT-2, Falcon-7B, Mistral-7B, GPT-J-6B, GPT-Neo-2.7B, T5-3B) across multiple tasks to map energy-performance dynamics.
• Investigate the effects of input characteristics (sequence length, batch size) on energy, throughput, and latency.
• Assess hardware-level DVFS impacts on LLM inference energy efficiency and performance to identify practical optimization strategies.

Proposed method

• Benchmark six pre-trained models (T5-3B, GPT-Neo-2.7B, GPT-2, Mistral-7B, Falcon-7B, GPT-J-6B) on NVIDIA A100 GPUs.
• Evaluate on tasks including SQuADv2, BoolQ, XSum, HellaSwag, OpenBookQA, and others using metrics like BLEU, F1, accuracy, energy, and throughput.
• Analyze input sequence length intervals (0-50, 51-100, 101-150, 151-200) and batch size effects with 8-wide batches across 100 queries per task.
• Examine DVFS configurations ([210, 405, 615, 810, 1005, 1215, 1410] MHz) to measure latency and energy trade-offs.
• Use real-time power monitoring via nvidia-smi and statistical analysis to derive insights.

Experimental results

Research questions

• RQ1 How do different NLP tasks (text generation, QA, commonsense reasoning, summarization) impact energy consumption and performance across LLMs?
• RQ2 How do input characteristics (sequence length, batch size) affect latency, throughput, and energy efficiency during inference?
• RQ3 What is the impact of hardware DVFS settings on energy efficiency and performance for various models and tasks?

Key findings

• Smaller models (e.g., GPT-2) consistently use less energy and offer higher throughput in simpler tasks, making them suitable for energy-constrained scenarios.
• Larger mid-sized models (Falcon-7B, Mistral-7B) balance performance and energy and excel in complex tasks like SQuADv2 and HellaSwag, though they consume more energy than GPT-2.
• GPT-J-6B and GPT-Neo-2.7B show high energy usage and memory demands, indicating inefficiency for many tasks.
• DVFS tuning to higher GPU clocks (1005–1410 MHz) reduces overall energy consumption by shortening runtimes, compared to lower clocks, across several tasks.
• Input sequence length increases generally raise latency and energy for all models, with larger models more affected, highlighting task-specific model choice for extended contexts.
• Batch size increases (up to 8) substantially improve energy efficiency and reduce latency for BoolQ and SquadV2 tasks, underscoring batch optimization as a practical lever.