[Paper Review] NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks

Ermao Cai, Da-Cheng Juan+2 more|arXiv (Cornell University)|Oct 15, 2017

Advanced Neural Network Applications13 references77 citations

TL;DR

NeuralPower provides a layer-wise polynomial regression framework to predict power, runtime, and energy of CNNs on GPUs, enabling energy-aware architecture selection before training. It also introduces the Energy-Precision Ratio to balance accuracy and energy efficiency.

ABSTRACT

"How much energy is consumed for an inference made by a convolutional neural network (CNN)?" With the increased popularity of CNNs deployed on the wide-spectrum of platforms (from mobile devices to workstations), the answer to this question has drawn significant attention. From lengthening battery life of mobile devices to reducing the energy bill of a datacenter, it is important to understand the energy efficiency of CNNs during serving for making an inference, before actually training the model. In this work, we propose NeuralPower: a layer-wise predictive framework based on sparse polynomial regression, for predicting the serving energy consumption of a CNN deployed on any GPU platform. Given the architecture of a CNN, NeuralPower provides an accurate prediction and breakdown for power and runtime across all layers in the whole network, helping machine learners quickly identify the power, runtime, or energy bottlenecks. We also propose the "energy-precision ratio" (EPR) metric to guide machine learners in selecting an energy-efficient CNN architecture that better trades off the energy consumption and prediction accuracy. The experimental results show that the prediction accuracy of the proposed NeuralPower outperforms the best published model to date, yielding an improvement in accuracy of up to 68.5%. We also assess the accuracy of predictions at the network level, by predicting the runtime, power, and energy of state-of-the-art CNN architectures, achieving an average accuracy of 88.24% in runtime, 88.34% in power, and 97.21% in energy. We comprehensively corroborate the effectiveness of NeuralPower as a powerful framework for machine learners by testing it on different GPU platforms and Deep Learning software tools.

Motivation & Objective

• Motivate the need to predict CNN inference energy before training for deployment on diverse GPU platforms.
• Develop a layer-wise predictive framework to estimate power, runtime, and energy of CNNs during service, without running the networks.
• Enable rapid identification of runtime, power, or energy bottlenecks to guide architecture search for energy efficiency.
• Propose metrics and validation across multiple CNN architectures and GPU platforms to demonstrate accuracy and generalizability.

Proposed method

• Propose a hierarchical NeuralPower framework that models layer-level power and runtime for convolutional, fully-connected, and pooling layers using sparse polynomial regression.
• Use two-part layer models: (1) regular polynomial terms over layer configuration features and (2) special terms capturing operations like memory accesses and FLOPs; apply Lasso and cross-validation to select model terms.
• Extend layer-level models to network-level predictions by summing per-layer runtimes and computing energy from per-layer power and runtime estimates.
• Collect a dataset by profiling a set of CNN architectures on an Nvidia Titan X with fixed GPU state, TensorFlow, and nvidia-smi measurements to train the models.
• Evaluate network-level predictions against actual measurements across multiple CNNs (e.g., VGG, NIN, CIFAR nets) to quantify runtime, power, and energy accuracy.
• Introduce Energy-Precision Ratio as a metric to trade off classification accuracy and energy consumption when selecting architectures.

Experimental results

Research questions

• RQ1 Can layer-level polynomial regression accurately predict per-layer power and runtime for CNNs on GPUs across different architectures and frameworks?
• RQ2 How accurately can NeuralPower predict network-level runtime, power, and energy for diverse CNNs on GPU platforms?
• RQ3 Does the Energy-Precision Ratio effectively guide the selection of energy-efficient CNN architectures without sacrificing excessive accuracy?

Key findings

• NeuralPower achieves average network-level accuracy of about 88.24% in runtime and 88.34% in power, and 97.21% in energy predictions across tested CNNs.
• Layer-level models using sparse polynomial regression outperform the prior state-of-the-art Paleo model in runtime prediction, with improvements up to 68.5% in RMSE/RMSPE metrics.
• Power predictions at the layer level show RMSPE below 9% across convolutional, pooling, and fully connected layers.
• Network-level energy predictions achieve an average RMSPE around 2.79% when compared to per-layer actuals.
• The framework provides detailed per-layer breakdowns to identify runtime, power, and energy bottlenecks within networks.
• Energy-Precision Ratio offers a tunable metric to balance accuracy and energy, guiding energy-efficient CNN selection for different application requirements.