[Paper Review] Hardware-Aware Machine Learning: Modeling and Optimization

Diana Marculescu, Dimitrios Stamoulis+1 more|arXiv (Cornell University)|Sep 14, 2018

Parallel Computing and Optimization Techniques6 citations

TL;DR

This paper presents a comprehensive survey of hardware-aware machine learning, focusing on predictive modeling and optimization of deep neural networks (DNNs) for hardware efficiency. It proposes hardware-aware hyper-parameter optimization techniques that jointly minimize latency and energy consumption while maintaining accuracy, enabling co-design of DNNs and accelerators for diverse platforms from mobile devices to datacenters.

ABSTRACT

Recent breakthroughs in Deep Learning (DL) applications have made DL models a key component in almost every modern computing system. The increased popularity of DL applications deployed on a wide-spectrum of platforms have resulted in a plethora of design challenges related to the constraints introduced by the hardware itself. What is the latency or energy cost for an inference made by a Deep Neural Network (DNN)? Is it possible to predict this latency or energy consumption before a model is trained? If yes, how can machine learners take advantage of these models to design the hardware-optimal DNN for deployment? From lengthening battery life of mobile devices to reducing the runtime requirements of DL models executing in the cloud, the answers to these questions have drawn significant attention. One cannot optimize what isn't properly modeled. Therefore, it is important to understand the hardware efficiency of DL models during serving for making an inference, before even training the model. This key observation has motivated the use of predictive models to capture the hardware performance or energy efficiency of DL applications. Furthermore, DL practitioners are challenged with the task of designing the DNN model, i.e., of tuning the hyper-parameters of the DNN architecture, while optimizing for both accuracy of the DL model and its hardware efficiency. Therefore, state-of-the-art methodologies have proposed hardware-aware hyper-parameter optimization techniques. In this paper, we provide a comprehensive assessment of state-of-the-art work and selected results on the hardware-aware modeling and optimization for DL applications. We also highlight several open questions that are poised to give rise to novel hardware-aware designs in the next few years, as DL applications continue to significantly impact associated hardware systems and platforms.

Motivation & Objective

• Address the growing challenge of deploying energy- and latency-constrained DNNs on diverse hardware platforms, from mobile to datacenter systems.
• Overcome the limitations of traditional DNN design, which focuses solely on accuracy and ignores hardware efficiency during hyper-parameter tuning.
• Enable early prediction of DNN runtime and energy consumption before training, using predictive models to guide efficient model and hardware co-design.
• Advance the state of the art in automated neural architecture search (NAS) by integrating hardware performance as a first-class constraint.
• Identify open research directions for cross-platform modeling and multi-objective optimization in hardware-aware DNN design.

Proposed method

• Survey and evaluate existing predictive models for DNN hardware performance, including runtime and energy estimation frameworks like Eyeriss, Paleo, and NeuralPower.
• Introduce hardware-aware hyper-parameter optimization using Bayesian optimization and NAS-based methods that incorporate hardware metrics (e.g., latency, energy) as objectives.
• Propose extending predictive models to handle nonlinear DNN architectures, especially those used in NAS with complex, non-sequential operations.
• Develop cross-platform predictive models capable of generalizing across diverse hardware, including GPUs, FPGAs, and reconfigurable accelerators.
• Integrate hardware-aware models into co-design pipelines that jointly optimize DNN architecture and hardware parameters (e.g., number of processing elements).
• Formulate multi-objective optimization problems that balance accuracy, latency, energy, and hardware utilization across different platforms.

Experimental results

Research questions

• RQ1 Can accurate predictive models be built to estimate DNN runtime and energy consumption before training, based on architectural hyper-parameters?
• RQ2 How can hardware efficiency be effectively integrated into neural architecture search (NAS) and hyper-parameter optimization without sacrificing accuracy?
• RQ3 What are the key challenges in extending hardware-aware models to nonlinear and complex DNN architectures such as those used in NAS?
• RQ4 How can predictive models be made portable across diverse hardware platforms, including mobile, edge, and server systems?
• RQ5 What are the benefits of co-optimizing DNN architecture and hardware parameters (e.g., number of PEs) for improved performance and energy efficiency?

Key findings

• Significant energy differences—up to 40×—can exist among DNNs achieving the same accuracy, highlighting the need for hardware-aware design.
• Hardware-aware NAS and hyper-parameter optimization can identify models closer to the Pareto front of accuracy, latency, and energy efficiency.
• Existing predictive models are largely limited to linear or pipelined DNNs and require extension to handle nonlinear, cell-based NAS architectures.
• Cross-platform hardware-aware models are essential for transferring knowledge across platforms and enabling portable, efficient DNN deployment.
• Co-optimization of DNNs and hardware accelerators, such as reconfigurable FPGAs, can yield substantial gains in energy and performance when guided by predictive models.
• Multi-objective optimization extending beyond accuracy and latency—such as hardware utilization—can lead to more efficient and scalable deployments.