[Paper Review] ECC: Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model.

Haichuan Yang, Yuhao Zhu+1 more|arXiv (Cornell University)|Dec 5, 2018

Advanced Neural Network Applications4 citations

TL;DR

ECC proposes a novel energy-constrained DNN compression framework using a bilinear regression model to estimate energy consumption, enabling constrained optimization that minimizes accuracy loss under strict energy budgets. By combining ADMM with gradient-based learning, ECC achieves higher accuracy than state-of-the-art methods under equivalent or lower energy constraints across diverse hardware platforms.

ABSTRACT

Many DNN-enabled vision applications constantly operate under severe energy constraints such as unmanned aerial vehicles, Augmented Reality headsets, and smartphones. Designing DNNs that can meet a stringent energy budget is becoming increasingly important. This paper proposes ECC, a framework that compresses DNNs to meet a given energy constraint while minimizing accuracy loss. The key idea of ECC is to model the DNN energy consumption via a novel bilinear regression function. The energy estimate model allows us to formulate DNN compression as a constrained optimization that minimizes the DNN loss function over the energy constraint. The optimization problem, however, has nontrivial constraints. Therefore, existing deep learning solvers do not apply directly. We propose an optimization algorithm that combines the essence of the Alternating Direction Method of Multipliers (ADMM) framework with gradient-based learning algorithms. The algorithm decomposes the original constrained optimization into several subproblems that are solved iteratively and efficiently. ECC is also portable across different hardware platforms without requiring hardware knowledge. Experiments show that ECC achieves higher accuracy under the same or lower energy budget compared to state-of-the-art resource-constrained DNN compression techniques.

Motivation & Objective

• Address the growing need for energy-efficient deep neural networks in mobile and edge devices like UAVs, AR headsets, and smartphones.
• Minimize accuracy loss when compressing DNNs under strict energy budget constraints.
• Develop a hardware-agnostic compression framework that does not require detailed hardware-specific knowledge.
• Formulate DNN compression as a constrained optimization problem with energy as a hard constraint.
• Enable portable, cross-platform deployment of compressed models without retraining for each hardware target.

Proposed method

• Model DNN energy consumption using a novel bilinear regression function that captures non-linear interactions between layer-wise parameters and operations.
• Formulate the compression task as a constrained optimization problem that minimizes the DNN loss subject to a user-defined energy budget.
• Introduce an optimization algorithm combining Alternating Direction Method of Multipliers (ADMM) with gradient-based learning to solve the non-convex, constrained problem.
• Decompose the original problem into subproblems that are solved iteratively, enabling efficient and stable convergence.
• Use alternating updates between model weights and auxiliary variables to enforce energy constraints while preserving model accuracy.
• Ensure portability across hardware platforms by avoiding hardware-specific energy estimation, relying instead on the generalizable bilinear energy model.

Experimental results

Research questions

• RQ1 Can a bilinear regression model accurately estimate DNN energy consumption across diverse hardware platforms?
• RQ2 How effectively can constrained optimization with energy budgets reduce accuracy loss during DNN compression?
• RQ3 Does the proposed ADMM-based optimization framework outperform existing gradient-based or heuristic compression methods under the same energy constraints?
• RQ4 To what extent is the framework portable across different hardware platforms without retraining or hardware-specific tuning?
• RQ5 Can the method achieve higher accuracy than state-of-the-art techniques while meeting or reducing energy consumption?

Key findings

• ECC achieves higher accuracy than state-of-the-art resource-constrained DNN compression techniques under the same or lower energy budgets.
• The bilinear regression model provides an effective and generalizable estimate of DNN energy consumption without requiring hardware-specific calibration.
• The ADMM-based optimization framework successfully handles the nontrivial constraints of the energy-constrained compression problem.
• The framework is portable across different hardware platforms without needing hardware-specific knowledge or retraining.
• Experiments demonstrate consistent performance gains in accuracy under energy constraints compared to existing methods.
• The method enables efficient, iterative compression that balances accuracy and energy efficiency effectively.