[Paper Review] An Analysis of Deep Neural Network Models for Practical Applications

Alfredo Canziani, Adam Paszke+1 more|arXiv (Cornell University)|May 24, 2016

Advanced Neural Network Applications11 references981 citations

TL;DR

The paper analyzes state-of-the-art DNN architectures for ImageNet across accuracy, memory, parameters, compute, inference time, and power, highlighting hyperbolic accuracy-time trade-offs and the impact of energy constraints.

ABSTRACT

Since the emergence of Deep Neural Networks (DNNs) as a prominent technique in the field of computer vision, the ImageNet classification challenge has played a major role in advancing the state-of-the-art. While accuracy figures have steadily increased, the resource utilisation of winning models has not been properly taken into account. In this work, we present a comprehensive analysis of important metrics in practical applications: accuracy, memory footprint, parameters, operations count, inference time and power consumption. Key findings are: (1) power consumption is independent of batch size and architecture; (2) accuracy and inference time are in a hyperbolic relationship; (3) energy constraint is an upper bound on the maximum achievable accuracy and model complexity; (4) the number of operations is a reliable estimate of the inference time. We believe our analysis provides a compelling set of information that helps design and engineer efficient DNNs.

Motivation & Objective

• Motivate the need to evaluate DNNs beyond accuracy alone for practical deployments.
• Provide a cross-architecture comparison on multiple resource metrics relevant to real-world applications.
• Identify relationships between accuracy, computation, and energy under practical constraints.
• Propose insights to guide efficient network design and deployment strategies.

Proposed method

• Collect and re-evaluate top-1 accuracies using single central-crop sampling across networks to enable fair comparisons.
• Measure inference time, memory usage, and power on a Jetson TX1 with Torch7 and cuDNN/cuD back-end.
• Compute operation counts with a custom open-source tool to relate them to inference time.
• Analyze power consumption, including high-bandwidth current measurements, across batch sizes.
• Visualize accuracy versus operations, parameters, and throughput to identify trade-offs.
• Introduce ENet as an efficient architecture designed for practical efficiency.

Experimental results

Research questions

• RQ1 How do modern DNN architectures compare in accuracy, memory footprint, parameter count, operations, inference time, and power consumption on practical hardware?
• RQ2 What are the relationships between accuracy, throughput, and energy under real-world deployment constraints?
• RQ3 Can an architecture like ENet outperform larger models in terms of information density per parameter and overall efficiency?

Key findings

• ResNet and Inception variants substantially surpass earlier networks in accuracy, with improvements of at least 7% over prior architectures.
• VGG-16/19 are the most expensive architectures in both computation and parameter count, forming an isolated cluster apart from other networks.
• There is a hyperbolic relationship between accuracy and inference time across architectures.
• The number of operations is a reliable estimator of inference time, especially at fixed batch sizes.
• Power consumption is largely independent of batch size and architecture, with full utilization adding about 11.8 W over idle power.
• ENet achieves the best information density, using up to 24x fewer parameters than VGG-19 while preserving competitive accuracy.