[Paper Review] Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks

Vahid Partovi Nia, Alireza Ghaffari+2 more|arXiv (Cornell University)|Feb 18, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a stochastic multi-dimensional Pareto frontier to objectively evaluate deep neural networks by jointly optimizing for accuracy, inference latency (on cloud and edge hardware), and training cost. By incorporating random variations through parametric bootstrapping, the method enables robust ranking of models across diverse hardware, revealing that MobileNetV3 and SqueezeNetV1.1 consistently dominate in efficiency despite performance variability.

ABSTRACT

Performance optimization of deep learning models is conducted either manually or through automatic architecture search, or a combination of both. On the other hand, their performance strongly depends on the target hardware and how successfully the models were trained. We propose to use a multi-dimensional Pareto frontier to re-define the efficiency measure of candidate deep learning models, where several variables such as training cost, inference latency, and accuracy play a relative role in defining a dominant model. Furthermore, a random version of the multi-dimensional Pareto frontier is introduced to mitigate the uncertainty of accuracy, latency, and throughput of deep learning models in different experimental setups. These two complementary methods can be combined to perform objective benchmarking of deep learning models. Our proposed method is applied to a wide range of deep image classification models trained on ImageNet data. Our method combines competing variables with stochastic nature in a single relative efficiency measure. This allows ranking deep learning models that run efficiently on different hardware, and combining inference efficiency with training efficiency objectively.

Motivation & Objective

• To address the lack of objective, multi-dimensional benchmarking for deep learning models that accounts for both inference and training efficiency.
• To mitigate performance variability in DNNs due to stochastic training and hardware-specific latency by introducing a stochastic Pareto frontier.
• To unify competing metrics—accuracy, latency on multiple hardware platforms, and training cost—into a single relative efficiency measure for model comparison.
• To enable objective model selection across diverse hardware targets by defining stochastic dominance relationships among models.

Proposed method

• Proposes a multi-dimensional Pareto frontier that incorporates three key objectives: top-1 accuracy, NVIDIA V100 inference latency, and ARM Kirin 970 edge latency, along with training cost in dollars.
• Introduces a stochastic Pareto frontier using parametric bootstrapping over 5 experimental runs to model the random fluctuations in accuracy, latency, and training cost.
• Defines a relative efficiency measure, θ̂, as a normalized score that reflects a model’s dominance across all objectives, enabling cross-model comparison.
• Employs stochastic dominance to rank models, where an arrow from model A to model B indicates A dominates B across all objectives under uncertainty.
• Uses empirical measurements from ImageNet-1K trained models to compute efficiency distributions and generate boxplots for visualizing performance variability.
• Applies the method to a wide range of image classification models, including MobileNet, ResNet, DenseNet, and MNASNet, across multiple hardware platforms.

Experimental results

Research questions

• RQ1 How can we objectively compare deep learning models when they trade off accuracy, inference latency, and training cost across different hardware platforms?
• RQ2 To what extent do random variations in model performance (accuracy, latency, training cost) affect the reliability of model rankings?
• RQ3 Which deep learning models demonstrate consistent dominance across diverse hardware and training conditions when using a multi-objective efficiency metric?
• RQ4 Can a stochastic Pareto frontier effectively capture and mitigate performance uncertainty in deep learning model evaluation?

Key findings

• MobileNetV3 and SqueezeNetV1.1 achieve the highest relative efficiency with narrow boxplot distributions, indicating robust performance across multiple runs and hardware platforms.
• ShuffleNetV2 and MobileNetV3 show strong stochastic dominance, remaining efficient even under performance variability, with efficiency scores consistently near 100%.
• DualPathNet92 exhibits the worst performance, with its efficiency distribution falling below the 20% threshold, indicating poor robustness and dominance.
• The inclusion of training cost dramatically alters model rankings: while MobileNetV3 leads under latency-only criteria, ShuffleNetV2 becomes more efficient when training cost is considered.
• The stochastic dominance diagram in Figure 3(b) confirms that models like MobileNetV3 and SqueezeNetV1.1 dominate all others across all objectives, even under uncertainty.
• The method successfully redefines efficiency by integrating training cost—previously overlooked—into a unified, objective benchmarking framework for DNNs.