[Paper Review] Searching for Fast Model Families on Datacenter Accelerators

Sheng Li, Mingxing Tan+5 more|arXiv (Cornell University)|Feb 10, 2021

Advanced Neural Network Applications48 references4 citations

TL;DR

This paper proposes a hardware-aware neural architecture search (NAS) and latency-aware compound scaling (LACS) to optimize CNN model families for datacenter accelerators like TPUv3 and GPUv100. By designing a search space with accelerator-friendly operations—such as space-to-depth, fused convolutions, and block-wise activations—and jointly optimizing for accuracy and latency, the method discovers EfficientNet-X, which achieves up to 2.3× faster inference than EfficientNet with comparable accuracy.

ABSTRACT

Neural Architecture Search (NAS), together with model scaling, has shown remarkable progress in designing high accuracy and fast convolutional architecture families. However, as neither NAS nor model scaling considers sufficient hardware architecture details, they do not take full advantage of the emerging datacenter (DC) accelerators. In this paper, we search for fast and accurate CNN model families for efficient inference on DC accelerators. We first analyze DC accelerators and find that existing CNNs suffer from insufficient operational intensity, parallelism, and execution efficiency. These insights let us create a DC-accelerator-optimized search space, with space-to-depth, space-to-batch, hybrid fused convolution structures with vanilla and depthwise convolutions, and block-wise activation functions. On top of our DC accelerator optimized neural architecture search space, we further propose a latency-aware compound scaling (LACS), the first multi-objective compound scaling method optimizing both accuracy and latency. Our LACS discovers that network depth should grow much faster than image size and network width, which is quite different from previous compound scaling results. With the new search space and LACS, our search and scaling on datacenter accelerators results in a new model series named EfficientNet-X. EfficientNet-X is up to more than 2X faster than EfficientNet (a model series with state-of-the-art trade-off on FLOPs and accuracy) on TPUv3 and GPUv100, with comparable accuracy. EfficientNet-X is also up to 7X faster than recent RegNet and ResNeSt on TPUv3 and GPUv100.

Motivation & Objective

• To address the growing performance gap between theoretical FLOPs and actual inference latency on datacenter accelerators like TPUs and GPUs.
• To identify hardware-specific bottlenecks in state-of-the-art CNNs, such as low operational intensity and poor parallelism.
• To design a NAS search space tailored for datacenter accelerators, improving operational intensity and execution efficiency.
• To develop a multi-objective compound scaling method (LACS) that jointly optimizes for accuracy and inference latency.
• To generate a new model family, EfficientNet-X, that achieves superior accuracy-latency trade-offs on modern datacenter accelerators.

Proposed method

• Conduct a quantitative analysis of TPUv3 and GPUv100 using a roofline model to reveal that low operational intensity and suboptimal parallelism cause FLOPs-latency nonproportionality.
• Design a DC-accelerator-optimized NAS search space incorporating space-to-depth, space-to-batch, hybrid fused convolutions (vanilla and depthwise), and block-wise activation functions.
• Propose latency-aware compound scaling (LACS), a multi-objective scaling method that jointly optimizes for accuracy and inference latency during model family generation.
• Use separate NAS and LACS processes on TPUv3 and GPUv100 to generate accelerator-specific model families, ensuring hardware-aware optimization.
• Train and evaluate the resulting model family, EfficientNet-X, on TPUv3 and GPUv100, comparing latency and accuracy against SOTA baselines.
• Validate generalization across accelerator generations by measuring speedup when migrating from TPUv2 to TPUv3.

Figure 1: Unified accelerator-optimized NAS and Latency-aware Compound Scaling (LACS) to search model families optimized for TPUs and GPUs. The same multi-objective with both latency and accuracy is used for both NAS and model scaling. For a given accelerator, a base model (m1) is obtained via NAS w

Experimental results

Research questions

• RQ1 Why do FLOPs-latency relationships break down on datacenter accelerators like TPUs and GPUs compared to CPUs?
• RQ2 How can neural architecture search be enhanced to better exploit hardware-specific characteristics of TPUs and GPUs?
• RQ3 What architectural components improve operational intensity and execution efficiency on datacenter accelerators?
• RQ4 How does multi-objective compound scaling (optimizing for both accuracy and latency) differ from single-objective scaling in terms of optimal scaling coefficients?
• RQ5 To what extent does the proposed method generalize across different generations of datacenter accelerators?

Key findings

• EfficientNet-X achieves up to 2.3× faster inference than EfficientNet on TPUv3 and up to 2.1× faster on GPUv100, with comparable top-1 accuracy.
• On TPUv3, EfficientNet-X is 7× faster than RegNet and 48% faster than ResNeSt, with 7X and 48% speedup respectively on average.
• On GPUv100, EfficientNet-X is 82% faster than RegNet and 48% faster than ResNeSt on average.
• When migrating from TPUv2 to TPUv3 (2× peak performance), EfficientNet-X achieves ~1.9× geometric mean speedup, compared to ~1.5× for EfficientNet, demonstrating 30% better scaling efficiency.
• LACS discovers that network depth should grow significantly faster than image resolution and width, contradicting previous compound scaling practices.
• The method generalizes well across accelerator generations, showing that hardware-aware design improves performance gains when upgrading hardware.

Figure 2: Rooflines of TPUv3, Volta SMX2 GPU, and Xeon Skylake CPU. TPU and GPU have overlapped rooflines because of their similar peak computation rate and memory bandwidth.