[Paper Review] MoGA: Searching Beyond MobileNetV3

Xiangxiang Chu, Bo Zhang+1 more|arXiv (Cornell University)|Aug 4, 2019

Advanced Neural Network Applications23 references4 citations

TL;DR

MoGA proposes the first Mobile GPU-Aware Neural Architecture Search (NAS) framework to optimize neural networks specifically for mobile GPU inference, using a weighted fitness strategy that prioritizes accuracy and latency over parameter count. Trained in just 12 GPU days—200× less than MnasNet—it generates models like MoGA-A (75.9% top-1 ImageNet accuracy) and MoGA-C, which outperform MobileNetV3 on GPU with similar FLOPs and parameters.

ABSTRACT

The evolution of MobileNets has laid a solid foundation for neural network applications on mobile end. With the latest MobileNetV3, neural architecture search again claimed its supremacy in network design. Unfortunately, till today all mobile methods mainly focus on CPU latencies instead of GPU, the latter, however, is much preferred in practice for it has faster speed, lower overhead and less interference. Bearing the target hardware in mind, we propose the first Mobile GPU-Aware (MoGA) neural architecture search in order to be precisely tailored for real-world applications. Further, the ultimate objective to devise a mobile network lies in achieving better performance by maximizing the utilization of bounded resources. Urging higher capability while restraining time consumption is not reconcilable. We alleviate the tension by weighted evolution techniques. Moreover, we encourage increasing the number of parameters for higher representational power. With 200x fewer GPU days than MnasNet, we obtain a series of models that outperform MobileNetV3 under the similar latency constraints, i.e., MoGA-A achieves 75.9% top-1 accuracy on ImageNet, MoGA-B meets 75.5% which costs only 0.5 ms more on mobile GPU. MoGA-C best attests GPU-awareness by reaching 75.3% and being slower on CPU but faster on GPU.The models and test code is made available here https://github.com/xiaomi-automl/MoGA.

Motivation & Objective

• To address the gap in mobile neural architecture search by shifting focus from CPU to GPU inference, which is more commonly used in practice.
• To improve model performance within strict latency and resource constraints by optimizing for accuracy and latency while allowing higher parameter counts to prevent underfitting.
• To drastically reduce NAS search cost through a one-shot supernet and latency look-up table, enabling efficient adaptation across multiple mobile platforms.
• To develop a flexible, once-for-all NAS pipeline that supports rapid reconfiguration for new devices without retraining the supernet.

Proposed method

• Introduces Mobile GPU-Aware NAS (MoGA), a search framework tailored for mobile GPU inference, replacing CPU-centric design with GPU latency as a primary optimization target.
• Employs a weighted fitness function combining accuracy, latency, and parameter count, with higher weights on accuracy and latency, and a bonus for increased parameters to enhance representational capacity.
• Uses a one-shot supernet training with single-path training to reduce memory and compute costs, enabling efficient architecture search with minimal overhead.
• Constructs an accurate latency look-up table for efficient latency estimation during search, avoiding expensive real-device measurements.
• Applies hierarchical mutation in a weighted NSGA-II algorithm to improve search efficiency and convergence, outperforming random mutation baselines.
• Decouples search from training, allowing the same supernet to be reused across different mobile platforms with only a lightweight re-search using updated latency tables.

Experimental results

Research questions

• RQ1 Can neural architecture search be effectively adapted to mobile GPU inference, which is more common in real-world deployments than CPU inference?
• RQ2 How can the trade-off between accuracy, latency, and parameter count be optimized in mobile NAS when underfitting is a greater risk than overfitting?
• RQ3 Can the search cost of NAS be reduced to near-zero for new mobile platforms by reusing a pre-trained supernet and latency look-up table?
• RQ4 Does increasing the number of parameters in mobile models improve performance without increasing latency, especially on GPU?
• RQ5 How does GPU-aware architecture search compare to CPU-centric approaches in terms of real-world inference speed and accuracy?

Key findings

• MoGA-A achieves 75.9% top-1 accuracy on ImageNet under similar latency constraints to MobileNetV3, outperforming it on mobile GPU.
• MoGA-B achieves 75.5% top-1 accuracy with only 0.5 ms additional latency on mobile GPU compared to MoGA-A.
• MoGA-C reaches 75.3% top-1 accuracy with identical FLOPs and parameter count to MobileNetV3, but is faster on mobile GPU and slower on CPU, confirming its GPU-aware design.
• The entire MoGA search pipeline requires only 12 GPU days—about 200× less than MnasNet—demonstrating a significant reduction in search cost.
• The one-shot supernet and latency look-up table enable efficient, once-for-all deployment, requiring only o(1) additional search cost for new devices.
• Hierarchical mutation in the weighted NSGA-II algorithm improves search efficiency, producing a Pareto front that dominates random mutation baselines.