[Paper Review] Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?

Yunyang Xiong, Ronak Mehta+1 more|arXiv (Cornell University)|Apr 8, 2019

Advanced Neural Network Applications41 references4 citations

TL;DR

This paper proposes a submodularity-inspired heuristic for resource-constrained neural architecture search (NAS), framing architecture search as optimizing a set function over reusable blocks. By leveraging greedy submodular optimization under FLOPs and parameter constraints, the method identifies efficient, high-accuracy models on CIFAR-100 and ImageNet—achieving state-of-the-art performance with up to 3× fewer parameters and 300× less compute than prior RL-based approaches.

ABSTRACT

The design of neural network architectures is frequently either based on human expertise using trial/error and empirical feedback or tackled via large scale reinforcement learning strategies performed over distinct discrete architecture choices. In the latter case, the optimization is often non-differentiable and also not very amenable to derivative-free optimization methods. Most methods in use today require sizable computational resources. And if we want networks that additionally satisfy resource constraints, the above challenges are exacerbated because the search must now balance accuracy with certain budget constraints on resources. We formulate this problem as the optimization of a set function -- we find that the empirical behavior of this set function often (but not always) satisfies marginal gain and monotonicity principles -- properties central to the idea of submodularity. Based on this observation, we adapt algorithms within discrete optimization to obtain heuristic schemes for neural network architecture search, where we have resource constraints on the architecture. This simple scheme when applied on CIFAR-100 and ImageNet, identifies resource-constrained architectures with quantifiably better performance than current state-of-the-art models designed for mobile devices. Specifically, we find high-performing architectures with fewer parameters and computations by a search method that is much faster.

Motivation & Objective

• To address the high computational cost and lack of resource-awareness in existing neural architecture search (NAS) methods.
• To develop a scalable, differentiable, and efficient NAS framework that respects user-defined constraints on model size and FLOPs.
• To investigate whether submodular optimization principles can guide effective architecture search despite the non-submodular nature of the underlying accuracy function.
• To identify high-performing, mobile-ready neural architectures that balance accuracy, parameter count, and inference efficiency.

Proposed method

• Formulates architecture search as optimizing a set function over reusable functional blocks, where each block choice contributes to the final model’s accuracy and resource cost.
• Empirically observes that the accuracy function exhibits diminishing returns and monotonicity—key properties of submodularity—enabling heuristic application of submodular optimization.
• Adapts a greedy algorithm with performance guarantees from submodular optimization to perform efficient architecture search under FLOPs and parameter constraints.
• Uses early stopping and weight sharing to accelerate evaluation of candidate architectures during search.
• Searches directly on ImageNet without pre-training on smaller datasets, enabling transferability to large-scale benchmarks.
• Deploys final models using CoreML on mobile devices to measure real-world inference latency.

Experimental results

Research questions

• RQ1 Can submodular optimization heuristics effectively guide neural architecture search under strict resource constraints?
• RQ2 Does the empirical marginal gain behavior of neural architecture accuracy functions resemble submodularity, even if not mathematically submodular?
• RQ3 Can a greedy submodular-inspired search achieve better accuracy-efficiency trade-offs than existing reinforcement learning or evolutionary NAS methods?
• RQ4 Can models searched on ImageNet directly achieve performance comparable to manually designed mobile networks like MobileNetV2?

Key findings

• On CIFAR-100, the proposed RCAS method identifies a model with 3.4M parameters and 294M FLOPs that achieves 72.2% top-1 accuracy, outperforming MobileNetV2 by 0.2% under similar constraints.
• On ImageNet, RCNet achieves 74.7% top-1 accuracy with 4.7M parameters and 471M FLOPs, surpassing DARTS by 1.6% and DPP-Net-Panaca by 0.7% under comparable computational budgets.
• The RCNet model achieves comparable inference speed to MobileNetV2 (183.5ms vs. 197.2ms) on an iPhone 5s, while using 14.6MB model size versus 14.7MB.
• The search process requires only 8 GPU days on ImageNet, a 300× reduction compared to prior RL-based NAS methods like DARTS, which require ~3150 GPU days.
• The final architecture uses diverse block types (e.g., [5,1,4,5,1,5,2,4,1,6,4,6,5,3,3,6,3,6]), suggesting block diversity improves performance over repeated block patterns.
• Transferability is demonstrated: a model pre-trained on CIFAR-100 and retrained on ImageNet achieves performance comparable to MobileNetV2 with similar complexity.