[Paper Review] Fine-Grained Stochastic Architecture Search

Shraman Ray Chaudhuri, Elad Eban+3 more|arXiv (Cornell University)|Jun 17, 2020

Advanced Neural Network Applications41 references4 citations

TL;DR

Fine-Grained Stochastic Architecture Search (FiGS) introduces a differentiable, fine-grained neural architecture search method that applies structured sparse regularization via Logistic-Sigmoid distributions to jointly select and modify operators and feature maps, enabling exploration of a vastly larger architecture space than prior differentiable NAS methods. FiGS achieves state-of-the-art accuracy on ImageNet (75.4% top-1 with 2.6M parameters) and improves mAP by +4 on COCO object detection using SSDLite, outperforming MobileNetV3 and MnasNet.

ABSTRACT

State-of-the-art deep networks are often too large to deploy on mobile devices and embedded systems. Mobile neural architecture search (NAS) methods automate the design of small models but state-of-the-art NAS methods are expensive to run. Differentiable neural architecture search (DNAS) methods reduce the search cost but explore a limited subspace of candidate architectures. In this paper, we introduce Fine-Grained Stochastic Architecture Search (FiGS), a differentiable search method that searches over a much larger set of candidate architectures. FiGS simultaneously selects and modifies operators in the search space by applying a structured sparse regularization penalty based on the Logistic-Sigmoid distribution. We show results across 3 existing search spaces, matching or outperforming the original search algorithms and producing state-of-the-art parameter-efficient models on ImageNet (e.g., 75.4% top-1 with 2.6M params). Using our architectures as backbones for object detection with SSDLite, we achieve significantly higher mAP on COCO (e.g., 25.8 with 3.0M params) than MobileNetV3 and MnasNet.

Motivation & Objective

• To address the limitation of existing differentiable NAS methods that explore only a narrow subspace of architectures due to fixed operator sets.
• To enable joint search over both operator selection and architectural modifications (e.g., channel pruning) for greater expressiveness.
• To reduce search cost while maintaining or improving model efficiency and accuracy across diverse resource constraints.
• To provide a plug-and-play method applicable to any existing search space without redesigning the search space.
• To enable the discovery of Pareto-efficient models for both model size and FLOPs/latency, supporting broader deployment on mobile and edge devices.

Proposed method

• FiGS introduces a stochastic masking mechanism where each output channel or neuron is assigned a Bernoulli mask, relaxed via the Logistic-Sigmoid distribution to enable differentiable training.
• The method applies a structured sparse regularization penalty based on the Logistic-Sigmoid distribution to learn optimal masking probabilities using gradient descent.
• Masks are applied after each layer and operation, allowing fine-grained control over both intermediate features and operator outputs.
• The final architecture is exported by sampling from the learned mask distributions at inference time.
• The approach is compatible with any existing DNAS search space, requiring only insertion of mask layers after each layer.
• FiGS enables early stopping with minimal accuracy loss, reducing search time by up to 5x while maintaining high performance.

Experimental results

Research questions

• RQ1 Can a differentiable NAS method explore a significantly larger architecture space by modifying not only operator choices but also feature map dimensions during optimization?
• RQ2 How does the performance of FiGS compare to original search algorithms when applied to established search spaces such as One-Shot and MnasNet?
• RQ3 To what extent can FiGS discover Pareto-efficient models for both model size and FLOPs/latency, and how do these trade-offs vary across different search spaces?
• RQ4 Can FiGS produce state-of-the-art small models on ImageNet when trained from scratch, and how do they perform in downstream tasks like object detection?
• RQ5 Does FiGS enable significant reduction in search cost through early stopping without substantial accuracy degradation?

Key findings

• FiGS achieves 75.4% top-1 accuracy on ImageNet with only 2.6 million parameters, setting a new state-of-the-art for parameter-efficient models.
• On COCO object detection using SSDLite, FiGS-One-Shot-Large achieves 25.8 mAP with 3.02M parameters, outperforming MobileNetV3-Large (21.8 mAP) and MnasNet-A1 (23.0 mAP) despite having fewer parameters.
• When applied to the One-Shot search space, FiGS matches or exceeds the performance of the original search algorithm, demonstrating strong generalization across search spaces.
• FiGS enables early stopping at 40 epochs with only a 0.3–0.6% drop in accuracy compared to full training, reducing search time by 2.5x.
• FiGS outperforms standard pruning baselines when applied to ResNet models, showing superior performance in structured architectural search.
• The method reveals inherent biases in existing search spaces: some favor model size efficiency, others favor FLOPs/latency efficiency, enabling better understanding of search space design trade-offs.