[Paper Review] Neural Architecture Search for Lightweight Non-Local Networks

Yingwei Li, Xiaojie Jin+8 more|arXiv (Cornell University)|Apr 4, 2020

Advanced Neural Network Applications50 references4 citations

TL;DR

This paper proposes AutoNL, a method that designs a lightweight Non-Local (LightNL) block and uses differentiable neural architecture search to optimize its placement and configuration in mobile neural networks. By reducing computation by 400× compared to standard Non-Local blocks and enabling end-to-end search, AutoNL achieves 77.7% ImageNet top-1 accuracy at 350M FLOPs—surpassing MobileNetV3, MnasNet, and FBNet with lower FLOPs and faster search (32 GPU hours).

ABSTRACT

Non-Local (NL) blocks have been widely studied in various vision tasks. However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks. We propose AutoNL to overcome the above two obstacles. Firstly, we propose a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features. With the novel design choices, the proposed LightNL block is 400x computationally cheaper} than its conventional counterpart without sacrificing the performance. Secondly, by relaxing the structure of the LightNL block to be differentiable during training, we propose an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner. Notably, using only 32 GPU hours, the searched AutoNL model achieves 77.7% top-1 accuracy on ImageNet under a typical mobile setting (350M FLOPs), significantly outperforming previous mobile models including MobileNetV2 (+5.7%), FBNet (+2.8%) and MnasNet (+2.1%). Code and models are available at https://github.com/LiYingwei/AutoNL.

Motivation & Objective

• To address the high computational cost of Non-Local (NL) blocks in mobile vision systems, which limits their deployment in resource-constrained environments.
• To overcome the challenge of manually selecting optimal locations and configurations for NL blocks in mobile networks, which is intractable due to high complexity.
• To design a search-compatible, computationally efficient NL block that maintains performance while drastically reducing FLOPs.
• To develop a differentiable neural architecture search algorithm that jointly optimizes the placement, channel sampling, and spatial stride of LightNL blocks in an end-to-end manner.
• To achieve state-of-the-art performance on ImageNet under mobile settings with minimal search cost and FLOP budget.

Proposed method

• Proposes a Lightweight Non-Local (LightNL) block that replaces heavy operations like full matrix multiplications and standard 1×1 convolutions with depthwise convolutions and small, compact operations.
• Introduces a differentiable relaxation of the LightNL block’s structure, enabling gradient-based optimization over discrete choices such as channel sampling rate (25% or 12.5%) and spatial stride (1 or 2).
• Uses a differentiable search space that includes five candidate configurations per potential insertion point: presence/absence of LightNL, channel sampling, and spatial sampling.
• Employs a multi-objective loss combining cross-entropy and computation cost to guide the search toward high accuracy and low FLOPs.
• Reuses intermediate affinity matrices across multiple candidate configurations in a single forward pass to reduce redundant computation and accelerate the search process.
• Performs end-to-end search on ImageNet using MobileNetV2 blocks as the base architecture, jointly optimizing depth multipliers, expansion ratios, and SE module ratios.

Experimental results

Research questions

• RQ1 Can a Non-Local block be made computationally efficient enough for deployment on mobile devices without sacrificing performance?
• RQ2 What is the optimal configuration (location, channel sampling, spatial stride) for inserting Non-Local blocks into a mobile neural network?
• RQ3 Can differentiable neural architecture search be effectively applied to jointly optimize the structure and placement of lightweight Non-Local blocks?
• RQ4 How does the performance of a NAS-optimized model with LightNL blocks compare to handcrafted mobile models and other NAS-based models under the same FLOP budget?
• RQ5 Can the search process be accelerated while maintaining high-quality model discovery for mobile vision tasks?

Key findings

• The proposed LightNL block reduces computation by 400× compared to standard Non-Local blocks while preserving performance, making it viable for mobile deployment.
• AutoNL-S achieves 76.5% top-1 accuracy on ImageNet with only 267M FLOPs, outperforming MobileNetV3 (76.6% at 356M FLOPs) with 25% fewer FLOPs.
• AutoNL-L achieves 77.7% top-1 accuracy at 353M FLOPs—surpassing MobileNetV3 by 1.1% accuracy and FBNet by 2.8% with comparable FLOP cost.
• AutoNL-L improves upon EfficientNet-B0 by 1.4% top-1 accuracy while using 10% fewer FLOPs, demonstrating superior efficiency-accuracy trade-off.
• The search process takes only 32 GPU hours, which is 2,800× faster than the 91,000 GPU hours required for MixNet, while achieving better accuracy.
• Under extremely low FLOP settings (e.g., 192×192 input, 0.5 depth multiplier), AutoNL-S achieves 69.6% top-1 accuracy—5.7% higher than MobileNetV2 and 3.7% higher than FBNet.