[Paper Review] Learnable Embedding Space for Efficient Neural Architecture Compression

Shengcao Cao, Xiaofang Wang+1 more|arXiv (Cornell University)|Feb 1, 2019

Advanced Neural Network Applications17 citations

TL;DR

This paper proposes a learnable embedding space for neural architecture search (NAS) that enables efficient, data-driven architecture selection via Bayesian optimization. By learning a compact, transferable representation of architectures, the method outperforms random search, reinforcement learning, and state-of-the-art handcrafted models like ShuffleNet, achieving superior accuracy-accuracy trade-offs in compressed model search.

ABSTRACT

We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training. Code is publicly available here: this https URL .

Motivation & Objective

• To address the inefficiency of existing neural architecture search methods in exploring the vast search space of compressed models.
• To develop a compact, learnable representation of neural architectures that captures structural and performance-related relationships.
• To enable transferable architecture search across different teacher networks without retraining.
• To improve search efficiency and model performance through kernel-based Bayesian optimization over the learned embedding space.

Proposed method

• The method learns a continuous, low-dimensional embedding space for neural architecture representations using a differentiable architecture encoding mechanism.
• A kernel function is defined over the learned embedding space to guide Bayesian optimization in selecting promising architectures for evaluation.
• Bayesian optimization uses the kernel to model the accuracy-architecture relationship and select architectures with high expected improvement.
• The embedding space is trained on a teacher network and can be directly transferred to new search tasks with different teacher models.
• The approach avoids expensive training of all candidate architectures by focusing evaluation on the most promising candidates identified via the embedding-based acquisition function.

Experimental results

Research questions

• RQ1 Can a learnable embedding space improve the efficiency and performance of neural architecture search for compressed models?
• RQ2 How well does the learned embedding generalize across different teacher networks and architecture families?
• RQ3 Can the embedding-based search outperform random search and reinforcement learning in accuracy and sample efficiency?
• RQ4 To what extent can the embedding space be transferred to new search settings without retraining?

Key findings

• The proposed method significantly outperforms random search and reinforcement learning in both accuracy and sample efficiency during architecture search.
• The compressed architectures discovered are more accurate than the state-of-the-art hand-designed architecture ShuffleNet.
• The learned embedding space is transferable to new search tasks, including larger teacher networks and different architecture families, without any additional training.
• The method achieves superior performance with fewer evaluations, demonstrating high sample efficiency in the search process.