[Paper Review] LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization

Muhammad Umair Nasir, Sam Earle+3 more|arXiv (Cornell University)|Jun 1, 2023

Topic Modeling4 citations

TL;DR

LLMatic proposes a novel Neural Architecture Search (NAS) framework that combines large language models (LLMs) for code-based neural architecture generation with Quality-Diversity (QD) optimization to discover diverse, high-performing architectures. By leveraging a procedural, prompt-driven approach using a QD archive of prompts and network behaviors, LLMatic achieves near-state-of-the-art accuracy on CIFAR-10, CIFAR-100, and ImageNet16-120 using only 2,000 evaluations, demonstrating efficient and effective NAS without prior exposure to benchmark models.

ABSTRACT

Large Language Models (LLMs) have emerged as powerful tools capable of accomplishing a broad spectrum of tasks. Their abilities span numerous areas, and one area where they have made a significant impact is in the domain of code generation. Here, we propose using the coding abilities of LLMs to introduce meaningful variations to code defining neural networks. Meanwhile, Quality-Diversity (QD) algorithms are known to discover diverse and robust solutions. By merging the code-generating abilities of LLMs with the diversity and robustness of QD solutions, we introduce exttt{LLMatic}, a Neural Architecture Search (NAS) algorithm. While LLMs struggle to conduct NAS directly through prompts, exttt{LLMatic} uses a procedural approach, leveraging QD for prompts and network architecture to create diverse and high-performing networks. We test exttt{LLMatic} on the CIFAR-10 and NAS-bench-201 benchmarks, demonstrating that it can produce competitive networks while evaluating just $2,000$ candidates, even without prior knowledge of the benchmark domain or exposure to any previous top-performing models for the benchmark. The open-sourced code is available in \url{https://github.com/umair-nasir14/LLMatic}.

Motivation & Objective

• To address the inefficiency and lack of diversity in traditional Neural Architecture Search (NAS) methods that rely on trial-and-error evaluation.
• To explore whether large language models (LLMs) can be effectively leveraged to generate meaningful architectural variations for neural networks.
• To integrate LLM-generated code with Quality-Diversity (QD) optimization to discover a diverse set of high-performing, robust architectures.
• To demonstrate that a small, efficient LLM (6.1B parameters) can achieve competitive NAS performance with minimal search budget.
• To validate the necessity and effectiveness of each component in the proposed LLMatic framework through ablation studies.

Proposed method

• LLMatic uses a code-generating LLM (CodeGen-6.1B) to produce neural network architectures in the form of searchable code snippets, constrained to a predefined set of operations.
• It employs a Quality-Diversity (QD) algorithm, specifically a MAP-Elites variant, to maintain two archives: one for prompts and one for trained network architectures.
• The prompt archive stores diverse, high-quality prompts that guide the LLM in generating varied network cells, while the network archive stores architectures with distinct behavior descriptors (e.g., FLOPS, parameter count).
• Each generation involves sampling prompts from the QD archive, prompting the LLM to generate a new network cell, training and evaluating the resulting architecture, and updating the network archive based on performance and behavioral diversity.
• The method uses a procedural, iterative loop: prompt selection → LLM-based architecture generation → training and evaluation → archive update → next generation.
• The framework is designed to be computationally efficient, relying on only 2,000 total evaluations and a smaller LLM, while still achieving competitive performance.

Figure 1: The illustration of the best accuracy per generation for LLMatic and all ablation studies. Each experiment is conducted with 10 seeds. The shaded region is the standard deviation while the solid line represents the mean. EfficientNet-B0 is the best-performing EfficientNet on CIFAR-10.

Experimental results

Research questions

• RQ1 Can large language models be effectively used to generate diverse and high-performing neural network architectures through code generation?
• RQ2 How does combining LLM-based code generation with Quality-Diversity optimization improve search efficiency and diversity in Neural Architecture Search?
• RQ3 To what extent can a smaller, 6.1B-parameter LLM achieve competitive NAS performance with a minimal evaluation budget of 2,000 trials?
• RQ4 How does LLMatic compare to state-of-the-art NAS methods like Λ-DARTS and GPT-4-based GENIUS in terms of accuracy and diversity?
• RQ5 What is the contribution of each component in LLMatic, and is the framework robust to architectural constraints and limited computational resources?

Key findings

• On the NAS-bench-201 benchmark, LLMatic achieved a test accuracy of 94.26±0.13 on CIFAR-10, ranking 2nd among all methods tested, just below the optimal 94.47.
• On CIFAR-100, LLMatic achieved 71.62±1.73 accuracy, ranking 2nd, and on ImageNet16-120, it achieved 45.87±0.96 accuracy, ranking 11th, demonstrating strong generalization across datasets.
• LLMatic discovered more than 20 competitive networks on CIFAR-10, with a broad distribution across the behavior space, indicating effective diversity in architecture search.
• The ablation study confirmed that each component of LLMatic—LLM generation, prompt QD archive, and network QD archive—is essential, as removing any component degraded performance significantly.
• Despite using only 2,000 evaluations and a smaller 6.1B-parameter LLM, LLMatic outperformed simple GPT-4-based NAS (GENIUS) and matched the performance of Λ-DARTS, a state-of-the-art method.
• The framework demonstrated that efficient, low-resource NAS is feasible by combining LLMs with QD optimization, even without prior exposure to top-performing models or extensive hyperparameter tuning.

Figure 2: The illustration of how many trainable networks are created in a generation. The total number of networks created is 100 per generation.