[Paper Review] Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search

Aditya Rawal, Joel Lehman+3 more|arXiv (Cornell University)|May 27, 2020

Service-Oriented Architecture and Web Services22 references4 citations

TL;DR

This paper introduces the Synthetic Petri Dish, a novel surrogate model for Neural Architecture Search (NAS) that accelerates evaluation by testing architectural motifs in miniaturized networks trained on synthetic data. By preserving the learning dynamics of real models in a controlled, computationally efficient setting, it achieves higher prediction accuracy than black-box surrogates, especially with limited ground-truth data, and enables NAS to find high-performing architectures with 1/10th the compute of state-of-the-art methods.

ABSTRACT

Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available. Our hope is that this work can inspire a new research direction in studying the performance of extracted components of models in an alternative controlled setting.

Motivation & Objective

• To address the high computational cost of evaluating neural architecture motifs via full training in NAS.
• To improve surrogate model accuracy by embedding intrinsic learning dynamics from real models into a simplified evaluation environment.
• To enable rapid, reliable performance prediction of new motifs using minimal ground-truth evaluations.
• To explore whether a synthetic diagnostic environment can generalize better than black-box prediction models in low-data regimes.

Proposed method

• Architectural motifs are extracted from full-scale trained networks and instantiated into small, compact networks (motif-networks).
• Synthetic data is generated via a learned distribution such that the relative performance ordering of motifs in the synthetic setting matches their ground-truth performance ordering.
• The Synthetic Petri Dish evaluates each motif by training it on a few synthetic samples and measuring its performance in this miniaturized setting.
• The surrogate model is trained using only a small set of ground-truth evaluations (e.g., 15 points), enabling efficient warm-starting of the NAS process.
• NAS algorithms (GA and NAO) are hybridized with the Synthetic Petri Dish, using only the top-performing motifs from the surrogate for subsequent full training.
• The method leverages the intrinsic learning dynamics of real models, embedding real priors into the prediction process rather than relying on abstract architectural parsing.

Experimental results

Research questions

• RQ1 Can a miniaturized, synthetic evaluation environment preserve the performance ordering of neural architecture motifs better than black-box surrogate models?
• RQ2 Does embedding real learning dynamics into a surrogate model improve generalization, especially when ground-truth data is scarce?
• RQ3 Can the Synthetic Petri Dish significantly reduce the number of expensive ground-truth evaluations required in NAS while maintaining or improving final performance?
• RQ4 How does the performance of the Synthetic Petri Dish compare to state-of-the-art NAS methods like NAO under equivalent compute constraints?

Key findings

• The Synthetic Petri Dish achieved a test perplexity of 57.1 for the best recurrent cell found, outperforming the original NAO method (56.0) when both were given the same number of ground-truth evaluations.
• When allowed to use more ground-truth evaluations, the Synthetic Petri Dish-NAO variant matched original NAO’s performance with only 1/10th of the compute, demonstrating significant efficiency gains.
• In the sigmoid slope experiment, the Synthetic Petri Dish correctly inferred the existence of a performance peak with a sharp falloff on the left, whereas a neural network surrogate failed to generalize beyond the training data.
• The method’s predictions were more accurate than black-box models because they embedded real learning dynamics and priors from actual training processes.
• The approach enabled hybrid NAS with GA and NAO that outperformed both the baseline methods and their own variants, showing strong synergy between the surrogate and optimization algorithm.
• The Synthetic Petri Dish demonstrated superior generalization in low-data regimes, where black-box models often fail due to lack of inductive bias.