[Paper Review] Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data

Felipe Petroski Such, Aditya Rawal+3 more|arXiv (Cornell University)|Dec 17, 2019

Adversarial Robustness in Machine Learning44 references48 citations

TL;DR

GTNs learn to generate synthetic training data and curricula via meta-gradients, accelerating learning and enabling faster NAS by evaluating architectures with GTN-produced data. GTN-NAS achieves competitive state-of-the-art results with substantially reduced computation.

ABSTRACT

This paper investigates the intriguing question of whether we can create learning algorithms that automatically generate training data, learning environments, and curricula in order to help AI agents rapidly learn. We show that such algorithms are possible via Generative Teaching Networks (GTNs), a general approach that is, in theory, applicable to supervised, unsupervised, and reinforcement learning, although our experiments only focus on the supervised case. GTNs are deep neural networks that generate data and/or training environments that a learner (e.g. a freshly initialized neural network) trains on for a few SGD steps before being tested on a target task. We then differentiate through the entire learning process via meta-gradients to update the GTN parameters to improve performance on the target task. GTNs have the beneficial property that they can theoretically generate any type of data or training environment, making their potential impact large. This paper introduces GTNs, discusses their potential, and showcases that they can substantially accelerate learning. We also demonstrate a practical and exciting application of GTNs: accelerating the evaluation of candidate architectures for neural architecture search (NAS), which is rate-limited by such evaluations, enabling massive speed-ups in NAS. GTN-NAS improves the NAS state of the art, finding higher performing architectures when controlling for the search proposal mechanism. GTN-NAS also is competitive with the overall state of the art approaches, which achieve top performance while using orders of magnitude less computation than typical NAS methods. Speculating forward, GTNs may represent a first step toward the ambitious goal of algorithms that generate their own training data and, in doing so, open a variety of interesting new research questions and directions.

Motivation & Objective

• Investigate whether a data-generating network can produce synthetic data that teaches a learner efficiently.
• Develop a meta-learning framework (GTN) with inner learner training and outer generator optimization.
• Show that GTN-generated data can accelerate neural architecture search (NAS) under real-data baselines.
• Demonstrate stability improvements and curriculum advantages for GTN training.
• Apply GTN to CIFAR10 NAS to demonstrate accelerated, competitive architecture search.

Proposed method

• Two nested training loops: an inner loop trains a learner on GTN-generated data; an outer loop updates the GTN to maximize the learner performance on real data.
• A generator G(z,y) creates synthetic data x from Gaussian noise z and labels y; the learner is trained for a fixed number of inner steps with SGD or Adam.
• Outer-loop meta-objective evaluates the learner on real data and backpropagates through the inner loop to update GTN parameters; hyperparameters of the inner-loop optimizer are also learned.
• Weight normalization is applied to stabilize meta-gradient training, parameterizing weights as W=g·V/||V|| to update g and V instead of W.
• Curriculum learning is enabled by learning the input z-codes or fixed input samples, allowing the GTN to control data generation and presentation order (full curriculum outperforming others).
• GTN-based data is used to accelerate NAS by replacing real data with GTN data during architecture evaluation, enabling faster proxy assessments of CIFAR10 architectures.

Experimental results

Research questions

• RQ1 Can a neural network generator learn to produce synthetic data that accelerates learning across different learner architectures and initializations?
• RQ2 Does a learned curriculum improve GTN effectiveness over purely random data generation?
• RQ3 Can GTN-generated data dramatically reduce NAS evaluation cost while preserving or improving architecture quality?
• RQ4 What stability techniques (e.g., weight normalization) improve meta-gradient training in GTNs?
• RQ5 How well do GTN-generated data generalize to NAS search spaces and surrogate evaluations?

Key findings

• GTNs produce synthetic data that enable faster learning, achieving better few-step accuracy than real-data or dataset-distillation baselines.
• Weight normalization substantially improves the stability and robustness of meta-gradient GTN training.
• A full GTN curriculum (learning both sample order and composition) outperforms other curriculum variants.
• GTN-based data accelerates MNIST and CIFAR10 few-step learning and enables efficient NAS proxy evaluations that correlate with real data performance.
• In CIFAR10 NAS, GTN evaluation speeds up architecture assessment by about 9x compared to real-data evaluation and achieves competitive or superior results when combined with Cutout augmentation.
• GTN-based proxies achieve state-of-the-art results within the tested search setup and show improved correlation with true performance for top architectures.