[Paper Review] Peephole: Predicting Network Performance Before Training

Boyang Deng, Junjie Yan+1 more|arXiv (Cornell University)|Dec 9, 2017

Anomaly Detection Techniques and Applications23 references84 citations

TL;DR

Peephole predicts a neural network’s final performance purely from architecture using Unified Layer Code and an LSTM, enabling fast, architecture-driven performance estimation and reliable rankings across datasets.

ABSTRACT

The quest for performant networks has been a significant force that drives the advancements of deep learning in recent years. While rewarding, improving network design has never been an easy journey. The large design space combined with the tremendous cost required for network training poses a major obstacle to this endeavor. In this work, we propose a new approach to this problem, namely, predicting the performance of a network before training, based on its architecture. Specifically, we develop a unified way to encode individual layers into vectors and bring them together to form an integrated description via LSTM. Taking advantage of the recurrent network's strong expressive power, this method can reliably predict the performances of various network architectures. Our empirical studies showed that it not only achieved accurate predictions but also produced consistent rankings across datasets -- a key desideratum in performance prediction.

Motivation & Objective

• Mitigate the cost and time of network design by predicting performance without training.
• Develop a universal layer representation to unify heterogeneous architectures.
• Integrate layer representations across variable-depth/topology networks with an LSTM.
• Enable quick feedback for architecture search to prioritize promising designs.
• Validate predictions on CIFAR-10 and MNIST and assess ranking consistency.

Proposed method

• Unified Layer Code encodes each layer as a fixed-dimension vector using integer coding (TY, KW, KH, CH) and Layer Embedding.
• An LSTM aggregates layer representations along network topology to produce a structural feature.
• Epoch information is encoded and combined with the structural feature in an MLP to predict final accuracy.
• Block-based Generation builds a diverse set of plausible architectures via Markov-driven blocks to train the predictor within a budget.
• Training objective uses a smooth L1 loss between predicted and actual final-epoch accuracies for each architecture.

Experimental results

Research questions

• RQ1 Can network performance be predicted accurately before any training based solely on architecture?
• RQ2 How can heterogeneous layer types be uniformly represented for a predictive model?
• RQ3 Can sequentially integrating layer information with an LSTM capture architectural patterns across depths/topologies?
• RQ4 Is the predicted performance ranking stable and correlated with true performance across datasets?

Key findings

• Peephole achieves lower MSE than baseline methods on CIFAR-10 (MSE 0.0010; Tau 0.7696; R^2 0.8596).
• Peephole demonstrates strong ranking correlation, outperforming Bayesian NN and ν-SVR in multiple metrics on CIFAR-10.
• On MNIST, Peephole also outperforms baselines across the reported metrics, indicating robustness across datasets.
• The learned structural features from the LSTM reflect architectural patterns, such as heightened responses at convolution layers, supporting the representation’s meaningfulness.