[Paper Review] Backprop Evolution

Maximilian Alber, Irwan Bello+4 more|arXiv (Cornell University)|Aug 8, 2018

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes an evolutionary search method to automatically discover novel back-propagation update rules by treating the back-propagation equation as a composition of primitive functions in a domain-specific language. It finds several equations that outperform standard back-propagation in early training, achieving up to 15% higher accuracy on larger models after 20 epochs, while matching standard performance at convergence.

ABSTRACT

The back-propagation algorithm is the cornerstone of deep learning. Despite its importance, few variations of the algorithm have been attempted. This work presents an approach to discover new variations of the back-propagation equation. We use a domain specific lan- guage to describe update equations as a list of primitive functions. An evolution-based method is used to discover new propagation rules that maximize the generalization per- formance after a few epochs of training. We find several update equations that can train faster with short training times than standard back-propagation, and perform similar as standard back-propagation at convergence.

Motivation & Objective

• To automatically discover improved back-propagation update rules that enhance generalization performance in deep learning.
• To address the lack of practical variations in back-propagation despite its foundational role in training neural networks.
• To explore whether evolutionary search over a domain-specific language of mathematical operations can yield better training dynamics than standard back-propagation.
• To evaluate whether discovered rules generalize across model architectures and training durations.
• To identify update rules that accelerate convergence without sacrificing final performance, especially useful in early-stopping or hyperparameter search settings.

Proposed method

• The method uses a domain-specific language (DSL) to represent back-propagation update equations as compositions of primitive functions: operands (e.g., weights, activations, gradients), unary functions (e.g., transpose, ReLU, dropout), and binary functions (e.g., addition, element-wise multiplication).
• An evolutionary controller generates and mutates candidate update equations by selecting from predefined sets of functions and operands, forming expressions for the error gradient update $ b^{p}_i $.
• Each candidate equation is evaluated by training a fixed neural network architecture (e.g., WRN 16-2, WRN 10-1) using the proposed update rule, with validation accuracy reported after a fixed number of epochs.
• The search is conditioned to maximize generalization performance after a few epochs, with results tested across different model sizes and training durations to assess robustness and transferability.
• The evolutionary process uses a controller-worker architecture: the controller sends batches of mutated equations to workers, which train models and return validation accuracy for selection and further mutation.
• The method evaluates both standard SGD and SGD with momentum, and compares results to the baseline standard back-propagation.

Experimental results

Research questions

• RQ1 Can evolutionary search over a domain-specific language discover back-propagation update rules that generalize better than standard back-propagation in early training stages?
• RQ2 Do the discovered update rules generalize to larger neural network architectures, such as deeper or wider versions of the same model?
• RQ3 Do the discovered rules maintain performance at convergence, or do they only improve early training dynamics?
• RQ4 How does the inclusion of momentum in optimization affect the robustness of the discovered update rules?
• RQ5 Are there consistent structural patterns in the discovered equations that could inform future design of more effective back-propagation variants?

Key findings

• The evolutionary search discovered update equations that outperform standard back-propagation in terms of validation accuracy after 20 epochs of training on WRN 16-2, with improvements of up to 1.2% for SGD and 0.8% for SGD with momentum.
• On the larger WRN 28-10 model, the best discovered equations achieved up to 15% higher accuracy than standard back-propagation when trained for 20 epochs with SGD, and 10% higher with momentum.
• When trained for 100 epochs, the discovered equations performed similarly to standard back-propagation, indicating that they primarily accelerate early convergence rather than improving final generalization.
• The best-performing equations were found to be robust across different model architectures and training regimes, suggesting that smaller models can be used effectively in the search phase.
• The search results for SGD with momentum were comparable to the baseline, indicating that momentum may reduce sensitivity to the choice of update rule.
• Specific equations such as $ (0.5g^{p}_{i}) / ( ext{softplus}(rac{ ext{d}h_{i}}{ ext{d}h^{p}_{i}}}) + 0.1) $ achieved 87.72% accuracy, while $ (g^{p}_{i} imes ext{clip}_{1.0}(b^{p}_{i+1} rac{ ext{d}h^{p}_{i+1}}{ ext{d}h_{i}})) $ reached 88.93% accuracy, both outperforming the baseline in early training.