[Paper Review] GradNets: Dynamic Interpolation Between Neural Architectures

Diogo Almeida, Nate Sauder|arXiv (Cornell University)|Nov 21, 2015

Neural Networks and Applications16 references3 citations

TL;DR

GradNets introduce a dynamic architecture interpolation framework that gradually transitions neural network components (e.g., linear to ReLU, identity to dropout) during training using a learnable annealing schedule. This approach improves optimization stability, enables training of extremely deep networks (up to 200 layers), and boosts accuracy on CIFAR-10 and MNIST without significant computational overhead.

ABSTRACT

In machine learning, there is a fundamental trade-off between ease of optimization and expressive power. Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train. The nature of that optimization challenge changes over the course of learning. Traditionally in deep learning, one makes a static trade-off between the needs of early and late optimization. In this paper, we investigate a novel framework, GradNets, for dynamically adapting architectures during training to get the benefits of both. For example, we can gradually transition from linear to non-linear networks, deterministic to stochastic computation, shallow to deep architectures, or even simple downsampling to fully differentiable attention mechanisms. Benefits include increased accuracy, easier convergence with more complex architectures, solutions to test-time execution of batch normalization, and the ability to train networks of up to 200 layers.

Motivation & Objective

• Address the fundamental trade-off between optimization ease and model expressiveness in deep learning.
• Overcome limitations of static architectures that sacrifice early training stability for late-stage expressiveness.
• Enable training of very deep networks (e.g., 200-layer MLPs) that fail with standard static architectures.
• Improve generalization and convergence by dynamically adapting architectural components during training.
• Facilitate the use of complex components like spatial transformer networks and batch normalization in a stable manner.

Proposed method

• Use a weighted mean of two architectural components, where the interpolation weight $g = \min(t/\tau, 1)$ is annealed linearly from 0 to 1 over training epochs.
• Apply the GradNet framework to various component pairs: e.g., identity to ReLU (GReLU), identity to dropout, mean pooling to max pooling, etc.
• Maintain consistent tensor shapes across components to allow differentiable interpolation during backpropagation.
• Use a single hyperparameter $\tau$ to control the annealing schedule, with $\tau = 100$ for CIFAR-10 and $\tau = 5$ for MNIST.
• Leverage orthogonal weight initialization and Adam optimization with early stopping to ensure stable training.
• Apply the framework to complex components such as spatial transformer networks and batch normalization, enabling stable training of otherwise unstable modules.

Experimental results

Research questions

• RQ1 Can dynamic interpolation between architectural components improve training stability and final performance in deep networks?
• RQ2 Can GradNets enable convergence in networks that diverge under standard static architectures, particularly with high dropout rates?
• RQ3 Can GradNets support the training of extremely deep feedforward networks (e.g., 200-layer MLPs) on MNIST?
• RQ4 Can GradNets stabilize the training of complex components like spatial transformer networks and batch normalization?
• RQ5 Does dynamic architectural adaptation lead to consistent performance gains across multiple benchmarks (CIFAR-10, MNIST, Cluttered MNIST) without significant computational cost?

Key findings

• GradNets enabled successful training of 200-layer multilayer perceptrons on MNIST, a task that fails with standard static architectures.
• On CIFAR-10, GradNets with gradual ReLU and gradual dropout achieved higher test accuracy than their static counterparts.
• Gradual dropout allowed convergence at dropout rates that caused static networks to diverge, demonstrating improved optimization stability.
• Gradual Batch Normalization enabled stable training of models that otherwise failed due to test-time batch normalization issues.
• The spatial transformer network with GradNet interpolation achieved 98.2% mean accuracy across 10 runs, with no divergence, outperforming static versions.
• The framework incurred almost no computational overhead, as the interpolation is a simple weighted average with no additional parameters or operations.