[Paper Review] AdaNet: Adaptive Structural Learning of Artificial Neural Networks

Corinna Cortes, Xavi Gonzalvo+3 more|arXiv (Cornell University)|Jul 5, 2016

Machine Learning and ELM46 references107 citations

TL;DR

AdaNet learns both the structure and weights of neural networks with data-dependent generalization guarantees, demonstrated on CIFAR-10 binary tasks where it competitively matches standard approaches.

ABSTRACT

We present new algorithms for adaptively learning artificial neural networks. Our algorithms (AdaNet) adaptively learn both the structure of the network and its weights. They are based on a solid theoretical analysis, including data-dependent generalization guarantees that we prove and discuss in detail. We report the results of large-scale experiments with one of our algorithms on several binary classification tasks extracted from the CIFAR-10 dataset. The results demonstrate that our algorithm can automatically learn network structures with very competitive performance accuracies when compared with those achieved for neural networks found by standard approaches.

Motivation & Objective

• Motivate adaptive learning of network architecture to avoid manual design and hyperparameter tuning.
• Provide theoretical generalization guarantees for learning both architecture and weights.
• Develop and analyze AdaNet algorithms that grow network structure while controlling complexity.
• Demonstrate empirical competitiveness of adaptive networks on CIFAR-10 derived tasks.

Proposed method

• Define a broad, general network hypothesis class that allows connections across layers and outputs.
• Derive data-dependent generalization bounds using layer-wise Rademacher complexities and margin guarantees.
• Propose AdaNet objective as a convex surrogate with an explicit data-dependent regularization term based on Rademacher complexities.
• Use a boosting-like block coordinate descent to iteratively add subnetworks, either same-depth or deeper, minimizing a convex surrogate of empirical loss plus complexity penalties.
• Describe two candidate subnetworks per iteration and select the one that most improves the objective, possibly using different weak-learning strategies.

Experimental results

Research questions

• RQ1 Can network architecture be learned adaptively during training while providing generalization guarantees?
• RQ2 How can data-dependent complexity measures guide the addition of network structure to balance fit and capacity?
• RQ3 Does adaptively learned architecture achieve competitive performance compared to fixed architectures and simple baselines on image classification tasks?
• RQ4 What are the practical implications of the theory for constructing and selecting subnetworks during training?

Key findings

• AdaNet provides data-dependent generalization bounds for learning both architecture and weights.
• The bound depends on a weighted average of layer complexities with weights given by the network’s connection strengths, encouraging lower-layer emphasis when beneficial.
• Empirical results on CIFAR-10 binary tasks show AdaNet achieving competitive accuracy compared with logistic regression and standard neural networks.
• AdaNet architectures tend to be sparse and shallow for many pairs, but can become deeper when needed (e.g., cat-dog).
• Different AdaNet variants and weak-learner strategies yield robustness in accuracy across tasks.