[Paper Review] Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks

Tom Véniat, Ludovic Denoyer|arXiv (Cornell University)|May 31, 2017

Advanced Neural Network ApplicationsComputer Science41 references74 citations

TL;DR

This paper introduces Budgeted Super Networks (BSN) that automatically discover neural architectures under a budget constraint for either computation, memory, or distributed cost, using a stochastic optimization framework and policy-gradient training.

ABSTRACT

We propose to focus on the problem of discovering neural network architectures efficient in terms of both prediction quality and cost. For instance, our approach is able to solve the following tasks: learn a neural network able to predict well in less than 100 milliseconds or learn an efficient model that fits in a 50 Mb memory. Our contribution is a novel family of models called Budgeted Super Networks (BSN). They are learned using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the nature of this cost. We present a set of experiments on computer vision problems and analyze the ability of our technique to deal with three different costs: the computation cost, the memory consumption cost and a distributed computation cost. We particularly show that our model can discover neural network architectures that have a better accuracy than the ResNet and Convolutional Neural Fabrics architectures on CIFAR-10 and CIFAR-100, at a lower cost.

Motivation & Objective

• Motivate the need for neural architectures that balance predictive quality with inference cost.
• Propose a budgeted learning framework that enforces a maximum cost without assuming a specific cost form.
• Enable discovery of architectures that optimize accuracy under computation, memory, or distributed cost constraints.
• Demonstrate effectiveness on CIFAR-10/100 and image segmentation tasks compared to ResNet and CNF baselines.

Proposed method

• Define a large Super Network (S-network) as a DAG where edges are modules with learnable parameters; architectures correspond to subgraphs of this S-network.
• Introduce Budgeted Super Networks (BSN) that embed a cost constraint C and a maximum allowed cost C̄ into a soft objective using a penalty term.
• Relax the combinatorial search by introducing Stochastic Super Networks (SS-network) where a binary mask H over edges is sampled from a parameterized distribution Γ before each prediction.
• Formulate a budgeted objective that minimizes predictive loss plus a cost penalty λ max(0, C(H ⊙ E) − C̄).
• Derive a policy-gradient-like gradient for jointly updating θ (module parameters) and Γ (edge-sampling parameters).
• Demonstrate that solving the stochastic problem yields an optimal constrained solution (Proposition 1).

Experimental results

Research questions

• RQ1 Can BSN discover network architectures that meet a specified cost budget while maintaining or improving accuracy?
• RQ2 How does BSN perform across different cost types (computation, memory, distributed computation) compared to standard architectures?
• RQ3 Do stochastic edge-sampling methods converge to deterministic, budget-compliant architectures during training?
• RQ4 Are discovered budgeted architectures competitive with or superior to ResNet/CNF baselines on CIFAR-10/100 and Part Label segmentation?

Key findings

• BSN can discover architectures that achieve higher accuracy than ResNet/CNF baselines at similar or lower costs on CIFAR-10/ CIFAR-100.
• For CIFAR-10, B-ResNet achieves 92.39% accuracy at 39.25 million FLOPs, outperforming ResNet-20 at 40.9 million FLOPs (92.19%).
• For CIFAR-10, B-CNF achieves notable accuracy improvements at substantially reduced FLOPs versus CNF baselines (e.g., 93.14% at 103 million FLOPs vs CNF 92.54% at 406 million FLOPs).
• On CIFAR-100, budgeted variants attain competitive accuracy with reduced FLOPs compared to standard ResNets and MSDNet baselines.
• BSN learns architectures that can be distributed across multiple cores, revealing parallelized structures when n=2 or n=4 cores are used, and adapts to the given distributed cost.
• The training dynamics show the edge-probabilities become deterministic in the end, indicating convergence to a unique budgeted architecture.