[Paper Review] Multi-Scale Dense Convolutional Networks for Efficient Prediction.

Gao Huang, Danlu Chen+4 more|arXiv (Cornell University)|Mar 29, 2017

Advanced Neural Network Applications116 citations

TL;DR

This paper proposes a multi-scale dense convolutional network with early-exit classifiers for efficient image classification under test-time computational constraints. By integrating multiple classifiers with varying resource demands through dense skip connections and multi-scale feature fusion, the framework enables anytime and budgeted batch classification, achieving state-of-the-art performance on three image classification benchmarks.

ABSTRACT

In this paper we investigate image classification with computational resource limits at test time. Two such settings are: 1. anytime classification, where the network's prediction for a test example is progressively updated, facilitating the output of a prediction at any time; and 2. budgeted batch classification, where a fixed amount of computation is available to classify a set of examples that can be spent unevenly across easier and harder inputs. In contrast to most prior work, such as the popular Viola and Jones algorithm, our approach is based on convolutional neural networks. We train multiple classifiers with varying resource demands, which we adaptively apply during test time. To maximally re-use computation between the classifiers, we incorporate them as early-exits into a single deep convolutional neural network and inter-connect them with dense connectivity. To facilitate high quality classification early on, we use a two-dimensional multi-scale network architecture that maintains coarse and fine level features all-throughout the network. Experiments on three image-classification tasks demonstrate that our framework substantially improves the existing state-of-the-art in both settings.

Motivation & Objective

• To address the challenge of efficient image classification when computational resources are limited at test time.
• To enable anytime classification, where predictions can be generated at any time during inference.
• To support budgeted batch classification, where a fixed computation budget is allocated adaptively across inputs based on difficulty.
• To maximize computation reuse across multiple classifiers through dense connectivity in a single deep network.
• To improve classification accuracy and efficiency using multi-scale feature representation throughout the network.

Proposed method

• Training multiple classifiers with varying computational costs to support flexible inference under resource constraints.
• Integrating classifiers as early-exit heads within a single deep convolutional neural network to enable progressive prediction.
• Using dense connectivity between all layers and classifiers to enhance feature reuse and gradient flow.
• Employing a two-dimensional multi-scale architecture to preserve both coarse and fine-grained features across all network stages.
• Adapting the selection of exit heads dynamically during inference based on input complexity and available budget.
• Optimizing the network end-to-end to balance accuracy and computational efficiency across all exit heads.

Experimental results

Research questions

• RQ1 Can a single deep network with multiple early-exit classifiers achieve better efficiency and accuracy than existing methods in anytime classification?
• RQ2 How effectively can a multi-scale dense architecture maintain high feature quality across different stages of inference?
• RQ3 To what extent can computational resources be optimally allocated across diverse inputs in budgeted batch classification?
• RQ4 Does dense connectivity between classifiers improve feature reuse and model performance under resource constraints?
• RQ5 Can the proposed framework outperform state-of-the-art methods in both anytime and budgeted classification settings?

Key findings

• The proposed framework achieves state-of-the-art performance on three image classification benchmarks in both anytime and budgeted batch classification settings.
• The use of multi-scale features throughout the network significantly improves early prediction accuracy.
• Dense connectivity enables effective feature reuse across classifiers, enhancing model efficiency and accuracy.
• The adaptive selection of exit heads based on input difficulty leads to better utilization of the available computational budget.
• The framework demonstrates substantial improvements over prior methods in terms of accuracy under constrained inference time or computation.
• The results confirm that combining early exits with multi-scale and dense architectures is highly effective for efficient inference.