[Paper Review] Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

Chun-Fu Chen, Quanfu Fan+3 more|arXiv (Cornell University)|Jul 10, 2018

Advanced Image and Video Retrieval Techniques53 references60 citations

TL;DR

Big-Little Net (bL-Net) uses a two-branch multi-scale architecture to fuse high- and low-resolution features, achieving substantial FLOP reductions with accuracy gains on vision tasks and improved WER with 30% FLOPs savings in speech recognition.

ABSTRACT

In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy. This is achieved by using a multi-branch network, which has different computational complexity at different branches. Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation. The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks,using popular architectures including ResNet and ResNeXt. For object recognition, our approach reduces computation by 33% on object recognition while improving accuracy with 0.9%. Furthermore, our model surpasses state-of-the-art CNN acceleration approaches by a large margin in accuracy and FLOPs reduction. On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains. The codes are available at https://github.com/IBM/BigLittleNet

Motivation & Objective

• Motivate efficient deployment of CNNs by balancing speed and accuracy in multi-scale feature representations.
• Propose a simple, generalizable multi-branch architecture that merges features from different scales to enhance representations.
• Demonstrate that the approach reduces computation while improving accuracy on object recognition and lowers WER with fewer FLOPs in speech recognition.

Proposed method

• Introduce Big-Little Net (bL-Net) consisting of Big-Branch (high accuracy, high cost) and Little-Branch (lower cost) operating at different scales.
• Within each Big-Little Module, merge K branches (different scales) via a weighted linear combination of transformed features.
• Use upsampling and 1x1 convolutions to align feature maps across branches before fusion.
• Control Little-Branch complexity with two factors, alpha (width) and beta (depth), to reduce FLOPs.
• Fuse branch outputs with a residual block (F(·)) after merging.
• Demonstrate compatibility with backbone architectures like ResNet, ResNeXt, and SEResNeXt to achieve ~2x speedups at similar or better accuracy.
• Show that linear combination merging outperforms concatenation in object recognition.

Experimental results

Research questions

• RQ1 Can multi-scale, multi-branch fusion yield better speed-accuracy trade-offs than single-scale baselines?
• RQ2 How do the Big-Branch and Little-Branch interact across scales to preserve information while reducing computation?
• RQ3 What is the impact of Little-Branch complexity parameters (alpha, beta) on performance and FLOPs across vision and speech tasks?
• RQ4 Does the approach generalize across architectures (ResNet, ResNeXt, SEResNeXt) and modalities (vision and speech)?

Key findings

• On ImageNet object recognition, bL-Net achieves about 1/3 less computation with accuracy gains over 1 percentage point across several backbones, with potential gains up to 1/2 FLOPs reduction without accuracy loss.
• Across ResNet, ResNeXt, and SEResNeXt backbones, bL-Net yields up to roughly 2x speedups while maintaining or improving top-1 accuracy compared to baselines.
• Deeper models benefit more from bL-Net, with resnet-152-based variants showing larger speedups (e.g., 2.3x) than shallower ones.
• When evaluated at higher input resolutions (e.g., 256x256), bL-Net maintains fewer FLOPs and delivers improved accuracy over baselines.
• In speech recognition, bL-Net reduces FLOPs by about 30% with slightly better word error rates (WER) than baselines, demonstrating cross-domain generalization.
• Linear merging of branches provides better accuracy and regularization than concatenation in both object and speech experiments.