[Paper Review] Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models

Xiaofang Wang, Dan Kondratyuk+4 more|arXiv (Cornell University)|Dec 3, 2020

Advanced Neural Network Applications40 references21 citations

TL;DR

This paper demonstrates that simple committee-based models—using off-the-shelf, independently pre-trained models in ensembles or cascades—can match or exceed state-of-the-art accuracy while reducing FLOPs by up to 5.4x, outperforming both sophisticated neural architecture search (NAS) methods and manually designed models across image classification, video classification, and semantic segmentation tasks.

ABSTRACT

Committee-based models (ensembles or cascades) construct models by combining existing pre-trained ones. While ensembles and cascades are well-known techniques that were proposed before deep learning, they are not considered a core building block of deep model architectures and are rarely compared to in recent literature on developing efficient models. In this work, we go back to basics and conduct a comprehensive analysis of the efficiency of committee-based models. We find that even the most simplistic method for building committees from existing, independently pre-trained models can match or exceed the accuracy of state-of-the-art models while being drastically more efficient. These simple committee-based models also outperform sophisticated neural architecture search methods (e.g., BigNAS). These findings hold true for several tasks, including image classification, video classification, and semantic segmentation, and various architecture families, such as ViT, EfficientNet, ResNet, MobileNetV2, and X3D. Our results show that an EfficientNet cascade can achieve a 5.4x speedup over B7 and a ViT cascade can achieve a 2.3x speedup over ViT-L-384 while being equally accurate.

Motivation & Objective

• To re-evaluate committee-based models (ensembles and cascades) as a viable, overlooked paradigm for efficient deep learning.
• To address the lack of systematic comparison between committee models and modern single-model architectures in terms of efficiency and accuracy.
• To demonstrate that simple, off-the-shelf committee models can surpass complex, search-optimized architectures without architectural tuning.
• To provide empirical evidence that committee models generalize across diverse tasks and model families, including ViT, EfficientNet, ResNet, MobileNetV2, and X3D.
• To advocate for committee models as strong baselines in future model development and practical deployment.

Proposed method

• Construct ensembles by averaging predictions from multiple independently pre-trained models without retraining or specialized weighting.
• Build cascades by sequentially applying pre-trained models and using a confidence threshold (maximum softmax probability) to determine early exit.
• For dense prediction tasks like semantic segmentation, compute a per-image confidence score as the average of pixel-wise confidence scores to decide whether to pass to the next model.
• Apply grid-based inference in segmentation cascades, where each grid cell is evaluated independently for early exit, reducing computation waste on hard regions.
• Use standard pre-trained models (e.g., EfficientNet-B5, ViT-L-384, X3D) as components without architectural modification or fine-tuning.
• Evaluate performance using standard metrics: top-1 accuracy on ImageNet, mIoU on Cityscapes, and top-1 accuracy on Kinetics-600, with FLOPs and latency as efficiency measures.

Experimental results

Research questions

• RQ1 Can simple committee models built from off-the-shelf, pre-trained models outperform state-of-the-art single models in accuracy and efficiency?
• RQ2 How do committee models compare to modern neural architecture search (NAS) methods in terms of speed-accuracy trade-offs?
• RQ3 Does the efficiency gain of cascades hold across different model families and tasks, including image, video, and dense prediction tasks?
• RQ4 Can cascades provide both speedup and worst-case FLOP guarantees, making them suitable for real-time deployment?
• RQ5 Can self-cascades using a single model at multiple resolutions achieve significant inference speedups without accuracy loss?

Key findings

• An ensemble of two EfficientNet-B5 models matches the accuracy of EfficientNet-B7 (a state-of-the-art ImageNet model) while using only 55% of the FLOPs (20.5B vs. 37B).
• A ResNet cascade achieves higher accuracy than Inception-v4 on ImageNet while using fewer FLOPs, demonstrating superiority over a strong single model.
• An EfficientNet cascade achieves a 5.4x speedup over EfficientNet-B7 with equal top-1 accuracy on ImageNet.
• A ViT cascade matches ViT-L-384 accuracy on ImageNet while achieving a 2.3x speedup and reducing average FLOPs by 2.3x.
• On Kinetics-600, a cascade of X3D models outperforms X3D-XL by 1.2% top-1 accuracy while using 3.7x fewer average FLOPs.
• In semantic segmentation, a cascade of DeepLabv3 models with grid-level inference achieves 1.3x faster inference than ResNet-101 while matching its mIoU of 78.1%.