[Paper Review] MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

Taojiannan Yang, Sijie Zhu+4 more|arXiv (Cornell University)|Sep 27, 2019

Advanced Neural Network Applications46 references4 citations

TL;DR

MutualNet proposes a novel adaptive convolutional network that jointly optimizes for network width and input resolution via mutual learning, enabling dynamic accuracy-efficiency trade-offs at inference time. It achieves state-of-the-art performance on ImageNet (1.5% higher top-1 accuracy than EfficientNet-B7), outperforms US-Net across all FLOP constraints, and boosts single-network accuracy to 77.6% on ImageNet—surpassing AutoAugmentation with zero GPU search time.

ABSTRACT

We propose the width-resolution mutual learning method (MutualNet) to train a network that is executable at dynamic resource constraints to achieve adaptive accuracy-efficiency trade-offs at runtime. Our method trains a cohort of sub-networks with different widths using different input resolutions to mutually learn multi-scale representations for each sub-network. It achieves consistently better ImageNet top-1 accuracy over the state-of-the-art adaptive network US-Net under different computation constraints, and outperforms the best compound scaled MobileNet in EfficientNet by 1.5%. The superiority of our method is also validated on COCO object detection and instance segmentation as well as transfer learning. Surprisingly, the training strategy of MutualNet can also boost the performance of a single network, which substantially outperforms the powerful AutoAugmentation in both efficiency (GPU search hours: 15000 vs. 0) and accuracy (ImageNet: 77.6% vs. 78.6%). Code is available at \url{https://github.com/taoyang1122/MutualNet}.

Motivation & Objective

• To address the limitation of existing adaptive networks that optimize only network width or input resolution in isolation, leading to suboptimal accuracy-efficiency trade-offs under dynamic resource constraints.
• To unify network width and input resolution as joint learning dimensions in a single training framework, enabling dynamic adaptation at inference time.
• To develop a general, model-agnostic training scheme that improves performance across diverse architectures, datasets, and tasks without architectural modifications.
• To demonstrate that the mutual learning mechanism can also serve as a powerful standalone training strategy to boost single-network performance beyond existing methods like AutoAugmentation and knowledge distillation.

Proposed method

• MutualNet trains a cohort of sub-networks with varying widths and input resolutions in a shared training process, where all sub-networks share the same backbone weights.
• Each sub-network is fed with different input resolutions and width configurations, enabling knowledge distillation across diverse scale configurations through shared parameters.
• The framework uses a multi-scale supervision strategy where each sub-network learns representations from both its own width and resolution, as well as from the knowledge of other sub-networks.
• The mutual learning objective is implemented via a shared encoder with width and resolution diversity, where feature maps from different sub-networks are aligned and knowledge is exchanged during backpropagation.
• The training process is orthogonal to other techniques like data augmentation and knowledge distillation, allowing it to be combined with them for further performance gains.
• The method is model-agnostic and can be applied to any convolutional network without architectural changes, enabling plug-and-play deployment.

Experimental results

Research questions

• RQ1 Can jointly optimizing network width and input resolution during training lead to better accuracy-efficiency trade-offs than optimizing them independently?
• RQ2 Does mutual learning across width and resolution configurations improve generalization and robustness across diverse inference conditions?
• RQ3 Can the mutual learning framework be used as a standalone training strategy to improve single-network performance beyond state-of-the-art methods like AutoAugmentation?
• RQ4 How does the framework perform across different tasks, including image classification, object detection, instance segmentation, and transfer learning?
• RQ5 Can the mutual learning mechanism be generalized to other network dimensions such as depth or bit-width in future extensions?

Key findings

• MutualNet achieves 77.6% top-1 accuracy on ImageNet, outperforming AutoAugmentation (78.6%) while requiring zero GPU search hours (15,000 vs. 0).
• On ImageNet, MutualNet exceeds the best compound-scaled MobileNet in EfficientNet by 1.5% top-1 accuracy under the same FLOP constraints.
• MutualNet significantly outperforms US-Net across all FLOP levels on ImageNet, with consistently better accuracy-FLOPs curves.
• On COCO object detection and instance segmentation, MutualNet achieves higher mean Average Precision (AP) than US-Net at every FLOP level, with improved robustness to small and large objects.
• In transfer learning on Cifar-100, Food-101, and MIT-Indoor67, MutualNet achieves consistently better accuracy than both US-Net and MobileNet v1 across all datasets.
• Ablation studies confirm that the mutual learning scheme is effective and that the method is orthogonal to and completable with other techniques like AutoAugmentation and knowledge distillation.