[Paper Review] Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification

Yulin Wang, Kangchen Lv+4 more|arXiv (Cornell University)|Oct 11, 2020

Advanced Neural Network Applications87 citations

TL;DR

GFNet reduces spatial redundancy by sequentially processing small image patches (glance then focus) guided by reinforcement learning, achieving adaptive inference with lower computation while maintaining accuracy.

ABSTRACT

The accuracy of deep convolutional neural networks (CNNs) generally improves when fueled with high resolution images. However, this often comes at a high computational cost and high memory footprint. Inspired by the fact that not all regions in an image are task-relevant, we propose a novel framework that performs efficient image classification by processing a sequence of relatively small inputs, which are strategically selected from the original image with reinforcement learning. Such a dynamic decision process naturally facilitates adaptive inference at test time, i.e., it can be terminated once the model is sufficiently confident about its prediction and thus avoids further redundant computation. Notably, our framework is general and flexible as it is compatible with most of the state-of-the-art light-weighted CNNs (such as MobileNets, EfficientNets and RegNets), which can be conveniently deployed as the backbone feature extractor. Experiments on ImageNet show that our method consistently improves the computational efficiency of a wide variety of deep models. For example, it further reduces the average latency of the highly efficient MobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and pre-trained models are available at https://github.com/blackfeather-wang/GFNet-Pytorch.

Motivation & Objective

• Motivate reducing computation by exploiting spatial redundancy in high-resolution image classification.
• Develop a two-stage framework (glance and focus) to adaptively process small inputs and terminate when confident.
• Ensure compatibility with lightweight backbones (e.g., MobileNet, EfficientNet, RegNet) for efficient inference.

Proposed method

• Process a sequence of small inputs starting with a down-sampled full image (glance).
• Use a patch proposal network to localize discriminative regions for subsequent focus steps.
• Employ a recurrent classifier to output predictions at each step with confidence-based early termination.
• Train patch selection with policy gradient to maximize incremental gains in class probability.
• Allow online control of computation via configurable confidence thresholds (budgeted/in anytime settings).
• Optionally regularize encoders to encourage linearly separable features and improve localization.

Experimental results

Research questions

• RQ1 Can a sequential, patch-based processing strategy maintain classification accuracy while reducing computation on high-resolution images?
• RQ2 How should discriminative regions be selected and how should computation be adaptively allocated across images?
• RQ3 How does GFNet perform with state-of-the-art lightweight CNN backbones on budgeted and anytime inference tasks?
• RQ4 What is the impact of gaze-like glance steps versus focus steps on accuracy and latency?

Key findings

• GFNet consistently improves efficiency across MobileNet-V3, RegNet-Y, EfficientNet, ResNet, and DenseNet backbones.
• For MobileNet-V3, GFNet reduces Multiply-Adds by up to 1.4x at similar accuracy.
• For ResNets and DenseNets, GFNet achieves roughly 2–3x reductions in computation for the same accuracy.
• On iPhone XS Max, GFNet yields about 1.3x to 2.9x speedups in latency for relevant backbones.
• GFNet enables online adjustment of compute to meet budgets with minimal loss in accuracy.
• GFNet achieves competitive or superior accuracy in budgeted batch classification and anytime prediction compared to baselines like MSDNet.