[Paper Review] Multi-scale prediction for robust hand detection and classification

Lu Ding, Yong Wang+3 more|arXiv (Cornell University)|Apr 23, 2018

Hand Gesture Recognition Systems15 references3 citations

TL;DR

This paper proposes Multi-scale Prediction Fully Convolutional Networks (MSP-RFCN), a novel approach that enhances hand detection and classification by leveraging multi-scale feature maps for dense region proposals and fused predictions. The method achieves state-of-the-art performance on VIVA, Oxford, and ARJ datasets, outperforming existing methods in accuracy and robustness under challenging conditions such as occlusion, low resolution, and illumination variation.

ABSTRACT

In this paper, we present a multi-scale Fully Convolutional Networks (MSP-RFCN) to robustly detect and classify human hands under various challenging conditions. In our approach, the input image is passed through the proposed network to generate score maps, based on multi-scale predictions. The network has been specifically designed to deal with small objects. It uses an architecture based on region proposals generated at multiple scales. Our method is evaluated on challenging hand datasets, namely the Vision for Intelligent Vehicles and Applications (VIVA) Challenge and the Oxford hand dataset. It is compared against recent hand detection algorithms. The experimental results demonstrate that our proposed method achieves state-of-the-art detection for hands of various sizes.

Motivation & Objective

• To address the persistent challenge of detecting small, deformable, and occluded hands in real-world scenarios.
• To overcome limitations of existing Faster R-CNN and RFCN-based methods in detecting small objects due to feature map downsampling and RoI pooling misalignment.
• To improve detection robustness under low resolution, illumination changes, and occlusions through multi-scale feature fusion.
• To achieve state-of-the-art performance in both hand detection and classification (left/right, driver/passenger) on multiple benchmark datasets.
• To design a lightweight, efficient detection framework that maintains high accuracy while supporting real-time inference.

Proposed method

• The method employs a multi-scale prediction scheme using feature maps from multiple layers of a CNN to generate class scores and bounding box regressions at different spatial resolutions.
• Instead of fusing feature maps via concatenation or addition, each feature map level independently predicts detection scores and bounding boxes, preserving scale-specific invariance and localization precision.
• The network integrates predictions from higher-level (context-rich) and lower-level (high-resolution) feature maps to balance semantic context and spatial detail.
• The approach is built upon the RFCN framework with modifications to the RPN and head network to support multi-scale region proposals and detection.
• Position-sensitive ROI pooling is used for localization, with potential improvements via ROI align to reduce quantization errors.
• The model is trained end-to-end using cross-entropy loss for classification and smooth L1 loss for bounding box regression.

Experimental results

Research questions

• RQ1 Can multi-scale feature maps improve detection accuracy for small and occluded hands in real-world images?
• RQ2 How does fusing predictions from multiple feature map levels enhance localization and classification performance compared to single-scale detection?
• RQ3 Does the proposed multi-scale RFCN architecture outperform existing state-of-the-art methods on challenging hand detection benchmarks?
• RQ4 To what extent does the method maintain robustness under illumination changes, low resolution, and heavy occlusion?
• RQ5 Can the model achieve high accuracy while maintaining real-time inference speed for practical deployment?

Key findings

• On the VIVA dataset, the proposed method achieved 95.7% AP and 95.3% AR at level L1, and 91.3% AP and 87.6% AR at level L2 using ResNet-101, outperforming all compared methods.
• For hand classification (left/right and driver/passenger), the method achieved 81.3% AP and 72.9% AR for left/right hands, and 81.7% AP and 73.4% AR for driver/passenger, surpassing all baselines.
• On the Oxford dataset, the method achieved 83.1% AP and 85.2% AR using ResNet-101, significantly outperforming prior methods such as Mittal et al. (48.2% AP) and MS-RFCN (75.1% AP).
• On the ARJ dataset, the method achieved 84.0% AP and 86.4% AR using ResNet-101, exceeding FRCNN (68.2% AP) and R-FCN (73.1% AP).
• The method demonstrated real-time inference with 4.0–5.0 FPS on VIVA using VGG16 and ResNet-101, respectively, indicating practical deployability.
• Qualitative results show robust detection across occlusions, illumination variations, and low-resolution conditions, confirming the method's resilience in real-world scenarios.