[Paper Review] Joint Hand Detection and Rotation Estimation by Using CNN

Xiaoming Deng, Ye Yuan+5 more|arXiv (Cornell University)|Dec 8, 2016

Hand Gesture Recognition Systems4 citations

TL;DR

This paper proposes a joint deep learning framework for hand detection and in-plane rotation estimation using a convolutional neural network (CNN) with a novel derotation layer, enabling end-to-end optimization. The method achieves state-of-the-art performance on Oxford and Egohands benchmarks, with a 7× reduction in inference time and improved detection accuracy through mutual benefit between rotation estimation and detection tasks.

ABSTRACT

Hand detection is essential for many hand related tasks, e.g. parsing hand pose, understanding gesture, which are extremely useful for robotics and human-computer interaction. However, hand detection in uncontrolled environments is challenging due to the flexibility of wrist joint and cluttered background. We propose a deep learning based approach which detects hands and calibrates in-plane rotation under supervision at the same time. To guarantee the recall, we propose a context aware proposal generation algorithm which significantly outperforms the selective search. We then design a convolutional neural network(CNN) which handles object rotation explicitly to jointly solve the object detection and rotation estimation tasks. Experiments show that our method achieves better results than state-of-the-art detection models on widely-used benchmarks such as Oxford and Egohands database. We further show that rotation estimation and classification can mutually benefit each other.

Motivation & Objective

• To address the challenge of hand detection in uncontrolled environments with high appearance variation due to wrist rotation and cluttered backgrounds.
• To jointly optimize hand detection and in-plane rotation estimation for improved accuracy and efficiency.
• To design a context-aware region proposal generation method that outperforms selective search and objectness in recall and mean average best overlap (MABO).
• To develop a supervised, differentiable derotation layer that enables end-to-end training for rotation alignment and detection.
• To reduce inference time by avoiding brute-force rotation search through explicit rotation estimation.

Proposed method

• Proposes a context-aware proposal generation algorithm that improves recall and MABO over selective search and objectness by leveraging discriminative features.
• Introduces a derotation layer that explicitly rotates feature maps to an upright orientation based on predicted rotation angles, enabling end-to-end training.
• Employs a two-stream CNN architecture where rotation estimation and detection are jointly optimized with shared feature extraction for efficiency.
• Uses a supervised rotation estimation network to predict in-plane rotation, which is then applied via the derotation layer to align input proposals before detection.
• Applies ROI pooling to extract fixed-size features from rotated proposals for classification, ensuring spatial consistency.
• Trains the entire network end-to-end with joint optimization, allowing detection and rotation tasks to mutually improve performance.

Experimental results

Research questions

• RQ1 Can joint learning of hand detection and in-plane rotation estimation improve detection accuracy compared to separate training?
• RQ2 Does explicit rotation alignment via a differentiable derotation layer enhance feature learning and model generalization?
• RQ3 Can a context-aware proposal generation method significantly outperform selective search and objectness in recall and MABO for hand detection?
• RQ4 To what extent does joint training reduce inference time while maintaining or improving performance?
• RQ5 How does rotation estimation quality affect detection performance, and vice versa?

Key findings

• The proposed method achieves a mean average precision (mAP) of 53.5% on the Oxford dataset, outperforming state-of-the-art models.
• On the Egohands dataset, the joint model achieves an mAP of 75.7%, surpassing the baseline method (73.3%) and demonstrating improved generalization.
• Rotation estimation accuracy reaches 49.01% within 10° error on Egohands, with 76.68% within 20°, showing strong performance on rotation estimation.
• The derotation layer enables a 7× reduction in inference time compared to brute-force rotation search, achieving 8 seconds per image.
• Joint training improves detection mAP by 2% and rotation estimation by 1% compared to non-jointly trained models, confirming mutual benefit.
• The model trained with ground-truth rotation achieves 50.9% mAP, and the proposed method reaches 48.3% mAP, indicating strong optimization and minimal performance gap.