[Paper Review] Beyond One Glance: Gated Recurrent Architecture for Hand Segmentation

Wei Wang, Kaicheng Yu+3 more|arXiv (Cornell University)|Nov 27, 2018

Hand Gesture Recognition Systems26 references4 citations

TL;DR

This paper proposes a gated recurrent architecture that refines hand segmentation by iteratively updating both the segmentation mask and the network's internal state across multiple encoder-decoder layers of a U-Net backbone. It outperforms state-of-the-art one-shot and simpler recurrent methods on multiple hand segmentation benchmarks and generalizes effectively to road segmentation, demonstrating superior accuracy with fewer parameters.

ABSTRACT

As mixed reality is gaining increased momentum, the development of effective and efficient solutions to egocentric hand segmentation is becoming critical. Traditional segmentation techniques typically follow a one-shot approach, where the image is passed forward only once through a model that produces a segmentation mask. This strategy, however, does not reflect the perception of humans, who continuously refine their representation of the world. In this paper, we therefore introduce a novel gated recurrent architecture. It goes beyond both iteratively passing the predicted segmentation mask through the network and adding a standard recurrent unit to it. Instead, it incorporates multiple encoder-decoder layers of the segmentation network, so as to keep track of its internal state in the refinement process. As evidenced by our results on standard hand segmentation benchmarks and on our own dataset, our approach outperforms these other, simpler recurrent segmentation techniques, as well as the state-of-the-art hand segmentation one. Furthermore, we demonstrate the generality of our approach by applying it to road segmentation, where it also outperforms other baseline methods.

Motivation & Objective

• To address the limitations of one-shot segmentation in egocentric hand analysis by mimicking human perception through iterative refinement.
• To improve segmentation accuracy by maintaining and updating the network's internal state across multiple refinement steps.
• To develop a generic recurrent architecture applicable beyond hand segmentation, demonstrating generalization to other semantic segmentation tasks.
• To create a large-scale, publicly available hand segmentation dataset (12.5K images) to support future research.

Proposed method

• Proposes a novel gated recurrent unit (GRU) that operates across multiple encoder-decoder layers of a U-Net, enabling internal state updates beyond a single hidden layer.
• Introduces two variants: Ours-DRU (Dense Recurrent Unit) and Ours-SRU (Simple Recurrent Unit), both incorporating the segmentation mask as input at each refinement step.
• Uses a recurrent mechanism that propagates high-level features through the network’s internal layers, allowing for deeper contextual refinement.
• Employs a multi-stage refinement process where the network processes the image and previous mask iteratively, updating its internal state at each step.
• Applies the same architecture to road segmentation using the Massachusetts Roads dataset, validating its generalization capability.
• Trains from scratch without pre-training on ImageNet, enabling comparison with methods that use large pre-trained backbones.

Experimental results

Research questions

• RQ1 Can a recurrent architecture that refines both the segmentation mask and the network’s internal state improve hand segmentation performance beyond one-shot methods?
• RQ2 How does incorporating recurrence across multiple encoder-decoder layers compare to standard RNNs or mask-only recurrence in refinement?
• RQ3 Does the proposed architecture generalize to other semantic segmentation tasks beyond hand segmentation?
• RQ4 Can the method achieve state-of-the-art performance with fewer parameters than existing SOTA models?

Key findings

• On the Ego-Youtube-Hands (EYTH) dataset, the proposed Ours-DRU(4) model achieves a mean Intersection over Union (mIOU) of 0.8355, outperforming the previous SOTA RefineNet.
• On the HandOverFace (HOF) dataset, the model achieves second-best performance, with mIOU of 0.784, despite the small training set size (198 images).
• In road segmentation on the Massachusetts Roads dataset, Ours-DRU(4) achieves an mIOU of 0.560 and a P/R score of 0.757, outperforming baselines including a method with a larger VGG19-based U-Net and a topology-aware loss.
• The model achieves superior performance with only 8 times fewer parameters than a comparable SOTA method, indicating high parameter efficiency.
• Visual comparisons show that the model produces more accurate predictions than ground truth in challenging cases, such as fine finger separations and occluded regions.
• Ablation studies confirm that removing the segmentation mask from input reduces mIOU on EYTH from 0.8355 to 0.8256, but the model still outperforms Rec-Middle (0.814), proving the importance of mask feedback.